Fara1.5-27B on RTX 3090: local browser computer-use agent with llama.cpp

What You'll Build

A local computer-use agent: llama-server hosting Fara1.5-27B (Q4_K_M GGUF + its vision projector) on a single RTX 3090, with Microsoft's own fara-cli harness pointed at that endpoint over the OpenAI-compatible API. Fara drives a Playwright browser, sees each page only as a screenshot, and answers with a <tool_call> block containing a click, type, scroll or navigate action at pixel coordinates it predicts itself.

Hardware data: RTX 3090 (24 GB VRAM) · Q4_K_M weights 16.33 GiB + vision projector 0.86 GiB, ~19.8 GiB accounted at 32K context · See benchmark data

⚠️ This install is off the vendor's supported path, deliberately. The model card's Requirements section asks for "A set of GPUs with enough memory for a 27B model in bf16 (A6000, A100, H100, and B200 have been tested). We recommend sharding the model over at least 2 GPUs." That recommendation is about bf16 — the safetensors set is 50.96 GiB across 10 shards, which does not fit one 24 GB card by any arrangement. Everything below runs a community 4-bit quantization Microsoft never published or blessed. What it costs is discussed under Results; read that section before you trust a click coordinate.

⚠️ This model clicks, types and submits in a real browser. The model card is blunt about it: "Don't run this model with unrestricted browser access on a machine that has anything sensitive on it." Fara is trained to pause at "critical points" (personal data, payments, sign-ins, irreversible submissions) but that is a behaviour, not an enforcement boundary. Run it in a container or a throwaway VM, keep --headful on while you are learning its behaviour, and never point it at a browser profile holding live sessions. Microsoft's recommended deployment is MagenticLite / Magentic-UI, which adds the sandbox, allow-lists, watch-mode and a pause button that fara-cli alone does not.

ℹ️ Not a chat model. Fara1.5-27B is in our multimodal vertical because it takes images and text, but it is not a general-purpose VLM. It is fine-tuned to emit chain-of-thought followed by a single computer_use tool call against a fixed system prompt. Ask it a normal question and you will get agent-shaped output. The harness is not optional garnish — it is the interface.

Requirements

VRAM budget

There is no measured peak for this pair — /check/fara1-5-27b/rtx-3090 has zero benchmarks. What follows is arithmetic from file sizes and the GGUF's own metadata, shown so you can check it.

Fara1.5-27B is built on Qwen3.5-27B, which is a hybrid architecture, and that changes the KV maths substantially. Its config.json lists 64 layers of which only every 4th is full_attention (full_attention_interval: 4) — 16 attention layers and 48 linear-attention layers. llama.cpp implements this as a hybrid memory: a normal KV cache for the 16 attention layers, and a fixed-size recurrent state for the other 48 (llama-model.cpp builds a llama_memory_hybrid for LLM_ARCH_QWEN35). Only the first term grows with context.

Per token, the attention half costs head_count_kv (4) × (key_length 256 + value_length 256) × 2 bytes × 16 layers = 64 KiB — all four values read straight out of the Q4_K_M GGUF's metadata. The recurrent half is context-independent: (conv_kernel−1) × (ssm_inner 6144 + 2 × group_count 16 × state_size 128) plus state_size × ssm_inner, in F32, over 48 layers = 149.625 MiB per sequence — and llama-server reserves one copy per slot, four of them, so the real figure is 598.5 MiB = 0.584 GiB.

That ×4 is the part every budget for this model gets wrong, and it also explains why the KV cache is not multiplied the same way. common/arg.cpp sets params.n_parallel = -1 for the server example specifically; tools/server/server.cpp resolves that sentinel with params.n_parallel = 4; params.kv_unified = true;; common/common.cpp then assigns cparams.n_seq_max = params.n_parallel. src/llama-model.cpp sizes the recurrent memory with /* recurrent_rs_size */ std::max((uint32_t) 1, cparams.n_seq_max) on the branch that names LLM_ARCH_QWEN35 explicitly, and src/llama-memory-recurrent.cpp allocates n_rows = mem_size * (1 + n_rs_seq), where n_rs_seq defaults to 0. So the recurrent state is allocated once per slot — four times — whether or not you ever use more than one. The same default line sets kv_unified = true, which gives all four slots one shared attention cache, so the 64 KiB/token figure above is not multiplied. One term scales with the slot count and the other does not; a budget that treats them alike is wrong in one direction or the other.

Each column below sums the exact byte counts and is rounded once at the end, which is why the Q4_K_M total sits a thousandth above its own rounded rows added by hand — rounding the inputs first is the error this budget is avoiding.

The slack still has to absorb the CUDA context, llama.cpp's compute buffers and the vision encoder's working set for a 5,040-patch image — none of which I could source a measurement for. Q4_K_M at 32K context leaves the most defensible margin of the options I could size; if it does not hold on your machine, the fallbacks are in Troubleshooting. File sizes are from the HuggingFace tree API and are quoted in GiB here, so they read ~7% below the decimal-GB figures on the repo's file listing.

How much context you actually need

The fara README says, of any non-vLLM endpoint: "Please ensure that context length is set to at least 15000 tokens and temperature to 0 for best results." That floor is reachable, and here is where it goes.

fara-cli renders the browser at 1440×900 (viewport_width / viewport_height in fara15_agent.py) and passes the screenshot through Qwen's smart_resize with factor = patch_size 16 × merge_size 2 = 32. 900 rounds to 896; 1440 is already a multiple of 32; the resulting 1,290,240 pixels sit inside the configured [3136, 12845056] window, so no rescale happens. That gives (1440/16) × (896/16) ÷ 4 = 1,260 tokens per screenshot. The same file caps history at max_n_images: 3, so at most 3,780 tokens of the prompt is imagery — the repo's own internal estimate (image_token_estimate: 1500) is in the same ballpark. The rest of the 15K is the system prompt with the full computer_use schema, plus accumulated thoughts and actions. -c 32768 gives a long trajectory real room; -c 16384 is the honest minimum and saves 1 GiB.

Installation

1. Get a llama.cpp build with qwen35 support

qwen35 is registered in mainline llama.cpp (LLM_ARCH_QWEN35 in llama-arch.cpp), and its vision half uses the qwen3vl_merger projector already present in mtmd. The GGUFs below were built on release b10087, so use that or newer.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

2. Download the weights and the vision projector

Microsoft ships no first-party GGUF for Fara1.5 — the org's repos for 4B/9B/27B are safetensors only. The community conversion used here is bartowski's imatrix build. Download both files; the model is useless without the projector.

pip install -U huggingface_hub

hf download bartowski/Fara1.5-27B-GGUF \
  --include "Fara1.5-27B-Q4_K_M.gguf" "mmproj-Fara1.5-27B-f16.gguf" \
  --local-dir ./fara1.5-27b

Four independent quantizers (bartowski, prithivMLmods, Abiray, DevQuasar) each ship a text GGUF and a matching mmproj, which is decent evidence that the conversion path works end to end rather than one person getting lucky. Their quant recipes differ, though — bartowski's imatrix Q4_K_M is 17.53 GB where the other three are 16.55 GB — so the budget table above only applies to the file it names.

3. Install the Fara harness

git clone https://github.com/microsoft/fara.git
cd fara
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
playwright install

Skip the [vllm] extra from the repo's README — you are not hosting with vLLM, and it pulls a large dependency tree you will not use.

Running

1. Start llama-server

./build/bin/llama-server \
  -m ./fara1.5-27b/Fara1.5-27B-Q4_K_M.gguf \
  --mmproj ./fara1.5-27b/mmproj-Fara1.5-27B-f16.gguf \
  --port 5000 \
  -ngl 99 \
  -c 32768 \
  --image-min-tokens 1024 \
  --reasoning-format none

Two of those flags are not optional and are the reason this recipe exists:

• --reasoning-format none. Fara opens a <think> tag and never closes it. Under llama-server's default reasoning extraction the whole reply — thoughts and the <tool_call> block — is filed as reasoning_content and message.content comes back empty, so any OpenAI-style client sees nothing. This is documented in the model's discussions thread #4, reported against llama.cpp b10107 with the 27B; the reporter adds "If you have a router/proxy in front doing its own reasoning extraction, disable it there too." A Microsoft team member (swhitehead, flagged as an org member on that thread) asked for it upstream, which became the still-open fara issue #82. The none value is llama-server's own documented setting — "leaves thoughts unparsed in message.content".
• --image-min-tokens 1024. llama.cpp warns at load for every Qwen-VL-family projector: "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks", pointing at issue #16842 on bounding-box correctness. A 1440×900 screenshot already lands at 1,260 tokens, so this flag changes nothing in the default case — it pins the floor so a smaller viewport or a future default can never quietly drop grounding below it.

llama-server's OpenAI-compatible endpoint takes images the way the harness sends them: for messages[i].content[j] with type == "image_url", the server docs state image_url.url may be a remote URL, raw base64, a data:image/...;base64 URI, or a local path. The harness serialises each screenshot as exactly that — {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{...}"}} in clients/messages.py — so no shim is needed between the two.

2. Point fara-cli at it

The harness defaults to http://localhost:5000/v1 with model name Fara1.5-9B, so the port lines up but the model name must be overridden.

fara-cli \
  --task "find the current UTC time on time.is and tell me the seconds" \
  --base_url http://localhost:5000/v1 \
  --api_key not-needed \
  --model Fara1.5-27B \
  --headful \
  --output_folder ./run1

A Chromium window opens on Bing, and each round prints the model's thoughts followed by the action it chose. Temperature is fixed at 0 inside the agent — you do not need to set it on the server. Per-step screenshots and a data_point.json with the full trajectory land in --output_folder. Drop --headful once you trust the setup; keep it while you are learning what the model does with your machine.

For the sandbox, allow-list and pause controls the model card asks for, run the same endpoint under Magentic-UI instead of the bare CLI.

Results

• VRAM usage: no measured peak exists yet. The accounted total is 19.778 GiB at -c 32768 (16.329 weights + 0.864 projector + 2.000 attention KV + 0.584 recurrent state), leaving 4.222 GiB of a 24 GiB card for compute buffers, the CUDA context and vision encoding. Derivation and sources are in VRAM budget. If you measure a real peak, please send it to /contribute — it becomes the first datapoint on /check/fara1-5-27b/rtx-3090.
• Speed: omitted. No benchmark of Fara1.5-27B on any consumer GPU has been published that I could find, /check has zero rows, and a search for one returned figures for other 27B Qwen models instead — which are not this model and are not quoted here. A generation-speed number is also the wrong headline for a CUA: what you feel is the round-trip per step, which includes prompt-processing a fresh 1,260-token screenshot plus a Playwright action. Measurements welcome at /contribute.
• Quality notes — read this one. Fara's whole job is emitting precise pixel coordinates, and 4-bit quantization of a grounding model is a real risk, not a formality. I found no evaluation of how Q4_K_M affects Fara1.5's grounding accuracy — not from Microsoft, not from the community. Microsoft's own accuracy figures (WebVoyager 89.3, Online-Mind2Web 72.3, WebTailBench outcome 40.2) are all bf16 on datacenter hardware; nothing licenses carrying them over to a 4-bit build. Treat a misclick here as possibly the quantization rather than the model, and if a task depends on precision, verify against a bf16-hosted endpoint before concluding anything.
• What is and isn't confirmed at runtime. The vision half is well attested statically: mainline llama.cpp registers this architecture's converter and the qwen3vl_merger projector, the mmproj file carries a coherent 27-block vision tower (its clip.vision.is_deepstack_layers array holds 27 entries, all false — matching the empty deepstack_visual_indexes in config.json, so this tower uses no deepstack layers), and four independent quantizers produced one. There is also a public report of this exact Q4_K_M file running under llama-server and being driven by an agent harness that feeds it screenshots (discussions #4 → issue #82). But I found no report that directly shows an image reaching this model under llama-server — no captioning transcript, no llama-mtmd-cli run, no maintainer confirmation. The harness report implies it strongly; it does not demonstrate it. Static evidence is not a tested path and this recipe does not present it as one.
• On the headline comparison. The fara README states that "Fara1.5-27B outperforms much larger proprietary systems such as OpenAI Operator and Gemini 2.5 Computer Use on Online-Mind2Web." Its table shows 72.3 for Fara1.5-27B against 58.3 and 57.3 — but the same table's footnote says "† denotes numbers sourced from the model's official release or leaderboard rather than re-run by us." Both competitors carry that dagger. So this is a vendor-run agentic benchmark comparing Microsoft's own three-run average against figures the competitors self-reported under their own harnesses, not a controlled head-to-head. Yutori Navigator (n1), also daggered, sits at 64.7; the GPT-5.4-based data-generation solver in the same table reaches 83.4.

For the full benchmark data, see /check/fara1-5-27b/rtx-3090.

Troubleshooting

Every task fails instantly with list index out of range

You forgot --reasoning-format none. The agent's parser does message.split("<tool_call>\n") and immediately indexes [1], so an empty content raises IndexError on the first round — visible in fara15_agent.py and described in the still-open PR #83, which proposes reading reasoning_content as a fallback. Under Magentic-UI the same failure surfaces as a retry loop. Restart llama-server with the flag.

Replies begin with a stray <think> that never closes

Expected, and cosmetic. With --reasoning-format none the raw output is passed through untouched, including the unpaired opening tag. Parsers that split on <tool_call> are unaffected; only the displayed "thoughts" look slightly off.

llama-server warns about 1024 image tokens at startup

That warning fires for every Qwen-VL projector because llama.cpp's configured floor is 8 tokens, not because your screenshot is too small. At the harness's 1440×900 default each screenshot is 1,260 tokens, comfortably above the grounding threshold. Passing --image-min-tokens 1024 as shown silences the ambiguity and guarantees the floor.

CUDA out of memory

In order of preference: drop to -c 16384 (saves 1.00 GiB and still clears the README's 15,000-token floor); add --no-mmproj-offload to keep the vision projector in system RAM (documented in docs/multimodal.md, saves 0.86 GiB at the cost of slower image encoding); or step down to IQ4_XS, which frees a further 2.05 GiB. Quantizing the KV cache is a fourth lever, but on a grounding model I would exhaust the other three first — and if you reach for it, quantize K only (--cache-type-k q8_0). A quantized V cache is coupled to flash attention in src/llama-context.cpp: with flash attention explicitly disabled the server refuses to start, logging quantized V cache requires flash_attn to be enabled, and under the default auto it turns flash attention on for you instead of failing, announcing it only at INFO level. Neither branch is something to discover from a startup log you were not reading.

The agent runs, but it is not the model you think

fara-cli ships two generations. Its --fara-7b flag and the GGUF/LM Studio/Ollama pointers in the repo's hosting section all refer to Fara-7B, the previous-generation Qwen2.5-VL model — not Fara1.5. Older issues in that repo describing "Q5_K_M (~5GB)" and 16 GB cards are Fara-7B threads from 2025. If you did not pass --model Fara1.5-27B, the harness silently used its default name Fara1.5-9B in the request.

Looking for an Ollama or LM Studio entry

There isn't one. ollama.com/library/fara1.5, /fara, /fara1-5 and /fara-7b all 404; there is no microsoft/Fara1.5-27B-GGUF repo; and neither the LM Studio nor the Jan catalogue has a Fara1.5 page. Any GGUF runtime that tracks llama.cpp b10087+ and can load a separate mmproj should work in principle, but llama-server is the only path with a public report of this model behind it.

No other widely-reported issues on this pair. Report problems via the submission form.