Fara1.5-27B on RTX 3090: local browser computer-use agent with llama.cpp

What You'll Build

A local computer-use agent: llama-server hosting Fara1.5-27B (Q4_K_M GGUF + its vision projector) on a single RTX 3090, with Microsoft's own fara-cli harness pointed at that endpoint over the OpenAI-compatible API. Fara drives a Playwright browser, sees each page only as a screenshot, and answers with a <tool_call> block containing a click, type, scroll or navigate action at pixel coordinates it predicts itself.

Hardware data: RTX 3090 (24 GB VRAM) · Q4_K_M weights 16.33 GiB + vision projector 0.86 GiB, ~19.3 GiB accounted at 32K context · See benchmark data

⚠️ This install is off the vendor's supported path, deliberately. The model card's Requirements section asks for "A set of GPUs with enough memory for a 27B model in bf16 (A6000, A100, H100, and B200 have been tested). We recommend sharding the model over at least 2 GPUs." That recommendation is about bf16 — the safetensors set is 50.96 GiB across 10 shards, which does not fit one 24 GB card by any arrangement. Everything below runs a community 4-bit quantization Microsoft never published or blessed. What it costs is discussed under Results; read that section before you trust a click coordinate.

⚠️ This model clicks, types and submits in a real browser. The model card is blunt about it: "Don't run this model with unrestricted browser access on a machine that has anything sensitive on it." Fara is trained to pause at "critical points" (personal data, payments, sign-ins, irreversible submissions) but that is a behaviour, not an enforcement boundary. Run it in a container or a throwaway VM, keep --headful on while you are learning its behaviour, and never point it at a browser profile holding live sessions. Microsoft's recommended deployment is MagenticLite / Magentic-UI, which adds the sandbox, allow-lists, watch-mode and a pause button that fara-cli alone does not.

ℹ️ Not a chat model. Fara1.5-27B is in our multimodal vertical because it takes images and text, but it is not a general-purpose VLM. It is fine-tuned to emit chain-of-thought followed by a single computer_use tool call against a fixed system prompt. Ask it a normal question and you will get agent-shaped output. The harness is not optional garnish — it is the interface.

Requirements

Component	Minimum	Tested
GPU	24 GB VRAM, CUDA	— not measured; budget below is derived (/contribute)
RAM	16 GB system RAM (more if you offload)	—
Storage	18.5 GB for weights (Q4_K_M 17.53 GB + mmproj 0.93 GB, decimal)	~22 GB including the harness and Playwright's Chromium
Software	llama.cpp b10087+, Python 3.10+, Playwright	—

VRAM budget

There is no measured peak for this pair — /check/fara1-5-27b/rtx-3090 has zero benchmarks. What follows is arithmetic from file sizes and the GGUF's own metadata, shown so you can check it.

Fara1.5-27B is built on Qwen3.5-27B, which is a hybrid architecture, and that changes the KV maths substantially. Its config.json lists 64 layers of which only every 4th is full_attention (full_attention_interval: 4) — 16 attention layers and 48 linear-attention layers. llama.cpp implements this as a hybrid memory: a normal KV cache for the 16 attention layers, and a fixed-size recurrent state for the other 48 (llama-model.cpp builds a llama_memory_hybrid for LLM_ARCH_QWEN35). Only the first term grows with context.

Per token, the attention half costs head_count_kv (4) × (key_length 256 + value_length 256) × 2 bytes × 16 layers = 64 KiB — all four values read straight out of the Q4_K_M GGUF's metadata. The recurrent half is context-independent: (conv_kernel−1) × (ssm_inner 6144 + 2 × group_count 16 × state_size 128) plus state_size × ssm_inner, in F32, over 48 layers = 0.15 GiB total.

Component	Q4_K_M	IQ4_XS
Text weights	16.33 GiB	14.28 GiB
Vision projector mmproj-…-f16.gguf	0.86 GiB	0.86 GiB
Attention KV @ 32,768 ctx (f16)	2.00 GiB	2.00 GiB
Recurrent state (48 layers, context-independent)	0.15 GiB	0.15 GiB
Accounted total	19.34 GiB	17.29 GiB
Slack on a 24 GiB card	4.66 GiB	6.71 GiB

The slack still has to absorb the CUDA context, llama.cpp's compute buffers and the vision encoder's working set for a 5,040-patch image — none of which I could source a measurement for. Q4_K_M at 32K context leaves the most defensible margin of the options I could size; if it does not hold on your machine, the fallbacks are in Troubleshooting. File sizes are from the HuggingFace tree API and are quoted in GiB here, so they read ~7% below the decimal-GB figures on the repo's file listing.

How much context you actually need

The fara README says, of any non-vLLM endpoint: "Please ensure that context length is set to at least 15000 tokens and temperature to 0 for best results." That floor is reachable, and here is where it goes.

fara-cli renders the browser at 1440×900 (viewport_width / viewport_height in fara15_agent.py) and passes the screenshot through Qwen's smart_resize with factor = patch_size 16 × merge_size 2 = 32. 900 rounds to 896; 1440 is already a multiple of 32; the resulting 1,290,240 pixels sit inside the configured [3136, 12845056] window, so no rescale happens. That gives (1440/16) × (896/16) ÷ 4 = 1,260 tokens per screenshot. The same file caps history at max_n_images: 3, so at most 3,780 tokens of the prompt is imagery — the repo's own internal estimate (image_token_estimate: 1500) is in the same ballpark. The rest of the 15K is the system prompt with the full computer_use schema, plus accumulated thoughts and actions. -c 32768 gives a long trajectory real room; -c 16384 is the honest minimum and saves 1 GiB.

Installation

1. Get a llama.cpp build with qwen35 support

qwen35 is registered in mainline llama.cpp (LLM_ARCH_QWEN35 in llama-arch.cpp), and its vision half uses the qwen3vl_merger projector already present in mtmd. The GGUFs below were built on release b10087, so use that or newer.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

2. Download the weights and the vision projector

Microsoft ships no first-party GGUF for Fara1.5 — the org's repos for 4B/9B/27B are safetensors only. The community conversion used here is bartowski's imatrix build. Download both files; the model is useless without the projector.

pip install -U huggingface_hub

hf download bartowski/Fara1.5-27B-GGUF \
  --include "Fara1.5-27B-Q4_K_M.gguf" "mmproj-Fara1.5-27B-f16.gguf" \
  --local-dir ./fara1.5-27b

Four independent quantizers (bartowski, prithivMLmods, Abiray, DevQuasar) each ship a text GGUF and a matching mmproj, which is decent evidence that the conversion path works end to end rather than one person getting lucky. Their quant recipes differ, though — bartowski's imatrix Q4_K_M is 17.53 GB where the other three are 16.55 GB — so the budget table above only applies to the file it names.

3. Install the Fara harness

git clone https://github.com/microsoft/fara.git
cd fara
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
playwright install

Skip the [vllm] extra from the repo's README — you are not hosting with vLLM, and it pulls a large dependency tree you will not use.

Running

1. Start llama-server

./build/bin/llama-server \
  -m ./fara1.5-27b/Fara1.5-27B-Q4_K_M.gguf \
  --mmproj ./fara1.5-27b/mmproj-Fara1.5-27B-f16.gguf \
  --port 5000 \
  -ngl 99 \
  -c 32768 \
  --image-min-tokens 1024 \
  --reasoning-format none

Two of those flags are not optional and are the reason this recipe exists:

• --reasoning-format none. Fara opens a <think> tag and never closes it. Under llama-server's default reasoning extraction the whole reply — thoughts and the <tool_call> block — is filed as reasoning_content and message.content comes back empty, so any OpenAI-style client sees nothing. This is documented in the model's discussions thread #4, reported against llama.cpp b10107 with the 27B; the reporter adds "If you have a router/proxy in front doing its own reasoning extraction, disable it there too." A Microsoft team member (swhitehead, flagged as an org member on that thread) asked for it upstream, which became the still-open fara issue #82. The none value is llama-server's own documented setting — "leaves thoughts unparsed in message.content".
• --image-min-tokens 1024. llama.cpp warns at load for every Qwen-VL-family projector: "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks", pointing at issue #16842 on bounding-box correctness. A 1440×900 screenshot already lands at 1,260 tokens, so this flag changes nothing in the default case — it pins the floor so a smaller viewport or a future default can never quietly drop grounding below it.

llama-server's OpenAI-compatible endpoint takes images the way the harness sends them: for messages[i].content[j] with type == "image_url", the server docs state image_url.url may be a remote URL, raw base64, a data:image/...;base64 URI, or a local path. The harness serialises each screenshot as exactly that — {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{...}"}} in clients/messages.py — so no shim is needed between the two.

2. Point fara-cli at it

The harness defaults to http://localhost:5000/v1 with model name Fara1.5-9B, so the port lines up but the model name must be overridden.

fara-cli \
  --task "find the current UTC time on time.is and tell me the seconds" \
  --base_url http://localhost:5000/v1 \
  --api_key not-needed \
  --model Fara1.5-27B \
  --headful \
  --output_folder ./run1

A Chromium window opens on Bing, and each round prints the model's thoughts followed by the action it chose. Temperature is fixed at 0 inside the agent — you do not need to set it on the server. Per-step screenshots and a data_point.json with the full trajectory land in --output_folder. Drop --headful once you trust the setup; keep it while you are learning what the model does with your machine.

For the sandbox, allow-list and pause controls the model card asks for, run the same endpoint under Magentic-UI instead of the bare CLI.

Results

• VRAM usage: no measured peak exists yet. The accounted total is 19.34 GiB at -c 32768 (16.33 weights + 0.86 projector + 2.00 attention KV + 0.15 recurrent state), leaving 4.66 GiB of a 24 GiB card for compute buffers, the CUDA context and vision encoding. Derivation and sources are in VRAM budget. If you measure a real peak, please send it to /contribute — it becomes the first datapoint on /check/fara1-5-27b/rtx-3090.
• Speed: omitted. No benchmark of Fara1.5-27B on any consumer GPU has been published that I could find, /check has zero rows, and a search for one returned figures for other 27B Qwen models instead — which are not this model and are not quoted here. A generation-speed number is also the wrong headline for a CUA: what you feel is the round-trip per step, which includes prompt-processing a fresh 1,260-token screenshot plus a Playwright action. Measurements welcome at /contribute.
• Quality notes — read this one. Fara's whole job is emitting precise pixel coordinates, and 4-bit quantization of a grounding model is a real risk, not a formality. I found no evaluation of how Q4_K_M affects Fara1.5's grounding accuracy — not from Microsoft, not from the community. Microsoft's own accuracy figures (WebVoyager 89.3, Online-Mind2Web 72.3, WebTailBench outcome 40.2) are all bf16 on datacenter hardware; nothing licenses carrying them over to a 4-bit build. Treat a misclick here as possibly the quantization rather than the model, and if a task depends on precision, verify against a bf16-hosted endpoint before concluding anything.
• What is and isn't confirmed at runtime. The vision half is well attested statically: mainline llama.cpp registers this architecture's converter and the qwen3vl_merger projector, the mmproj file carries a coherent 27-block vision tower (its clip.vision.is_deepstack_layers array holds 27 entries, all false — matching the empty deepstack_visual_indexes in config.json, so this tower uses no deepstack layers), and four independent quantizers produced one. There is also a public report of this exact Q4_K_M file running under llama-server and being driven by an agent harness that feeds it screenshots (discussions #4 → issue #82). But I found no report that directly shows an image reaching this model under llama-server — no captioning transcript, no llama-mtmd-cli run, no maintainer confirmation. The harness report implies it strongly; it does not demonstrate it. Static evidence is not a tested path and this recipe does not present it as one.
• On the headline comparison. The fara README states that "Fara1.5-27B outperforms much larger proprietary systems such as OpenAI Operator and Gemini 2.5 Computer Use on Online-Mind2Web." Its table shows 72.3 for Fara1.5-27B against 58.3 and 57.3 — but the same table's footnote says "† denotes numbers sourced from the model's official release or leaderboard rather than re-run by us." Both competitors carry that dagger. So this is a vendor-run agentic benchmark comparing Microsoft's own three-run average against figures the competitors self-reported under their own harnesses, not a controlled head-to-head. Yutori Navigator (n1), also daggered, sits at 64.7; the GPT-5.4-based data-generation solver in the same table reaches 83.4.

For the full benchmark data, see /check/fara1-5-27b/rtx-3090.

Troubleshooting

Every task fails instantly with list index out of range

You forgot --reasoning-format none. The agent's parser does message.split("<tool_call>\n") and immediately indexes [1], so an empty content raises IndexError on the first round — visible in fara15_agent.py and described in the still-open PR #83, which proposes reading reasoning_content as a fallback. Under Magentic-UI the same failure surfaces as a retry loop. Restart llama-server with the flag.

Replies begin with a stray <think> that never closes

Expected, and cosmetic. With --reasoning-format none the raw output is passed through untouched, including the unpaired opening tag. Parsers that split on <tool_call> are unaffected; only the displayed "thoughts" look slightly off.

llama-server warns about 1024 image tokens at startup

That warning fires for every Qwen-VL projector because llama.cpp's configured floor is 8 tokens, not because your screenshot is too small. At the harness's 1440×900 default each screenshot is 1,260 tokens, comfortably above the grounding threshold. Passing --image-min-tokens 1024 as shown silences the ambiguity and guarantees the floor.

CUDA out of memory

In order of preference: drop to -c 16384 (saves 1.00 GiB and still clears the README's 15,000-token floor); add --no-mmproj-offload to keep the vision projector in system RAM (documented in docs/multimodal.md, saves 0.86 GiB at the cost of slower image encoding); or step down to IQ4_XS, which frees a further 2.05 GiB. Quantizing the KV cache is a fourth lever, but on a grounding model I would exhaust the other three first.

The agent runs, but it is not the model you think

fara-cli ships two generations. Its --fara-7b flag and the GGUF/LM Studio/Ollama pointers in the repo's hosting section all refer to Fara-7B, the previous-generation Qwen2.5-VL model — not Fara1.5. Older issues in that repo describing "Q5_K_M (~5GB)" and 16 GB cards are Fara-7B threads from 2025. If you did not pass --model Fara1.5-27B, the harness silently used its default name Fara1.5-9B in the request.

Looking for an Ollama or LM Studio entry

There isn't one. ollama.com/library/fara1.5, /fara, /fara1-5 and /fara-7b all 404; there is no microsoft/Fara1.5-27B-GGUF repo; and neither the LM Studio nor the Jan catalogue has a Fara1.5 page. Any GGUF runtime that tracks llama.cpp b10087+ and can load a separate mmproj should work in principle, but llama-server is the only path with a public report of this model behind it.

No other widely-reported issues on this pair. Report problems via the submission form.