KAT-Coder V2.5 Dev on RTX 3090: 35B-A3B Agentic Coding at 128K Context via llama.cpp

What You'll Build

A private, local agentic-coding endpoint: KAT-Coder-V2.5-Dev — Kwaipilot's Apache-2.0 Mixture-of-Experts coding model (35B total parameters, 3B activated) — served as an OpenAI-compatible API by llama.cpp on a single 24 GiB RTX 3090, ready for any OpenAI-compatible coding agent such as opencode. The recipe uses the community Q4_K_M GGUF and, unusually for a 35B-class model on a 24 GiB card, runs it at a 128K-token context window rather than the 8–16K you would expect — because this architecture keeps a growing KV cache on only a quarter of its layers.

Hardware data: RTX 3090 (24 GiB VRAM) · Q4_K_M GGUF, 19.92 GiB of weights · 128K context with 8-bit KV cache · See benchmark data

ℹ️ An MoE keeps every expert resident — the file size is the VRAM cost. KAT-Coder-V2.5-Dev activates 3B parameters per token out of 35B, but all 256 experts stay loaded, so the memory footprint is the whole quant file (19.92 GiB at Q4_K_M) and not some smaller "active" fraction. A low active-parameter count buys throughput, not VRAM.

⚠️ This is a text-only release — there is no vision path. The model card states that "the vision/multimodal components are not included and are unavailable." Kwaipilot's liushiyang confirms in discussion #3: "This checkpoint targets text-only code agent tasks, and we have not conducted any training for vision capabilities." Do not look for an mmproj file for this repo — the checkpoint ships none (the safetensors index lists 31,333 tensors, zero of them vision), and bartowski removed the two mmproj files he had briefly published on 2026-07-24. You may see community comments on that same discussion about grafting a base-model Qwen mmproj onto these GGUFs; that is unsupported, out of scope for this recipe, and not something Kwaipilot endorses.

⚠️ No official GGUF, and no Ollama library entry. Kwaipilot publishes only bf16 safetensors and documents only SGLang, vLLM, KTransformers and Transformers. There is no ollama.com/library entry for this model (https://ollama.com/library/kat-coder returns HTTP 404, checked 2026-07-29). The GGUF this recipe uses is community-published by bartowski, quantized with llama.cpp release b10087. bartowski's card lists several GGUF front-ends (LM Studio, koboldcpp, Jan AI and others) alongside llama.cpp, with the standing caveat "Note: if it's a newly supported model, you may need to wait for an update from the developers." — this recipe therefore pins llama.cpp itself, the runtime that produced the quants and whose master branch has a first-class implementation of this architecture.

Requirements

Component	Minimum	Tested
GPU	24 GiB VRAM (Q4_K_M is 19.92 GiB before any KV cache)	RTX 3090 (24 GiB, Ampere GA102, sm_86)
RAM	16 GB system RAM	—
Storage	~22 GB free for the Q4_K_M GGUF	19.92 GiB = 21.39 GB (Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf)
Software	llama.cpp CUDA build (b10087 or newer); huggingface-cli to download	llama.cpp llama-server

Why the GGUF path, and not vLLM or SGLang

The model card documents SGLang (≥ 0.5.10), vLLM (≥ 0.19.0), KTransformers and Transformers — but every launch example it gives uses --tp-size 8 / --tensor-parallel-size 8, i.e. eight GPUs. The unquantized checkpoint is 64.56 GiB of safetensors across the canonical file tree, so it cannot be held on one 24 GiB card in bf16. On a single RTX 3090 the quantized GGUF is the only path that fits, and Ampere's lack of FP8 tensor cores is irrelevant here because GGUF K-quants are integer formats.

Picking a quant for 24 GiB

Per-file sizes below are the byte counts from the bartowski GGUF file tree. bartowski's own table lists decimal GB; the GiB column is what nvidia-smi and llama.cpp will report.

Quant	Size (GB, decimal)	Size (GiB)	Fits 24 GiB?
Q5_K_M	25.02	23.30	No — no room left for KV cache
Q4_K_L	21.77	20.27	Yes, tight
Q4_K_M	21.39	19.92	Yes — this recipe's pick
Q4_K_S	20.59	19.18	Yes
IQ4_XS	18.81	17.51	Yes, roomiest 4-bit tier
Q3_K_M	16.23	15.11	Yes, quality drops
Q2_K	12.62	11.75	Yes, quality drops sharply

Installation

1. Install llama.cpp with CUDA support

The RTX 3090 is Ampere (GA102, sm_86) and needs nothing beyond a standard CUDA build — no FP8 path, no separate flash-attn package (llama.cpp ships its own attention kernels). You must end up with a CUDA-enabled build, which narrows the options: llama.cpp's docs/install.md lists five pre-built distributions, and its conda-forge section is the only one that names a backend at all — listing CUDA for Windows and Linux. The Homebrew, Winget, MacPorts and Nix sections make no backend claim whatsoever, so do not assume brew install llama.cpp gives you CUDA.

# conda-forge — the only pre-built channel llama.cpp documents as CUDA-enabled
conda install -c conda-forge llama.cpp

# …or build from source with CUDA (always explicit, always correct)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Whichever route you take, confirm CUDA is actually active before continuing — llama-server prints its backend and the number of offloaded layers at startup. A CPU or Vulkan build will "work" and then run at a small fraction of the speed.

Use b10087 or newer — that is the release bartowski quantized against, per the GGUF model card. llama.cpp master registers this model as architecture qwen35moe and gives it a dedicated hybrid-memory implementation in src/llama-arch.cpp — that hybrid classification is what makes the 128K context below affordable (see Running).

2. Download the Q4_K_M GGUF

The download command is given verbatim on the bartowski GGUF model card:

pip install -U "huggingface_hub[cli]"

huggingface-cli download bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF --include "Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf" --local-dir ./

Alternatively llama-server can pull the file itself — the -hf flag takes <user>/<model>[:quant] per the canonical llama.cpp flag definitions:

llama-server -hf bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF:Q4_K_M

3. Fetch the current chat template — required, not optional

The GGUFs you just downloaded embed a chat template that Kwaipilot has since fixed, and it will break agent harnesses. The bartowski quants were uploaded on 2026-07-23; Kwaipilot patched chat_template.jinja two days later, on 2026-07-25, in commit 3a7d874090 ("Fix chat_template: render non-leading system messages instead of raising"). The pre-fix template raises a Jinja exception whenever a system message is not the first message — you can confirm it is still baked into the published GGUF from the template Hugging Face parses out of the file itself on the GGUF repo page. Coding agents routinely send a system message mid-conversation, so with the embedded template they fail with Unable to generate parser for this template / System message must be at the beginning.

The template is a separate file — it is not part of the GGUF download. Grab the fixed one from the canonical repo:

curl -L -O https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev/resolve/main/chat_template.jinja

Every llama-server command below passes it with --chat-template-file, which overrides the template embedded in the GGUF (flag definition). Provenance: Kwaipilot's AIGC-1024 published this exact one-line change in discussion #5 in response to a user hitting the failure, and it subsequently landed upstream as the commit above — so the file you download already contains it. If a redistributor later re-quantizes against a post-2026-07-25 snapshot, this step becomes redundant but stays harmless.

4. Install a coding agent (optional)

Any OpenAI-compatible agent works. opencode is the one this recipe configures below; the opencode provider docs publish a ready-made llama.cpp local-provider block.

Running

1. Serve KAT-Coder with a 128K context

llama-server \
    -m Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf \
    --host 127.0.0.1 --port 8080 \
    --jinja --chat-template-file chat_template.jinja \
    -ngl 99 \
    -c 131072 \
    -fa on \
    -ctk q8_0 -ctv q8_0 \
    --temp 1.0 --top-k 20 --top-p 0.95 --presence-penalty 1.5

Every flag above is defined in the canonical llama.cpp flag definitions (-ngl/--gpu-layers, -c/--ctx-size, -fa/--flash-attn which takes [on|off|auto], -ctk/--cache-type-k, -ctv/--cache-type-v, --jinja, --chat-template-file). The sampling values — temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5 — are the ones the model card uses in its own thinking-mode example. --jinja makes llama-server apply a Jinja chat template at all — which is what emits the <tool_call> blocks agents depend on — and --chat-template-file points it at the fixed template from step 3 instead of the stale one baked into the GGUF.

This exposes an OpenAI-compatible API at http://127.0.0.1:8080/v1, plus llama.cpp's built-in web UI at http://127.0.0.1:8080 for a quick smoke test.

Why 128K fits where a normal 35B would not

Most 30B-class models on a 24 GiB card are forced down to an 8–16K context, because a dense KV cache on every layer eats the few GB the weights leave behind. This model is different, and the reason is architectural rather than a matter of the weights being smaller.

In KAT-Coder-V2.5-Dev's config.json, every field below lives under the nested text_config object (the top level holds only architectures, model_type, text_config, vision_config and the token ids). text_config declares num_hidden_layers: 40 with full_attention_interval: 4, and its layer_types array spells the pattern out: 10 of the 40 layers are full_attention; the other 30 are linear_attention. Only the full-attention layers keep a KV cache that grows with context. llama.cpp models this directly — llm_arch_is_hybrid() in src/llama-arch.cpp returns true for LLM_ARCH_QWEN35MOE, and src/llama-model.cpp builds a llama_memory_hybrid whose filter_attn admits only the non-recurrent layers. The 30 linear layers get a fixed-size recurrent state instead, which does not grow as the conversation gets longer.

That makes the KV budget a derived envelope you can compute from the same text_config — num_key_value_heads: 2, head_dim: 256, so each token costs 2 × 256 × 2 (K and V) = 1024 values per full-attention layer:

• fp16 cache: 1024 × 2 bytes × 10 layers = 20 KiB per token → ~2.5 GiB at 131,072 tokens, ~5.0 GiB at the full 262,144.
• q8_0 cache (the -ctk q8_0 -ctv q8_0 above): roughly half that → ~1.3 GiB at 131,072 tokens, ~2.7 GiB at 262,144.

So the recipe's envelope is 19.92 GiB of weights + ~1.3 GiB of KV + compute buffers ≈ 22 GiB of the card's 24 GiB. These are derived figures, not a measurement — /check/kat-coder-v2-5-dev/rtx-3090 currently has no benchmark data, and a community measurement submitted via /contribute would replace this arithmetic. Community reports on other hardware are consistent with it: one user writes on discussion #12: "Happy to get this on my old RTX3060 with 260k context window." — and the llama-server command posted on discussion #8 runs -c 250000. Both are far past what a dense 35B would allow, though neither is a measurement on an RTX 3090.

You can push -c 262144 (the model's native maximum, per text_config.max_position_embeddings), but that leaves under ~1.5 GiB of slack on a 24 GiB card before compute buffers — verify with nvidia-smi under a real workload before relying on it.

2. Point opencode at the local server

Adapted from the llama.cpp provider block in the opencode provider docs, with the model id and context limit changed to match the server above. Per the opencode config docs, save it as opencode.json in your project root or as ~/.config/opencode/opencode.json for a user-wide setting:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "kat-coder-v2.5-dev": {
          "name": "KAT-Coder-V2.5-Dev (local)",
          "limit": {
            "context": 131072,
            "output": 65536
          }
        }
      }
    }
  }
}

3. Turning off thinking

The model reasons inside <think> blocks by default. Per-request, the model card documents passing chat_template_kwargs: {"enable_thinking": False} in extra_body, together with temperature 0.7 / top_p 0.8. Server-side, set it once with -rea/--reasoning, which takes [on|off|auto] per the canonical flag list:

llama-server -m Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf \
    --jinja --chat-template-file chat_template.jinja \
    --reasoning off \
    -ngl 99 -c 131072 -fa on -ctk q8_0 -ctv q8_0

Do not reach for --chat-template-kwargs '{"enable_thinking": false}' here: the same file special-cases that key and logs "Setting 'enable_thinking' via --chat-template-kwargs is deprecated." on startup, pointing the reader at --reasoning on / --reasoning off instead. --reasoning off sets the identical template variable internally.

The card also documents a preserve_thinking option that keeps reasoning traces from earlier turns — useful for agents, at the cost of more context per turn. That one has no deprecation and no dedicated flag, so pass it through the kwargs route (the deprecation warning fires only for the enable_thinking key): --chat-template-kwargs '{"preserve_thinking": true}'. Note that llama.cpp's superficially-similar --reasoning-preserve sets a preserve_reasoning variable, which is not the name this model's template reads.

Results

• Speed: Omitted. No throughput measurement on an RTX 3090 exists for this model — /check/kat-coder-v2-5-dev/rtx-3090 has no benchmark data, and the only community throughput figures on the model card discussions are from other vendors' cards. If you run it, please submit your numbers via /contribute so this section can carry a real measurement. Note also that this is a reasoning model: much of each turn is <think> content you discard, so effective throughput is lower than a raw tok/s figure suggests.
• VRAM usage: Q4_K_M weights are 19.92 GiB (21,391,448,480 bytes, from the bartowski GGUF file tree) and must stay resident. With q8_0 KV at a 131,072-token context the derived total is ~22 GiB of the RTX 3090's 24 GiB — see the arithmetic in Running. Derived, not measured; /check/ is where a measured figure will land.
• Quality notes: Kwaipilot reports SWE-bench Verified 69.40, SWE-bench Multilingual 63.00, SWE-bench Pro 45.96 and Terminal-Bench 2.1 41.02 on the model card. These are the vendor's own in-house reproductions under their unified harness — not independent measurements, and not measured on this GPU; the card's footnotes give the exact agent versions and sampling settings, and discussion #12 is an open community thread comparing them against third-party leaderboard runs. Note also that the numbers above were produced from the bf16 checkpoint; this recipe runs a 4-bit quant.

For the full benchmark data, see /check/kat-coder-v2-5-dev/rtx-3090.

Troubleshooting

Out of memory at launch

The 19.92 GiB of weights leave roughly 4 GiB on a 24 GiB card, so an OOM at startup almost always means the KV cache is too large. Work down this ladder:

1. Keep -fa on and -ctk q8_0 -ctv q8_0 — dropping either roughly doubles the cache.
2. Lower -c (65536, then 32768). Each halving halves the KV cache.
3. Drop a quant tier: IQ4_XS is 17.51 GiB, which buys back ~2.4 GiB versus Q4_K_M for more context at slightly lower quality.
4. Offload some experts to system RAM with -ncmoe/--n-cpu-moe N (flag definition) — this trades throughput for VRAM and is how users on smaller cards run this model at all.

Watch nvidia-smi during a real agent task rather than at idle: a hard problem produces a long <think> block that grows the cache mid-generation, so size for the peak.

An agent harness fails with "System message must be at the beginning"

You skipped installation step 3. The template embedded in the bartowski GGUFs predates Kwaipilot's 2026-07-25 fix and raises a Jinja exception on any non-leading system message; the symptom in an agent harness is a 400 with Unable to generate parser for this template. Download the current chat_template.jinja from the canonical repo and pass it with --chat-template-file as step 3 shows — do not hand-edit anything, the upstream file already carries the fix.

The model falls into a repetitive loop

A user reports on discussion #8 that a Q8_0 run at a 250,000-token context looped; another suggests swapping in a corrected Qwen-family chat template via --chat-template-file. This is a community-reported workaround, not a vendor-confirmed fix. If you hit it, try the corrected template first, then a shorter context.

--presence-penalty 1.5 and code quality

The value comes straight from the model card's own example, but it is contested: on discussion #9 one user reports it degrades generated code while another defends it as the recommended default. If output quality looks off, try lowering it before changing anything else. No vendor statement resolves the disagreement.

llama.cpp reports no GPU, or layers stay on CPU

Confirm the build has CUDA enabled (-DGGML_CUDA=ON from source) and that -ngl 99 is offloading every layer — llama.cpp prints the offloaded-layer count at startup. The RTX 3090 needs no special flags: you do not need FP8 support (K-quants are integer) and you do not need to pip install flash-attn (llama.cpp's -fa on uses its own kernels).