How much VRAM does KAT-Coder V2.5 Dev need?

About 48 GB — the minimum this recipe targets.

How hard is this setup?

Advanced — follow the steps above.

KAT-Coder V2.5 Dev on Apple M2 Max: 35B Agentic Coding via llama.cpp Metal (64GB Unified Memory)

What You'll Build

A private, local agentic-coding backend: KAT-Coder-V2.5-Dev — Kwaipilot's Apache-2.0, 35B-total / 3B-active Mixture-of-Experts coding model — served as an OpenAI-compatible endpoint on an Apple M2 Max, with no NVIDIA GPU, no CUDA and no FlashAttention wheel anywhere in the stack. The lead path is llama.cpp with the Metal backend running the community Q6_K GGUF (27.99 GiB); a fully-documented MLX alternative is in Installation → Option B. The M2 Max's 64 GB unified pool is the roomiest Apple tier for this model: Q6_K leaves plenty of room for a long context, and Q8_0 (34.38 GiB) still fits with headroom.

Hardware data: Apple M2 Max (64 GB unified memory) · KAT-Coder-V2.5-Dev Q6_K GGUF 27.99 GiB, or MLX 6-bit-XL 28.17 GiB · comfortable fit, long context · See benchmark data

ℹ️ Unified memory is not VRAM. The M2 Max has 64 GB of unified memory shared by the OS, CPU and GPU — it is not 64 GB of dedicated VRAM. macOS lets the GPU address roughly 75% of it by default (~48 GB via Metal's recommendedMaxWorkingSetSize on a ≥ 64 GB Mac). Every fit statement below is against that ~48 GB addressable pool, not the raw 64 GB. Q6_K (27.99 GiB) plus a 128K KV cache lands around 30.5 GiB — comfortable. Q8_0 (34.38 GiB) plus the same cache lands near 37 GiB — still inside the pool, but see the wired-limit note in Troubleshooting before pushing it with a very long context. Full BF16 (64.61 GiB) does not fit at all: it exceeds the entire machine, never mind the addressable share.

ℹ️ This is a text-only release — there is no vision path in this recipe. The model card states that the open-weight release ships only the language-model weights and that the vision/multimodal components are not included. The vision_config block still present in config.json is vestigial: the safetensors index lists 31,333 tensors and zero vision tensors. Kwaipilot's own liushiyang (a Kwaipilot org member) confirmed it in discussion #3: "This checkpoint targets text-only code agent tasks, and we have not conducted any training for vision capabilities." Community users in that same thread have experimented with borrowing an mmproj file from the base model; the vendor explicitly does not recommend this model for vision workloads, so this recipe documents no vision path.

ℹ️ An MoE keeps every expert resident — the file size is the memory cost. KAT-Coder-V2.5-Dev routes 8 of 256 experts per token (about 3B active parameters), but all experts must be loaded, so the footprint is the whole quant file — 27.99 GiB at Q6_K, not some smaller "active" fraction. The upside of the sparse routing is throughput, not memory.

⚠️ No Ollama entry, and no first-party GGUF. ollama.com/library/kat-coder returns HTTP 404 — there is no Ollama library model for this release, and this recipe makes no Ollama claim. Kwaipilot ships no GGUF of its own either; the model card documents only SGLang, vLLM, KTransformers and Transformers, all of which target multi-GPU datacenter serving (its example commands use --tp-size 8 / --tensor-parallel-size 8). The GGUF and MLX builds this recipe uses are community conversions, linked below.

Requirements

Component	Minimum	Tested
GPU / memory	64 GB unified memory for a comfortable Q6_K or Q8_0; 48 GB is the tight floor (see below)	Apple M2 Max (38-core GPU, 64 GB unified memory)
RAM	Same pool — unified memory	64 GB unified
Storage	~32 GB (the Q6_K GGUF is 27.99 GiB)	27.99 GiB Q6_K, or 34.38 GiB Q8_0
Software	llama.cpp b10087+ (Metal, the macOS default), or `mlx-lm`; Python 3.10+	llama.cpp `llama-server`; `mlx_lm.server`

Where min_vram_gb: 48 comes from, and why 48 GB is tight rather than comfortable. Q6_K weights (27.99 GiB) plus a 128K KV cache (~2.5 GiB at fp16 — derived below) is a ~30.5 GiB working set. Macs below 64 GB expose only about 66% of unified memory to the GPU, so ~30.5 GiB needs roughly 46 GB of unified memory, and 48 GB is the next real configuration — hence the frontmatter floor.

Be honest about what that margin actually is, though. A 48 GB Mac's default addressable share works out around 32 GB, and the ~30.5 GiB working set is ~32.75 GB in the decimal units that figure is usually quoted in — i.e. it lands on the default ceiling, not under it. There is also no published recommendedMaxWorkingSetSize reading for a 48 GB Mac to pin that ~32 GB down precisely; it is extrapolated from smaller machines. So on a 48 GB Mac, plan on raising the GPU wired-memory limit (Troubleshooting gives a 48 GB-appropriate value) and on running a shorter context than the 128K used here — or drop to Q5_K_M (23.30 GiB), which leaves a real margin. This recipe is written and sized for the 64 GB M2 Max, where Q6_K sits ~18 GiB under the default ceiling and needs no tuning at all.

Per-quant sizes, from the file tree of the bartowski GGUF repo. Byte counts converted to GiB; the repo's own README table states the same files in decimal GB, which is why its numbers read about 7% higher:

Quant	Size (GiB)	Size as listed on the card (GB)	Fit on the ~48 GB addressable pool
bf16	64.61	69.38	Does not fit (exceeds the whole machine)
Q8_0	34.38	36.91	Fits with headroom — maximum fidelity
Q6_K	27.99	30.05	Recommended lead — comfortable, long context
Q5_K_M	23.30	25.02	Fits easily
Q4_K_M	19.92	21.39	Fits easily — pick this only if you also want the maximum context
IQ4_XS	17.51	18.81	Fits easily
Q3_K_M	15.11	16.23	Fits, but well below what this machine can afford
Q2_K	11.75	12.62	Fits, not worth the quality loss on a 64 GB Mac

Why the KV cache is small here. config.json — every field named here lives under its text_config object, not at the top level — shows 40 layers whose layer_types are 30 linear_attention and only 10 full_attention (full_attention_interval: 4), with num_key_value_heads: 2 and head_dim: 256. Only the 10 full-attention layers hold a context-length-scaling KV cache, so at fp16 that is 10 × 2 × 256 × 2 bytes × 2 (K and V) = 20 KiB per token: ~1.25 GiB at 64K, ~2.5 GiB at 128K, ~5 GiB at the full 262,144-token context. Quantizing the cache to q8_0 halves those. This is a derived envelope from the published config, not a measured peak — see /check/kat-coder-v2-5-dev/m2-max for measured data as it lands.

Installation

1. Install llama.cpp with Metal

There is nothing CUDA-shaped to install on Apple Silicon — no CUDA toolkit, no cu12x wheel index, no flash-attn build, no FP8 or NVFP4 path, no bitsandbytes, no GPTQ/AWQ/Marlin/ExLlamaV2. llama.cpp's Metal backend runs the model on the Apple GPU and is on by default: "On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU." (llama.cpp build docs).

# Homebrew ships a Metal-enabled build on macOS
brew install llama.cpp

# …or build from source (Metal is on by default on macOS)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

Use a recent build. This model's qwen35moe architecture — hybrid linear + full attention over a 256-expert MoE — is registered in llama.cpp's architecture table in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE, and the GGUFs below were quantized with release b10087 (2026-07-22). Anything older than that is untested for these files; see Troubleshooting if the output comes out as uniform gibberish.

2. Download the current chat template (do not skip this)

Kwaipilot patched the chat template on 2026-07-25 in commit 3a7d8740, titled "Fix chat_template: render non-leading system messages instead of raising, to unblock tool-call parser generation (e.g. Claude Code)". The GGUFs below were uploaded on 2026-07-23/24 — before that fix — so the template baked into the GGUF still raises an exception on a non-leading system message, which is exactly what agent clients send. Fetch the current one and pass it explicitly:

curl -L -o kat-chat-template.jinja \
  https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev/raw/main/chat_template.jinja

3. Download the Q6_K GGUF

llama-server can pull the GGUF straight from Hugging Face and cache it. The -hf flag takes <user>/<model>[:quant] per the canonical llama.cpp flag definitions; name the quant explicitly, because the flag's own default is Q4_K_M:

# Downloads Kwaipilot_KAT-Coder-V2.5-Dev-Q6_K.gguf (27.99 GiB) on first use
llama-server -hf bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF:Q6_K --port 8000

The bartowski repo is an imatrix quantization of Kwaipilot/KAT-Coder-V2.5-Dev and declares that upstream as its base_model. If you prefer an explicit download, grab the file from the files tree and pass -m <path> instead. On this 64 GB Mac :Q8_0 (34.38 GiB) is a comfortable maximum-fidelity option; :Q4_K_M (19.92 GiB) is the choice if you want the absolute longest context.

Option B — the MLX path

MLX is Apple's own array framework and is the other native Apple Silicon runtime. Stock mlx-lm implements this architecture directly — the canonical repo ships mlx_lm/models/qwen3_5_moe.py — and two community conversions of these exact weights exist, both declaring Kwaipilot/KAT-Coder-V2.5-Dev as their base_model:

leonsarmiento/KAT-Coder-V2.5-Dev-6bit-XL-mlx — mixed 6/8-bit with bf16 router and lm_head, 28.17 GiB of safetensors across 6 shards (the card describes it as ~6.9 bits per weight). Its bundled chat_template.jinja already carries Kwaipilot's 2026-07-25 fix. This is the MLX build to prefer on a 64 GB Mac. Licence caveat: this repo declares no licence in its Hugging Face metadata (no license field, no license:* tag) while its README's own overview table states Apache-2.0 — the two disagree, so treat the redistribution's licence as unconfirmed. The upstream Kwaipilot weights are Apache-2.0 either way; if licence provenance on the artifact you download matters to you, use the GGUF or OptiQ path below, both of which declare Apache-2.0 explicitly.
mlx-community/KAT-Coder-V2.5-Dev-OptiQ-4bit — Apache-2.0 (declared in its HF metadata), mixed 4/8-bit at ~4.5 bits per weight, 20.42 GiB of safetensors. Its card documents serving through the third-party optiq package in addition to stock mlx-lm, and its bundled chat template is the pre-fix one, so pass the template you downloaded in step 2 if you use it with an agent client.

pip install -U mlx-lm

# One-shot generation. Flags per the mlx-lm CLI: --temp (not --temperature), --top-p, --top-k
mlx_lm.generate \
    --model leonsarmiento/KAT-Coder-V2.5-Dev-6bit-XL-mlx \
    --max-tokens 512 \
    --temp 1.0 --top-p 0.95 --top-k 20 \
    --prompt "Implement an LRU cache in Python with O(1) get and put."

# …or an OpenAI-compatible server on the same weights
mlx_lm.server --model leonsarmiento/KAT-Coder-V2.5-Dev-6bit-XL-mlx --port 8000

At 28.17 GiB the 6-bit-XL build lands in the same memory envelope as the Q6_K GGUF (27.99 GiB), so the fit arithmetic above applies unchanged to either runtime. Note that the leonsarmiento card's own quickstart passes --temperature, which the mlx-lm CLI does not accept — the flag is --temp, as used above.

Running

1. Serve the model with Metal, a long context and a quantized KV cache

Every flag below is defined in the canonical llama.cpp flag definitions (-ngl/--gpu-layers, -c/--ctx-size, -fa/--flash-attn which takes on|off|auto, -ctk/--cache-type-k, -ctv/--cache-type-v, --jinja, --chat-template-file). Sampling values are the model card's recommended thinking-mode settings (temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5):

llama-server \
    -hf bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF:Q6_K \
    --port 8000 \
    -ngl 99 \
    -c 131072 \
    -fa on \
    -ctk q8_0 -ctv q8_0 \
    --jinja --chat-template-file kat-chat-template.jinja \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5

-ngl 99 offloads all layers to the Apple GPU through Metal. There are no CUDA semantics involved and no -DGGML_CUDA=ON to set.
-c 131072 asks for a 128K context. That costs about 1.25 GiB of KV cache with -ctk q8_0 -ctv q8_0 (half of the ~2.5 GiB fp16 figure derived in Requirements) on top of the 27.99 GiB of weights — well inside the ~48 GB addressable pool. The model natively supports 262,144 tokens; raise -c toward that if you need it, and watch Activity Monitor's Memory-Pressure gauge as you climb.
--jinja --chat-template-file is what makes agent clients work. Skipping it is the single most likely cause of a broken tool-calling session — see Troubleshooting.
--presence-penalty 1.5 is the vendor's recommendation, and it is genuinely contested by users — see Troubleshooting.

This exposes an OpenAI-compatible API at http://localhost:8000/v1.

2. Talk to it

This is the card's Text-Only Input example with its thinking-mode sampling values unchanged — only the base URL, the prompt and the final print differ:

pip install -U openai
export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY="EMPTY"

from openai import OpenAI
# Configured by environment variables
client = OpenAI()

messages = [
    {"role": "user", "content": "Write a Python function that returns the n-th Fibonacci number."},
]

chat_response = client.chat.completions.create(
    model="Kwaipilot/KAT-Coder-V2.5-Dev",
    messages=messages,
    max_tokens=81920,
    temperature=1.0,
    top_p=0.95,
    presence_penalty=1.5,
    extra_body={"top_k": 20},
)
print(chat_response.choices[0].message.content)

The model thinks before it answers: each turn opens with a <think> block. To get direct answers, the card documents extra_body={"chat_template_kwargs": {"enable_thinking": False}} with temperature 0.7 / top_p 0.8; for multi-turn agent work it documents {"preserve_thinking": True}, which keeps earlier reasoning traces in context.

3. Wiring an agent

Any coding agent that speaks the OpenAI API can point at http://localhost:8000/v1. Kwaipilot's own published evaluations were run under the Claude Code and Terminus-2 harnesses, and community users in discussion #9 report driving it from opencode. Whichever client you use, serve with the current chat template (Installation step 2) — the agent-side failures reported on this model all trace back to the template, not to the Metal runtime.

Results

Speed: Omitted deliberately. /check/kat-coder-v2-5-dev/m2-max has no benchmark data, and no published tok/s measurement for this model on an M2 Max exists — so quoting a number here would mean inventing one or importing it from unrelated hardware. Token generation on Apple Silicon is memory-bandwidth-bound (the M2 Max runs at ~400 GB/s), so figures from a different Apple chip do not transfer. If you measure it, please submit the run via /contribute and it will replace this paragraph with a real number.
Memory usage: The Q6_K weights are 27.99 GiB and stay resident (all 256 experts — see the MoE note above), leaving roughly 20 GiB of the ~48 GB addressable pool for the KV cache and activations; a 128K context takes about 2.5 GiB of that at fp16, or 1.25 GiB with the cache quantized to q8_0. Q8_0 at 34.38 GiB also fits, with roughly 13 GiB left over. bf16 at 64.61 GiB does not fit on any 64 GB Mac. All file sizes are byte counts from the GGUF file tree; the KV figures are derived from the published config.json, not measured.
Quality notes: Kwaipilot reports SWE-bench Verified 69.40, SWE-bench Multilingual 63.00, SWE-bench Pro 45.96 and Terminal-Bench 2.1 41.02 on the model card. Those are the vendor's own in-house reproductions under vLLM or SGLang — not independent measurements, and not measured on this hardware or at Q6_K. The card names no precision for them; the released checkpoint is BF16 (text_config.dtype: bfloat16 in config.json), so that is the precision they were almost certainly run at, but the card does not say so. Treat them as a capability claim, not a benchmark of what you will run. Q6_K is a near-lossless tier, so quality loss versus the vendor's BF16 setup should be small, but nobody has published a quantization-quality study for this model.

For the full benchmark data, see /check/kat-coder-v2-5-dev/m2-max.

Troubleshooting

Agent client fails with "Unable to generate parser for this template" / "System message must be at the beginning"

This is the most common failure on this model, and it is a template bug, not an Apple one. A user reported it against Claude Code in discussion #5; AIGC-1024, a Kwaipilot org member, replied there with the exact edit — replace this line in chat_template.jinja:

{{- raise_exception('System message must be at the beginning.') }}

with:

{{- '<|im_start|>system\n' + content + '<|im_end|>' + '\n' }}

Kwaipilot then applied that change upstream on 2026-07-25 in commit 3a7d8740, so the current chat_template.jinja in the canonical repo is already fixed — but the GGUFs (uploaded 2026-07-23/24) and the mlx-community OptiQ build (2026-07-24) still embed the pre-fix template. Downloading the current template and passing --jinja --chat-template-file as shown in Installation step 2 and Running step 1 is the fix. The leonsarmiento MLX build already ships the fixed template.

The model falls into loops or repeats itself

A user reported looping output on the Q8_0 GGUF in discussion #8, running llama-server at a 250,000-token context with --presence-penalty 0 --repeat-penalty 1.1 — i.e. with the vendor's recommended presence penalty of 1.5 turned off. Another user in the same thread suggested swapping in a corrected Qwen-family chat template via --chat-template-file. This is a community report with no vendor response, so treat it as a lead rather than a diagnosis; the cheap things to try first are keeping --presence-penalty 1.5, using the current chat template, and running a shorter context than a quarter-million tokens.

Note that the vendor's recommended sampling is itself disputed: in discussion #9 one user reports that --presence-penalty 1.5 "generates bad code" while others report the card's defaults working well. Both positions are community reports. Start with the card's values and change one at a time.

Output is uniform gibberish — every token identical

Update llama.cpp before anything else. A community bug report, llama.cpp issue #25857 (open, no maintainer response as of writing), documents the qwen35moe architecture producing uniform logits — every token at the same log-probability — for the base Qwen3.6-35B-A3B model on build b10066, on both GPU and pure CPU. That is a different model and an older build than the one these GGUFs were quantized with (b10087), and multiple users report working llama.cpp runs of the KAT-Coder GGUFs in discussion #9. Architecture support for this model family is recent, so pull a current build before filing anything.

Loading Q8_0 or a very long context stalls or falls back to a slow path

macOS caps how much unified memory the GPU may wire — roughly 75% of the 64 GB, or ~48 GB. Q6_K at 27.99 GiB is far under that, but Q8_0 at 34.38 GiB with a large KV cache can approach it. Raise the limit (macOS Sonoma 14 / Sequoia 15+):

# On the 64 GB M2 Max this recipe targets
sudo sysctl iogpu.wired_limit_mb=57344   # 56 GB; leaves ~8 GB for macOS

On a 48 GB Mac, do not use the value above — 56 GB exceeds that machine's total memory. Use a value scaled to the machine instead:

# On a 48 GB Mac
sudo sysctl iogpu.wired_limit_mb=40960   # 40 GB; leaves ~8 GB for macOS

Always leave 8–16 GB of headroom for macOS; pushing toward 100% causes swapping and instability. The general form is total unified memory minus 8–16 GB, expressed in MB. The setting is temporary and resets on reboot — persist it via /etc/sysctl.conf if you want it to survive a restart, and sudo sysctl iogpu.wired_limit_mb=0 restores the default. On macOS Monterey 12 / Ventura 13 the knob is sudo sysctl debug.iogpu.wired_limit=<bytes> instead. Watch Activity Monitor's Memory-Pressure gauge while the model loads.

A guide told me to install CUDA, flash-attn, bitsandbytes, vLLM or SGLang

None of those are the Apple path. There is no CUDA on macOS, no FP8 or NVFP4 tensor cores, no FlashAttention wheel, no bitsandbytes GPU kernels, no GPTQ/AWQ/Marlin/ExLlamaV2 and no ROCm — llama.cpp uses its own Metal attention kernels (-fa on) with GGUF K-quants, and MLX uses its own. The vLLM and SGLang commands on the model card are real, but they are eight-GPU datacenter serving recipes (--tensor-parallel-size 8), not something to run on a Mac. If a tutorial tells you to pass -DGGML_CUDA=ON, pip install flash-attn, or --load-in-4bit, skip that step entirely.

Looking for `ollama run kat-coder`?

It does not exist. ollama.com/library/kat-coder returns HTTP 404, and Kwaipilot publishes no GGUF of its own — so there is no Ollama tag to pull and this recipe does not claim one. Use the llama.cpp or MLX paths above. If an Ollama library entry appears later, please flag it through /contribute so this recipe can be updated.