Qwen3.6-35B-A3B on RTX 3060: 38.9 tok/s from a 12GB card via MoE expert CPU-offload

What You'll Build

A local, OpenAI-compatible coding assistant powered by Qwen3.6-35B-A3B — a 35-billion-parameter Mixture-of-Experts model with 3B parameters active per token — served by llama.cpp on a single 12GB RTX 3060. The attention path stays on the GPU; the routed expert tensors of the first 24 layers live in system RAM. A community benchmark on this exact pair measured 38.9 tok/s at 9.8GB peak VRAM.

Hardware data: RTX 3060 (12GB VRAM) · 38.9 tok/s at UD-Q4_K_M, -ncmoe 24, 9.8GB peak · See benchmark data

⚠️ This model does not fit 12GB on its own. The UD-Q4_K_M GGUF is 22.66GB on disk (HF tree API, 22,663,387,424 bytes) — nearly 2× the card's VRAM. It runs because it is a sparse MoE: llama.cpp keeps attention, the dense path and the shared expert on the GPU and pushes the routed expert FFN tensors of the first N layers to CPU RAM with -ncmoe N. unsloth's own rule is that your "total available memory (VRAM + system RAM) exceeds the size of the quantized model file" (Qwen3.6 docs) — with 22.66GB of weights and 12GB on the card, plan on 32GB+ of system RAM.

ℹ️ Text/coding path only. The catalogue lists this model as multimodal and that is correct — config.json carries a real 27-layer vision_config, and the GGUF repo ships a separate mmproj-F16.gguf projector. This recipe is llm because the install it documents is text-only, which is also what the benchmark backing it measured (task: llm, run under llama-bench, which has no image input). The Vision on this card section below records what is and is not known about the image path here.

Requirements

The measured run's host was an i7-7700 on PCIe 3.0 x16 with the card headless. Both of those matter — see the Results section.

Installation

1. Build llama.cpp with CUDA

These are the build steps from the unsloth Qwen3.6-35B-A3B-MTP-GGUF model card:

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

The benchmark on this pair was produced on b10088 (commit 67b9b0e, released 2026-07-22). Anything at or after that tag carries the MTP support and the -ncmoe behaviour this recipe relies on; older checkouts are the cause of the blk.40 load failure in Troubleshooting below.

2. Download the UD-Q4_K_M GGUF

pip install -U "huggingface_hub[cli]"
huggingface-cli download unsloth/Qwen3.6-35B-A3B-MTP-GGUF \
    Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir ./models

That is 22,663,387,424 bytes, and every command below points at ./models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf.

UD-Q4_K_M is the tier the benchmark records (Q4_K_M (Unsloth UD)) and it is not the file the RTX 4070 recipe leads with — that is UD-Q4_K_XL, 22,853,663,008 bytes, 0.19GB larger, in the same repo. One caveat the benchmark cannot settle: Q4_K_M (Unsloth UD) names a quant tier, not a repo, and unsloth publishes a second UD-Q4_K_M in the non-MTP Qwen3.6-35B-A3B-GGUF at 22,134,528,992 bytes — 0.53GB smaller, that difference being the blk.40 MTP head. This recipe leads with the MTP repo because it is what the catalogue slug and the RTX 4070 sibling both name, and because it lets you switch MTP on later; which of the two the submitter measured is not recorded.

⚠️ Do not use export LLAMA_CACHE="unsloth/Qwen3.6-35B-A3B-MTP-GGUF" as a download step. It appears on the model card, but LLAMA_CACHE is llama.cpp's cache directory — common/common.cpp reads it into cache_directory, falling back to XDG_CACHE_HOME/HOME. Setting it to a repo id downloads nothing; it just parks the cache in a folder named after the repo. The variable that takes a <user>/<model>[:quant] is LLAMA_ARG_HF_REPO, the env form of -hf.

Running

The measured configuration

This is the command the benchmark on /check records, with only the model path substituted for the one you downloaded above — every flag is as recorded:

./llama.cpp/llama-bench -m ./models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -ngl 99 -ncmoe 24 -fa 1 -p 512 -n 128 -d 0,4096,8192 -r 3

It returns 38.9 / 38.5 / 38.2 tok/s generation at 0 / 4K / 8K tokens of context, and 413 / 394 / 386 tok/s prompt processing, at a 9.8GB peak.

Serving it

./llama.cpp/llama-server \
    -m ./models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
    -ngl 99 -ncmoe 24 -fa on -np 1 -c 32768 \
    --jinja

⚠️ If you swap -m for -hf, add --no-mmproj. The -hf help text says "mmproj is also downloaded automatically if available. to disable, add --no-mmproj", llama-server is in the list of examples that honours it, and both no_mmproj (false) and mmproj_use_gpu (true) default that way in common/common.h. This repo does ship mmproj-F16.gguf, so -hf on its own puts 0.838 GiB of vision projector on a card this recipe has already budgeted to the last few hundred megabytes — and this is a text-only recipe, so it buys you nothing. The RTX 3060 owner in discussion #18 passes --no-mmproj for the same reason. llama-bench is not affected: it is not in that list, and the measured command uses -m anyway, so the 9.8GB peak is projector-free.

• -ngl 99 puts every layer on the GPU as the starting point.
• -ncmoe 24 then walks the routed experts back off it. Per llama.cpp's own help text the flag will "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp) — and the implementation matches the wording: it installs a CPU buffer override for blk.0 through blk.(N-1), so the offloaded layers are the first N, counting from zero.
• -fa on is flash attention, on in the measured run.
• -c 32768 is a deliberate step past what was measured — see the budget below.

unsloth's recommended sampler for precise coding tasks is temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0 with the repeat penalty disabled or set to 1.0 (Qwen3.6 docs, thinking mode, precise-coding column). That page's copyable snippet writes the last one as repetition_penalty, which is the transformers spelling; llama.cpp's flag is --repeat-penalty, and the same page's own table uses repeat_penalty. The server listens on http://localhost:8080 with an OpenAI-compatible API.

Choosing -ncmoe for your context

The 38.9 tok/s figure is measured at 8,832 tokens of context or less. llama-bench sizes its context as n_prompt + n_gen + n_depth (tools/llama-bench/llama-bench.cpp), so the deepest run above allocated 512 + 128 + 8192 = 8,832 tokens. Everything past that is arithmetic, not measurement.

The arithmetic is tractable because of the architecture. config.json gives layer_types with exactly 10 full_attention layers out of 40 (full_attention_interval: 4), num_key_value_heads: 2 and head_dim: 256 (config.json). So the KV cache costs

10 layers × 2 KV heads × 256 dim × 2 (K and V) × 2 bytes (f16) = 20,480 bytes per token

and the other 30 layers are Gated DeltaNet, whose recurrent state is a fixed size that does not grow with context. That is the whole reason a six-figure context is even discussable on a 12GB card.

Budget, taking the measured 9.8GB (9.127 GiB) peak as the base and adding only the extra KV:

Read that last row against the submitter's own failure: -ncmoe 20 sat at 11.7GB (10.897 GiB), left 1.103 GiB spare, and OOM'd once real context loaded. A 128K context at -ncmoe 24 lands below that margin, so treat it as out of reach at this setting. Two levers, and you can use either:

• Raise -ncmoe. The submitter's sweep prices one step at 0.46GB of GPU memory: -ncmoe 40 = 2.5GB, 32 = 6.1GB, 24 = 9.8GB, 20 = 11.7GB — 9.2GB across 20 steps, with pairwise slopes of 0.475, 0.463 and 0.45 GB per step. 0.46GB is 0.428 GiB, so fully offsetting the 128K KV delta of 2.332 GiB takes 2.332 ÷ 0.428 = 5.4 steps — round up and use -ncmoe 30, which puts the base at 6.557 GiB and the 128K total at 8.888 GiB, 3.112 GiB spare. Two or three steps is not enough. Two leaves 1.398 GiB spare and three leaves 1.826 GiB — both above the 1.103 GiB that OOM'd, but only just, and that margin still has to absorb the compute buffers this table excludes.
• Halve the KV. --cache-type-k q8_0 --cache-type-v q8_0 takes the per-token cost from 20,480 to 10,240 bytes, so a 128K cache costs exactly what a 64K f16 cache costs — 1.250 GiB. Keeping -ncmoe 24 and quantizing the cache lands 128K at 10.208 GiB, 1.792 GiB spare, i.e. the same place as the 65,536 row.
• Or both, which is the combination the 3060 owner cited below actually runs. The levers add; nothing here forces you to pick one.

A separate RTX 3060 12GB owner runs --n-cpu-moe 25 at --ctx-size 32768 with both caches at q8_0 in discussion #18 on the GGUF repo — one step above this recipe's setting, at the context this recipe recommends, which is the corroboration you would want for the 32,768 row.

None of these rows is a measurement, the table excludes llama.cpp's compute buffers and any vision projector (see the --no-mmproj note above — loading one costs a further 0.838 GiB), and it assumes a headless card. Nor is the sweep a fit claim for a smaller card: -ncmoe 32 at 6.1GB and 40 at 2.5GB were measured on this 12GB card as a throughput-for-headroom trade. Nothing on this page validates them as an install on an 8GB or 10GB card, and this recipe does not document one — a smaller card changes the compute buffers and the context you can afford, neither of which the sweep varied. That is why the floor stays at 12: it is the smallest capacity the install below is known to run on, not the smallest the weights might fit in. The benchmark's own caveat is that a 3060 also driving a display loses 0.5–1GB and needs 1–2 more on -ncmoe. Verify against nvidia-smi after load, and if you measure throughput past 8K on this card please send it via the submission form — nobody has.

-fit on is the wrong tool here

llama.cpp can size the offload itself: -fit on adjusts "whether to adjust unset arguments to fit in device memory" (common/arg.cpp), with -fitt setting the margin it holds back. That is what the RTX 4070 recipe uses.

Prefer the explicit -ncmoe on this page, for two reasons. First, it is what was measured here. Second, there is an open regression report against --fit-target on this exact quant tier — llama.cpp issue #27171, filed 2026-08-16 by a community reporter, records a considerable llama-bench throughput drop with Qwen3.6-35B-A3B Q4_K_M under --fit-target 1024. A contributor has a fix in flight (PR #27207), unmerged at the time of writing. The report is on a different card, so it is not a claim about the RTX 3060 — it is a reason to pin the offload by hand until the fix lands.

Note that auto-fit is on by default, so "don't pass -fit" is not the same as "don't use it". At the pinned commit common/common.h:468 declares bool fit_params = true, and line 473 defaults the per-device target to 1024 * 1024*1024 — 1024 MiB, exactly the margin #27171 reports against. llama-bench defaults it off, which is why the measured run above is fit-free while a plain llama-server command is not. To pin the offload yourself, set -ncmoe explicitly and turn the fitter off rather than merely omitting --fit-target.

MTP is optional on this card

The model ships a trained multi-token-prediction head, and llama.cpp will drive it with --spec-type draft-mtp --spec-draft-n-max 2. The measured configuration above does not use it, and on a card in this regime that is not obviously a loss:

• unsloth's own MTP benchmarks put the gain at "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs) — so 1.15–1.25× for this model, before offload is taken into account.
• The RTX 3060 12GB owner in discussion #18, running --n-cpu-moe 25, reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" with the MTP flags on. A community reply in the same thread offers the explanation — that CPU-MoE offload neuters the MTP gain — which nobody in the thread has measured either way.

Adding the flags costs nothing to try. Just do not budget for a speed-up you have not seen on your own machine.

Results

• Speed: 38.9 tok/s generation, and 413 tok/s prompt processing, at -ncmoe 24 on an empty context, per the benchmark data. Generation is flat through 8K — 38.9 / 38.5 / 38.2 tok/s at 0 / 4K / 8K. The submitter's note attributes the flatness to the per-token cost of streaming experts over DDR4 being constant, i.e. independent of context depth. Beyond 8K it is unmeasured on this card.
• VRAM usage: 9.8GB peak at -ncmoe 24, headless, at the contexts above. The offload sweep from the same run: -ncmoe 40 = 28.1 tok/s at 2.5GB · 32 = 31.8 at 6.1GB · 24 = 38.9 at 9.8GB · 20 = 42.6 at 11.7GB. The 42.6 figure is a ceiling you should not aim at — the submitter reports it OOMs once real context loads.
• Quality notes: this is the coding/agentic variant of Qwen3.6 (SWE-bench Verified 73.4 per the model card). unsloth's Dynamic UD quants keep sensitive tensors at higher precision than a flat K-quant of the same name.

Why this page says 38.9 and the RTX 4070 page says 80.8

Both cards hold 12GB, and the RTX 4070 benchmark records 80.8 tokens/s against this one's 38.9. Neither number is wrong; they are not measuring the same thing. Four differences, largest effect first:

1. The host platform, which is what actually feeds an offloaded MoE. With -ncmoe 24, 24 of 40 layers' routed experts are not on the GPU at all, so per-token throughput is set by the host side rather than by the card. The RTX 3060 run names its host — i7-7700, DDR4-2133 dual channel (~34 GB/s), PCIe 3.0 x16. The RTX 4070 row records none: its whole notes field reads Benchmark: code_python. Achieved using llama.cpp with MTP support and 128K context. Using -fitt 1536 to balance GPU/CPU load. and its Reddit source does not serve the thread body to an automated fetch, so the comparison cannot be closed from this side. The nearest evidence is adjacent rather than direct, so take it as such: an RTX 3060 12GB owner profiling this same model at this same quant tier with -ncmoe 26 found prefill "bottlenecked by PCIe transfers, not GPU compute" (llama.cpp issue #25859 — a community report, self-described as single-box and gen4-specific, with the author noting he measured it around the b6xxx era and has not re-run against current master; a contributor replied triaging it as a duplicate of PR #21067, and no maintainer has). That establishes a host-side bottleneck on the prompt-processing path for this pair; the 38.9-vs-80.8 numbers are generation, and nothing measured isolates the decode path on this card. One decode-side measurement does exist in that same thread, and it is worth reading precisely because it is not this pair: a commenter on 2026-08-08 reports experts resident in host RAM on an RTX 4090 running Qwen3.6-35B-A3B at IQ2 through a fork, and times the per-token expert copy at roughly 60 µs against a sub-10 µs kernel — the copy, not the compute, setting the pace. Different card, different quant, different build, so it transfers nothing numerically to a 3060 at Q4_K_M; what it does is show the same host-side mechanism dominating decode when someone measures it directly. The submitter's own attribution here names a nearer mechanism still — expert streaming over DDR4 rather than PCIe transfers.
2. Speculative decoding. The RTX 4070 row says it was "Achieved using llama.cpp with MTP support" — it does not name the flag, but the only way to drive that head is --spec-type draft-mtp, and the command on this page does not use it. By unsloth's own numbers that is worth 1.15–1.25× on a MoE, and the 3060 owner cited above measured no gain at all with expert offload in play. It cannot account for a 2× gap.
3. The measurement itself. This page's figure is llama-bench -n 128 — pure generation, empty-to-8K context, three repeats. The RTX 4070 figure is a code_python workload at a 128K context. Two different questions with the same unit on the answer.
4. The quant file. UD-Q4_K_M here, UD-Q4_K_XL there. 22.66GB against 22.85GB — 0.8% apart on disk, and the smallest of the four differences.

And a fifth item that is an unknown rather than an explanation, because a reader comparing the two commands will notice it first: the sibling page documents --n-cpu-moe 36 against this page's -ncmoe 24. Taken at face value that cuts against everything above — this page has just shown that more offload is slower, so a deeper offload reaching 80.8 tok/s would be harder to explain, not easier. It does not survive checking. The 36 is the sibling's own manual-offload suggestion, not the benchmark's configuration: that run used auto-fit (-fit on -fitt 1536), which chooses the depth at load time and records it nowhere, and the benchmark row carries no offload depth either. So the 4070's actual offload depth is simply not known, on either page.

If you own an RTX 3060, plan for 38.9 tok/s.

For the full benchmark data, see /check/qwen3-6-35b-a3b-mtp-ud-q4-k-xl-gguf/rtx-3060. Measured a different number on your own 3060? Send it through the submission form so the next reader gets your data.

Vision on this card

The GGUF repo ships mmproj-F16.gguf (899,283,584 bytes) beside the weights, so the projector exists as a separate artifact. Three things are known and one is not:

• unsloth's model card, as published today, states that -np > 1 and --mmproj "are not yet supported with MTP" (model card).
• A community user in discussion #25 on that repo reports loading the projector alongside MTP and disabling speculation per-request with speculative.n_max: 0, so vision requests do not crash while other requests keep MTP. It is contested in-thread — a second community member challenges whether that really leaves MTP on for other requests, and the opener defends it at length — and the thread is three comments from two people, both with isOrgMember false, so nobody from the unsloth org has weighed in either way.
• Reading llama.cpp's server sources at commit 1729ed5 turned up no hard block of that shape: the only guard where multimodal input meets speculative decoding rejects a draft model whose input embedding width differs from the target's, which is not what an MTP context is. That is a bounded source-read of one tree, not a claim that the combination works.
• What is not known: any of this on a 12GB RTX 3060. The projector's 0.90GB would come out of the same headroom the KV cache needs in the budget above, and no measurement of that exists on this pair. That is why this recipe stops at the text path. Run it and tell us.

Troubleshooting

missing tensor 'blk.40.ssm_conv1d.weight' at load

Almost always a llama.cpp that predates MTP support. Layer 40 is the model's MTP head (mtp_num_hidden_layers: 1 in config.json), so this arch has 41 blocks where a loader without MTP expects 40. Issue #23033 reports exactly this against build b9143, and a project contributor answers that the missing tensor is the MTP head rather than an SSM one, and points at the then-in-flight PR #22673, "llama + spec: MTP Support" — merged 2026-05-16, two months before the b10088 build this recipe pins. A later commenter reports that pulling master and recompiling fixed it for them.

Rebuild from b10088 or later first. Note the issue is still open, and one commenter reports the same message on a ROCm machine after that PR merged, so a rebuild is the first thing to try rather than a guaranteed fix.

unknown speculative type: mtp

The flag value is draft-mtp, not mtp — one thread on the GGUF repo is people hitting exactly this (discussion #7). llama-server prints the accepted list for your build when it rejects the value, and that list grows: don't copy one out of an old forum post. The names live in common_speculative_type_from_name_map in common/speculative.cpp.

Set your expectations before you spend the evening on it. Of the repo's 26 discussion threads, four are people reporting that MTP bought them nothing — #5, #11, #18 and #22 — against one for the flag-name error above. #18 is the RTX 3060 12GB owner cited twice on this page. That is the modal outcome on a card in this class, and it is consistent with the rest of this page: when the expert copies set the pace, drafting more tokens per step does not help, because the bottleneck is not the GPU waiting on itself.

Out of memory once a long prompt lands, but not at load

Expected if -ncmoe is too low. Loading only allocates the weights; the KV cache grows with the prompt at 20,480 bytes per token, so a configuration that loads cleanly at -ncmoe 20 (11.7GB) can still die at 32K. Raise -ncmoe by 2–3, or quantize the cache with --cache-type-k q8_0 --cache-type-v q8_0. If the card is also driving a display, start 1–2 higher than this recipe's 24 — that caveat is the submitter's own.

System thrashing or swapping, with VRAM to spare

Your system RAM is the limit, not the card. At -ncmoe 24 more than half of the 22.66GB file lives in RAM, on top of the OS. 32GB is the practical floor; 16GB is not enough.

Throughput well below 38.9 tok/s

Three things to check before suspecting the model. -ncmoe too high pushes more layers onto the CPU and the sweep above prices that at 0.89 tok/s per step between 24 and 32. Single-channel memory halves the bandwidth the offloaded experts are read at. And a --fit-target in the command line may be hitting issue #27171 — drop it and set -ncmoe explicitly.