Nanbeige4.2-3B on RTX 4060: 32K Context on 8 GB Despite the Looped-Transformer KV Tax

What You'll Build

A local llama-server endpoint running Nanbeige4.2-3B on a single RTX 4060, with a 32,768-token context window fully resident in the card's 8 GB. The interesting part of this build is not whether the weights fit — a 2.40 GiB Q4_K_M on an 8 GB card is not a close call — but how much context you can buy with what is left over, because this model spends VRAM on context at twice the rate its config file suggests.

Hardware data: RTX 4060 (8 GB VRAM) · Q4_K_M weights 2.398 GiB + q8_0 KV at 32,768 tokens 2.922 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.125 GiB = 5.762 GiB derived · no benchmark submitted yet · See benchmark data

⚠️ Known issue — the KV cache is double what config.json implies. config.json reports num_hidden_layers: 22, but it also sets num_loops: 2. Nanbeige4.2 is a Looped Transformer. The model card puts it this way: "Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters." The 22 blocks execute twice per forward pass, and each pass keeps its own keys and values. llama.cpp implements this by expanding the layer count before it allocates anything — src/models/nanbeige.cpp carries the comment "Share physical weights across loops; each slot still has its own KV index." So the weights are stored once and the KV cache is sized for 44 layers, not 22. Budget from 44 or you will OOM at exactly half the context you planned for.

The Nanbeige team has confirmed this is deliberate. In discussion #18, team member leran1995 writes: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2", adding in discussion #10 that "In future versions, we plan to mitigate the KV-cache overhead through linear and sparse attention mechanisms." On an 8 GB card, that overhead is the whole story.

Requirements

Component	Minimum	This recipe
GPU	8 GB VRAM, CUDA	RTX 4060 (8 GB) — not measured; the budget below is derived from file sizes and llama.cpp's cache formula (/contribute)
RAM	8 GB system RAM	— (16 GB+ if you use --no-kv-offload for 256K context)
Storage	2.57 GB for Q4_K_M	2,574,807,904 bytes, from the HF tree API
Software	llama.cpp built from master after 2026-07-27	Architecture support merged in PR #25994; confirmed present at release tag b10205

Do not follow the model card's llama.cpp instructions. The card still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release, but mainline merged native nanbeige support on 2026-07-27 and the fork is no longer needed. The upstream request tracking this, issue #26086, was closed the same day.

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Confirm your checkout actually contains the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q4_K_M GGUF

There is no first-party GGUF: the Nanbeige org publishes safetensors plus -FP8 and -GPTQ-Int8 builds only. This recipe uses owao/Nanbeige4.2-3B-GGUF, the most-downloaded community conversion.

pip install -U huggingface_hub numpy
hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q4_K_M.gguf \
  --local-dir ./models

3. Verify the artifact carries the loop parameter

This is the one check worth doing by hand. If a GGUF was produced without num_loops, llama.cpp defaults it to 1, runs 22 layers instead of 44, and you get a silently different model. Read the metadata with the dumper that ships in the clone you just made:

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q4_K_M.gguf | grep -E "num_loops|block_count"

You want two lines, ending = 22 and = 2 respectively:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case, because the loader defaults the key to 1 without complaining. (Numbering will differ; only the names and values matter.)

Two commands that look like they should work here do not, so don't substitute them: llama-gguf <file> r n prints key names but never their values, and llama-cli suppresses the loader's metadata dump entirely at its default verbosity — that dump is emitted at trace level, so it needs -lv 4 — besides which llama-cli is now an interactive chat client that will sit waiting for input rather than exiting.

Running

Start llama-server with a quantized KV cache and Flash Attention. Both matter: without -ctk/-ctv q8_0 this context does not fit, and quantized V-cache wants Flash Attention on.

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --flash-attn on \
  --temp 0.6 --top-p 0.95 --top-k 20

The sampler values are the model card's recommendation for reasoning and chat. For agentic and tool-use work the card recommends --temp 1.0 instead. Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

Leave --parallel alone. With no flag, llama-server sets its slot count to 4 — but the same code path also switches the KV cache to unified, which makes one conversation's window the full --ctx-size and allocates the cache once. That is what the numbers below assume. Pass an explicit positive --parallel N and unified mode is no longer implied: the cache is allocated per slot and each conversation gets ctx-size / N, so you would lose three quarters of the window here and the budget below stops applying. (Only a negative value keeps the auto path — the branch tests for n_parallel < 0.) If you genuinely want concurrent slots on 8 GB, add --kv-unified back by hand.

Results

• Speed: omitted — no measurement exists for this pair. The closest datapoint is on a different card: in discussion #18 community user thermi6 reports 63.07→53.67 t/s on a first prompt, falling to 16.74→16.00 t/s by a third prompt, measured on an RTX 5060 8 GB with q4_0 KV at 65,536 context. That card is a generation newer than the RTX 4060 with substantially faster memory, and decode here is memory-bound — so treat it as a ceiling the RTX 4060 will not reach, not an estimate for it. If you measure this pair, please contribute the numbers so /check/nanbeige4-2-3b/rtx-4060 stops being empty.
• VRAM usage: 5.762 GiB derived for the configuration above — 2.398 GiB of Q4_K_M weights, 2.922 GiB of q8_0 KV cache, 0.317 GiB of reserved logits and 0.125 GiB of FlashAttention dequant scratch — leaving roughly 2.24 GiB of the card's 8 GiB for the CUDA context, the activation working set and your desktop. The same community user independently reports that "Right now this works on 8 GB cards: 4 bit quant + 32768 tokens context length. Any higher and it is too large for VRAM."
• Quality notes: the model is ~4.17 B total parameters, of which ~3.15 B are non-embedding — the "3B" in the name counts the non-embedding half, and the model card's own comparison table lists it as 4B total / 3B non-embedding. The extra billion is the untied 166,144-token vocabulary at both ends of the stack. Nothing about this changes the GGUF sizes, but it explains why an "8.36 GB" bf16 repo carries a 3B label.

The context budget, and where it comes from

llama.cpp sizes the KV cache as n_layer × (n_embd_k_gqa + n_embd_v_gqa) × bytes_per_element per token. Here n_embd_k_gqa = n_embd_v_gqa = 8 KV heads × 128 head_dim = 1024, and n_layer is the logical count of 44:

KV type	Bytes / token	16K	32K	64K
f16 (default)	180,224	2.750 GiB	5.500 GiB	11.000 GiB
q8_0	95,744	1.461 GiB	2.922 GiB	5.844 GiB
q4_0	50,688	0.773 GiB	1.547 GiB	3.094 GiB

Weights and cache are not the whole bill, though, and on a card this small the remaining two terms decide rows.

A flat 0.317 GiB of reserved logits. llama.cpp sizes one worst-case graph at startup, and src/llama-context.cpp sets the number of logit rows that graph must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is min(min(32768, 512), 2048) = 512 rows, each a full f32 distribution over this model's unusually large 166,144-token vocabulary: 512 × 166,144 × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB. It does not move when you change -c; it halves if you halve -ub.

0.125 GiB of FlashAttention dequant scratch — only because the cache is quantized. A quantized KV cache is expanded back to f16 before the FlashAttention kernel reads it, and that scratch is VRAM. ggml/src/ggml-cuda/fattn.cu sets need_f16_K = need_f16_V = true for the tile and MMA kernels, and fattn-common.cuh then allocates ggml_nelements(K) * ggml_type_size(GGML_TYPE_F16) — guarded by if (need_f16_K && K->type != GGML_TYPE_F16). So an f16 cache pays nothing here, and a quantized one pays (1024 + 1024) × 2 B = 4,096 bytes per token of context: 0.063 GiB at 16K, 0.125 GiB at 32K, 0.250 GiB at 64K.

Putting all four terms together, the practical envelope on an 8 GB card is:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Verdict on 8 GB
Q4_K_M + f16 @ 32K	2.398	5.500	0.317	0.000	8.215 GiB	Will not fit
Q4_K_M + f16 @ 16K	2.398	2.750	0.317	0.000	5.465 GiB	Fits
Q4_K_M + q8_0 @ 32K	2.398	2.922	0.317	0.125	5.762 GiB	Fits — this recipe
Q5_K_M + q8_0 @ 32K	2.782	2.922	0.317	0.125	6.146 GiB	Fits — spend the headroom on quality
Q4_K_M + q4_0 @ 64K	2.398	3.094	0.317	0.250	6.059 GiB	Fits — spend it on context instead
Q4_K_M + q8_0 @ 64K	2.398	5.844	0.317	0.250	8.809 GiB	Will not fit

The first row is the trap. A reader who budgets from num_hidden_layers: 22 predicts 2.75 GiB of f16 KV at 32K, concludes it fits with room to spare, and is wrong by exactly a factor of two.

The scratch term is small here but it is not free, and it points the other way from the intuition that quantizing the cache is always the cheaper move: the 16K f16 row costs 5.465 GiB against this recipe's 5.762 GiB, so on raw accounting the honest trade is half the context at full cache precision versus full context at 8-bit. This recipe takes the context, because 16K is thin for the agentic work this model is trained for — but if your prompts are short and you would rather not quantize the cache at all, the second row is a defensible read of the same numbers.

Reaching the full 256K context

The card states that "The model supports a context length of up to 262,144 tokens (256K)." At 44 logical layers that is 44.000 GiB of f16 KV, or 12.375 GiB even at q4_0 — so it cannot live in 8 GB under any quantization. It is still reachable by keeping the weights on the GPU and the cache in system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q4_K_M.gguf \
  --n-gpu-layers 99 --ctx-size 262144 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --flash-attn on --no-kv-offload

This trades decode speed for window size and needs ~13 GB of free system RAM. Treat it as a capability, not a default. (The community entrypoint linked above estimates the same 256K cache at "~18 GiB even quantized"; the 12.375 GiB here is computed from q4_0's exact 18-bytes-per-32-values block layout, and the two figures are an estimate and a derivation rather than a disagreement about the architecture — that comment's own num_loops=2 -> 44 KV layers matches this recipe exactly.)

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-4060.

Troubleshooting

llama-server OOMs at 32K context

You are almost certainly on the default f16 KV cache, which needs 5.500 GiB at 32,768 tokens on top of the weights — 8.215 GiB accounted once the logits reservation is counted, against a card that has 8. Add --cache-type-k q8_0 --cache-type-v q8_0.

If the cache flags are already right and you are a couple of hundred MiB short, reach for -ub before you touch -c. The logits reservation is scaled by the micro-batch, not by the context, so it is the one term you can cut without losing window:

-ub	reserved logits	vs. default
512 (default)	0.317 GiB	—
256	0.158 GiB	−0.158 GiB
128	0.079 GiB	−0.238 GiB

It costs prompt-processing throughput, not context and not quality. Beyond that, your desktop compositor is holding VRAM — either drop to --ctx-size 16384 or move to q4_0 for both cache types.

Roughly one tool call in six is returned as plain text

A known llama.cpp parser bug, not a model bug. The model emits <tool_call> followed by a space rather than a newline for a minority of calls, and the auto-generated parser requires the newline verbatim. The two reporters put the rate differently — roughly one call in six in discussion #17, about a quarter in PR #26324 — so read it as a broad "some calls", not a measured constant. The PR reports that "All such tool calls currently fail and are displayed verbatim to the user instead of being executed." — and it was still open at the time of writing, so it is a community-proposed fix with no maintainer verdict yet. It bites hardest at the card's recommended agentic temperature of 1.0: in the reporter's 4-sample probe every generation parsed at temp 0, while some temp-1.0 configurations dropped to 2 of 4. Lowering the temperature therefore appears to suppress it, on a sample far too small to call it eliminated. A further report is in discussion #2 on the GGUF repo.

response_format: json_schema fails before any token is generated

Grammar-constrained decoding fails at sampler init with Failed to initialize samplers: Unexpected empty grammar stack after accepting piece: assistant (13886). Two users reproduced it independently in discussion #6 — but both were running the vendor fork: one names the nanbeige42 branch, the other a commit dated three days before mainline support merged. Neither report covers the mainline build this recipe installs, so treat the status here as untested rather than known-broken. If you do hit it, restart the server with --no-jinja; plain generation is unaffected either way.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M command points at a namespace that does not exist on ollama.com. Community re-uploads exist under user namespaces, but searching Ollama for "nanbeige" returns mostly 4.1 builds, which are a different model generation. Use llama-server as above.

The model loads but the output is subtly poor

Check that your GGUF carries nanbeige.num_loops = 2 (step 3 above). Also confirm your build post-dates 2026-07-27; older mainline builds do not know the architecture at all and will refuse to load rather than misbehave, but a GGUF converted by a third party without the loop key will load and quietly run at half depth.