Nanbeige4.2-3B on RTX 3060: Q8_0 weights and a 65,536-token cache in 12 GB

What You'll Build

A local llama-server endpoint running Nanbeige4.2-3B on one RTX 3060 12 GB with near-lossless Q8_0 weights and a 65,536-token context, both resident in VRAM. 65,536 is not an arbitrary round number: it is the max-new-tokens figure the model card itself recommends for agentic and tool-use work, and 12 GB is the smallest card class that holds that whole window without quantizing the weights down.

Hardware data: RTX 3060 (12 GB VRAM) · Q8_0 weights 4.130 GiB + q8_0 KV at 65,536 tokens 5.844 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.250 GiB = 10.541 GiB derived · no benchmark submitted yet · See benchmark data

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2-3B is a Looped Transformer — the model card says "Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters." — and config.json sets num_loops: 2. The 22 blocks execute twice per forward pass over one shared set of weights, but each pass keeps its own keys and values. src/models/nanbeige.cpp does the expansion before anything is allocated — "Expand logical layer count before load_tensors() allocates layers / KV" — and src/llama-kv-cache.cpp then sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure you compute from num_hidden_layers: 22 is exactly half the truth.

This is deliberate, not an oversight. A Nanbeige team member on discussion #18: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2".

What the extra 4 GB actually buys

The weights were never the problem on this model — a 3B fits anything. The only question a card answers is how much context, at what cache precision. Against an 8 GB card the answer is unusually tidy: 12 GB buys exactly one doubling at every cache tier, and it buys the weight tier at the same time.

KV cache type	Largest context on 8 GB	Largest context on RTX 3060 12 GB
f16 (unquantized)	16,384	32,768
q8_0	32,768	65,536
q4_0	65,536	131,072

Every cell is the largest doubling whose four accounted terms — weights, KV cache, the reserved logits and the FlashAttention dequant scratch, all derived below — stay inside the card. The 8 GB column is computed with Q4_K_M weights (2.398 GiB), because that is the largest tier that leaves room for those contexts. The 12 GB column is computed with Q8_0 (4.130 GiB) — the extra capacity absorbs a 1.732 GiB weight upgrade and still doubles every window. That is the practical difference between the two tiers, and it is why this recipe leads with Q8_0 where the 8 GB tier leads with Q4_K_M.

What 12 GB does not buy is the model's full declared context. The card states "The model supports a context length of up to 262,144 tokens (256K)." At 44 logical layers that cache is 12.375 GiB even at q4_0 — more than the entire card, before a single weight byte. On this GPU 262,144 is reachable only by moving the cache to system RAM (last section of Running); in VRAM it needs a 24 GB card.

Requirements

Component	Minimum	This recipe
GPU	12 GB VRAM, CUDA (sm_86 or newer)	RTX 3060 (12 GB) — not measured; the budget below is derived from file bytes and llama.cpp's own allocation rule (/contribute)
RAM	8 GB system RAM	— (16 GB+ for the --no-kv-offload long-context mode)
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	4,434,787,168 bytes, from the HF tree API
Software	llama.cpp b10153+, CUDA toolkit, CMake	—

NVIDIA also shipped an 8 GB RTX 3060. Every figure on this page assumes the 12 GB card. If yours reports 8 GB, use the 8 GB tier's budget instead — the weight tier and the context both come down.

Do not follow the model card's llama.cpp instructions. The card still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release; mainline merged native nanbeige support in PR #25994 on 2026-07-27, and the tracking request issue #26086 was closed the same day. Third-party write-ups still repeating "requires the authors' branch" are stale.

The usable-VRAM assumption, stated up front

The card is 12 GiB (12,288 MiB), and the budget below accounts for four terms, not two. Weights and KV cache are the two you choose. The other two both live in llama.cpp's CUDA compute buffer, and on this recipe's configuration they come to 580.5 MiB between them. They flip no configuration on this page from fits to does-not-fit, but they are far too large to leave out of a sum, and at the long end they are what decides whether a configuration is desktop-viable or headless-only.

A flat 0.317 GiB of reserved logits. llama.cpp sizes one worst-case prompt-processing graph at startup, and src/llama-context.cpp sets the logit rows that graph must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is 512 rows regardless of -c, each a full f32 distribution over this model's unusually large 166,144-token vocabulary: 512 × 166,144 × 4 B = 340,262,912 B = 324.5 MiB. It is large only because the vocabulary is, and it scales with -ub, not with context.

4,096 bytes per token of FlashAttention dequant scratch — but only because the cache is quantized. The FA kernels read f16, so a quantized cache is expanded into scratch VRAM first. ggml/src/ggml-cuda/fattn.cu sets need_f16_K = need_f16_V = true for the tile and MMA kernels, and fattn-common.cuh then adds ggml_nelements(K) * ggml_type_size(GGML_TYPE_F16) for K and again for V — each behind if (need_f16_K && K->type != GGML_TYPE_F16), so an f16 cache pays exactly none of it. For this model that is (1024 + 1024) × 2 B per context token: 0.125 GiB at 32,768, 0.250 GiB at 65,536, 0.500 GiB at 131,072. It is per-layer and transient rather than ×44 — ggml_nelements(K) is one layer's view over the cache, and ggml's allocator reuses the space across the 44 unrolled layers.

Two properties of that second term are worth stating, because both are counter-intuitive:

• It is charged at the reserve, not just while a prompt is processing. The graph llama.cpp sizes at startup is the prompt-processing one, at n_tokens = min(n_ctx, n_ubatch) = 512 rows. In ggml_cuda_get_best_fattn_kernel the vector kernel — the only one that reads a quantized cache directly and needs no scratch — is reachable only at one or two rows, i.e. during decode; a 512-row batch returns BEST_FATTN_KERNEL_MMA_F16 (or BEST_FATTN_KERNEL_TILE on pre-Turing hardware), and both set need_f16_K = need_f16_V = true. So the allocation is made once, up front, on any CUDA card.
• -ub does not shrink it. Its size comes from -c, not from the micro-batch. Halving -ub halves the logits reservation and leaves this term exactly where it was.

Two more consumers sit on the same 12 GiB and are deliberately not in the accounted total, because nothing here measures them:

• The CUDA runtime's own context. A few hundred MiB per process that llama.cpp never reports. Budget for it; do not assume it away.
• Your desktop. A compositor driving a monitor off the same card typically holds several hundred MiB more.

So: 10.541 GiB accounted, 1.459 GiB left on a 12 GiB card for those two. On a headless card the lead configuration has comfortable room. On a desktop card driving a 4K monitor it is snug — and if it is too snug, the first lever is -ub 256, which buys back 162 MiB without costing a single token of context. (Budget one more thing outside VRAM entirely: the attention mask is n_kv × n_ubatch in f16 — 64 MiB here — and src/llama-graph.cpp asserts ggml_backend_buffer_is_host(self_kq_mask->buffer), so it is pinned system RAM, not card memory. Counting it as VRAM is a common way to over-budget this model.)

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Confirm the checkout actually carries the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q8_0 GGUF

Nanbeige publishes no first-party GGUF — the org's quantized builds of this model are FP8 and GPTQ-Int8 only — so this uses a community conversion. owao/Nanbeige4.2-3B-GGUF ships a 13-rung ladder; bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a 23-rung alternative. Both re-uploaded after the vendor's 2026-07-27 tokenizer fix. Pick one repo and stay in it: the two quantizers' files are not byte-identical, and mixing their numbers into one budget is how a sum goes quietly wrong (their Q4_K_M builds differ by 109 MB).

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q8_0.gguf \
  --local-dir ./models

numpy is not a dependency of huggingface_hub, and the verification step below needs it.

3. Verify the artifact carries the loop parameter

Worth doing by hand exactly once. If a GGUF was produced without num_loops, llama.cpp defaults the key to 1, runs 22 layers instead of 44, and you get a silently different model — no error, no warning, half the depth.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

Two lines come back, ending = 22 and = 2 respectively. (The leading index numbers vary from file to file; only the key names and the values matter.) Reading this exact artifact's GGUF header directly confirms it: general.architecture = nanbeige, nanbeige.block_count = 22, nanbeige.num_loops = 2, nanbeige.attention.head_count_kv = 8, key_length = 128, value_length = 128, context_length = 262144, 201 tensors. A missing num_loops line is the failure case.

Two commands that look like they should do this job do not, so do not substitute them. llama-gguf <file> r n prints key names only, never their values. And llama-cli suppresses the loader's metadata dump at its default verbosity — tools/cli/cli.cpp sets params.verbosity = LOG_LEVEL_ERROR, common/log.cpp maps INFO to trace level, and the resulting 4 <= 1 gate never passes — besides which llama-cli is an interactive chat client that will sit waiting for input rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 65536 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 1.0 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

--temp 1.0 with a 65,536-token budget is the model card's own recommendation for agentic and tool-use tasks. For reasoning and chat it recommends --temp 0.6; the card pairs that with 131,072 new tokens, which on this card means the q4_0 configuration below.

Four flags are load-bearing:

• -c 65536, set explicitly. Omitting -c is not neutral: llama.cpp's -fit on default interpolates the context downward until the model fits free memory, so you silently get some reduced number. -c 0 is not neutral either, and it is the more dangerous of the two — common/arg.cpp reads it as "give me the full trained context, do not shrink it" and sets fit_params_min_ctx = UINT32_MAX, which at the f16 cache default means a 44 GiB allocation on a 12 GiB card and an immediate abort.
• -fa on. A quantized V cache requires Flash Attention; without it llama.cpp refuses with quantized V cache was requested, but this requires Flash Attention. The auto default would enable it for you, but being explicit makes the failure mode legible.
• -ctk q8_0 -ctv q8_0 — matched types. A stock CUDA build instantiates Flash Attention kernels only for identical K and V types: ggml/src/ggml-cuda/fattn.cu contains if (K->type != V->type) { return BEST_FATTN_KERNEL_NONE; } outside GGML_CUDA_FA_ALL_QUANTS, and its default table covers f16, bf16, q4_0 and q8_0 in matched pairs only. A clever-looking -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. There is no ×4 multiplier hiding in these numbers. tools/server/server.cpp resolves the default -np -1 with params.n_parallel = 4; params.kv_unified = true; in the same branch — unified means one shared pool of -c cells that a single conversation may consume in full, so the 5.844 GiB above is the whole cache bill. Passing an explicit positive --parallel N skips that branch, leaves kv_unified = false, and gives each conversation -c / N for identical memory. If you want one slot, say -np 1.

The maximum-context configuration

Trade cache precision for window. q4_0 at 131,072 tokens is 6.188 GiB of cache, and with the same Q8_0 weights the accounted total is 11.135 GiB — inside the card, but with only 0.865 GiB left for the CUDA context and your display. Treat this one as headless-friendly and desktop-marginal. Doubling the context also doubles the dequant scratch to 0.500 GiB, which is the point at which it overtakes the logits reservation and becomes the larger of the two compute-buffer terms:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 131072 -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

That covers the card's recommended 131,072-token reasoning budget.

If you would rather keep the near-lossless cache and give up context instead, -c 32768 -ctk f16 -ctv f16 runs the cache completely unquantized — a configuration an 8 GB card cannot reach at all. It is also, counter-intuitively, the roomiest of the three Q8_0 configurations on this page: 9.947 GiB accounted, 2.053 GiB left over, against the lead configuration's 1.459 GiB and this section's 0.865 GiB. The KV column makes the two look close (5.500 GiB against 5.844 GiB), but an f16 cache pays no dequant scratch at all, so the real gap is 0.594 GiB rather than 0.344 GiB. Quantizing the cache is a way to buy context, not a way to save memory — and on this card the configuration framed as the sacrifice turns out to have the most headroom.

Reaching the full 262,144 tokens

Not possible in VRAM here: 12.375 GiB of q4_0 cache alone exceeds the card. Keep the weights on the GPU and put the cache in system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM and trades decode speed for window size. Treat it as a capability, not a default. A community user running an 8 GB card annotates both modes in the entrypoint script he posted on discussion #18 — "KV_OFFLOAD=1 (default) keeps KV in VRAM: fastest, ctx up to ~64k." — those are his own script comments for a smaller card, not a published benchmark.

Results

• VRAM usage: 10.541 GiB accounted at the lead configuration — 4.130 GiB of Q8_0 weights, 5.844 GiB of q8_0 cache at 65,536 tokens, 0.317 GiB of reserved logits and 0.250 GiB of FlashAttention dequant scratch — leaving 1.459 GiB of the card's 12 GiB for the CUDA context, the activation working set and your desktop. Derived from measured file bytes and llama.cpp's own allocation rules, not measured on hardware; see /check/nanbeige4-2-3b/rtx-3060. Separately, budget ~64 MiB of system RAM for the pinned attention mask.
• Throughput: no measurement exists for this pair, and none exists for this model on any 12 GB NVIDIA card. I enumerated all 27 discussion threads on the canonical repository and all 3 on the distribution repository, searched llama.cpp's tracker for nanbeige (5 results, all architecture-support or tool-call parsing, none GPU-specific), and searched the open web; the only throughput figures anywhere are on a single 8 GB card of a different generation, at a different context and cache type. Publishing a number derived from that would be inventing one. If you run this configuration, please contribute the measurement so /check/nanbeige4-2-3b/rtx-3060 stops being empty.
• Quality notes: the model is ~4.17 B parameters total, ~3.15 B non-embedding — the "3B" in the name counts the non-embedding half, and the extra billion is the untied 166,144-token vocabulary at both ends of the stack. I found no evaluation of this model under a quantized KV cache at any tier: treat q8_0 cache as the conservative choice on the ecosystem's general reputation, and q4_0 as the aggressive one, rather than as a measured quality difference on this architecture.

Where the numbers come from

llama.cpp sizes the cache as n_layer × (n_embd_k_gqa + n_embd_v_gqa) × bytes_per_element per token. config.json gives num_key_value_heads: 8 and head_dim: 128, so each layer stores 8 × 128 = 1024 elements for K and 1024 for V. Over 44 logical layers that is 90,112 elements per token:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

The block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 elements, from ggml/src/ggml-common.h. Multiplying out:

Context	f16 KV	q8_0 KV	q4_0 KV
16,384	2.750 GiB	1.461 GiB	0.773 GiB
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
98,304	16.500 GiB	8.766 GiB	4.641 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

A quantized cache also carries a flat 4,096 bytes per token of FlashAttention dequant scratch, derived in the usable-VRAM section — zero on an f16 cache. So the price of a quantized cache is the table above plus 4,096: 99,840 B/token effective at q8_0, 54,784 at q4_0.

Adding weights and both compute-buffer terms, the practical envelope on 12 GiB. Weight byte counts are from the owao repository via the HuggingFace tree API:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Unallocated on 12 GiB
Q8_0 · q8_0 · 65,536 (this recipe)	4.130	5.844	0.317	0.250	10.541 GiB	1.459 GiB
Q8_0 · f16 · 32,768	4.130	5.500	0.317	0.000	9.947 GiB	2.053 GiB
Q8_0 · q4_0 · 131,072	4.130	6.188	0.317	0.500	11.135 GiB	0.865 GiB — headless only
Q4_K_M · q4_0 · 131,072	2.398	6.188	0.317	0.500	9.402 GiB	2.598 GiB
Q4_K_M · q8_0 · 98,304	2.398	8.766	0.317	0.375	11.855 GiB	0.145 GiB — no room for the CUDA context
Q8_0 · q8_0 · 98,304	4.130	8.766	0.317	0.375	13.588 GiB	will not fit
Q4_K_M · f16 · 65,536	2.398	11.000	0.317	0.000	13.715 GiB	will not fit

The Q4_K_M rows are there to show what dropping the weight tier actually buys, and the answer is: not much. Q8_0 costs 1.732 GiB more than Q4_K_M, which spent on cache instead would buy 18,629 more tokens at q8_0 or 33,951 at q4_0 — under 30% more window in the first case. (Both at the effective per-token cost, dequant scratch included; on the sticker KV price alone they would read 19,426 and 36,694, and that gap is the tax.) On a model whose entire pitch is reasoning and agentic accuracy at 3B, that is the wrong trade on a card with room. It is the right trade on 8 GB, which is exactly why the two tiers lead with different quants.

Troubleshooting

Out of memory at startup

Almost always the default f16 cache. At 65,536 tokens that is 11.000 GiB before any weights — see the table above. Add -ctk q8_0 -ctv q8_0.

If it still fails, your display is holding VRAM, and the levers go in this order: -ub 256, then -c 49152, then -ctk q4_0 -ctv q4_0. Reach for -ub first because it is the only one that costs no context — but be clear about how much it buys. It halves the logits half of the compute buffer, from 324.5 MiB to 162.2 MiB, and does not touch the 256.0 MiB of dequant scratch at all, because that term is sized by -c. So -ub 256 cuts about 28% of the lead configuration's 580.5 MiB compute buffer, not half of it, and -ub 128 about 42%. Past that the cache tier is the bigger lever: q8_0 → q4_0 saves the full 45,056 bytes per token and adds nothing back, since the scratch is a flat 4,096 bytes per token on both.

unknown model architecture: 'nanbeige'

The build predates b10153. Rebuild from mainline master, or take a release tag at or above b10153 — that build's commit is the merge of PR #25994. Distribution channels lag: the model card notes that LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into the LM Studio backend directory.

Tool calls come back as plain text instead of executing

A llama.cpp parser bug, not a model bug and not your configuration. The model sometimes emits <tool_call> followed by a space rather than a newline, and the auto-generated parser requires the newline. It is tracked in PR #26324, still open at the time of writing, whose author puts the loss at roughly a quarter of calls; the model's own discussion #17 reports a lower rate, roughly one call in six. Both are community reports rather than a maintainer measurement, so read it as "some calls" rather than a constant — and note that #17's author measured his rate on the vendor fork at d28da86, not on the mainline build this recipe installs. Unlike the grammar bug below, that does not put the status in doubt: PR #26324 is filed against mainline master and reports the same failure there. It bites hardest at the --temp 1.0 this recipe recommends for agentic work — #17's small probe found every generation parsing at temperature 0. Until the PR lands, check the raw completion text when a call appears to vanish.

response_format: json_schema fails before any token is generated

Grammar-constrained decoding aborts at sampler init with Failed to initialize samplers: Unexpected empty grammar stack. Two users reproduced it independently on discussion #6 — but both were running the vendor fork rather than the mainline build installed here, so treat the status on mainline as untested rather than known-broken. If you hit it, restart the server with --no-jinja; plain generation is unaffected either way, and the cost is that you apply the prompt format yourself.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads exist under user namespaces, but most search hits for "nanbeige" are 4.1 builds — a different model generation with different architecture behaviour. Use llama-server as above.

The model loads but the output is subtly poor

Re-run step 3 and confirm the GGUF reports nanbeige.num_loops = 2. A file converted by a path that omits the key loads happily and runs at half depth. Also confirm the build post-dates 2026-07-27; older mainline builds refuse to load the architecture outright rather than misbehave.

Anything else, or a real throughput measurement on this card, is welcome via the submission form.