Nanbeige4.2-3B on RTX 4060 Ti 16GB: 96K Context Without a 4-bit KV Cache

What You'll Build

A llama-server endpoint running Nanbeige4.2-3B on an RTX 4060 Ti 16GB with a 98,304-token window and near-lossless quantization on both axes — Q8_0 weights and a q8_0 KV cache — for 13.588 GiB accounted, inside a card that has 16 GiB to give.

Hardware data: RTX 4060 Ti 16GB (16 GB VRAM) · Q8_0 weights 4.130 GiB + q8_0 KV at 98,304 tokens 8.766 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.375 GiB = 13.588 GiB derived · no benchmark submitted yet · See benchmark data

This page treats 16 GB as a tier in its own right rather than as a step on a ladder, because 16 GB is where this model's central problem changes shape. On a small card the only question is how much can I give up to make it fit. Here the model fits several different ways, and the question becomes which axis to spend the memory on — weight precision, cache precision, or context length. The answer turns out to be non-obvious, and it is the same answer on every 16 GB card.

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2 is a Looped Transformer — the model card says its "Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters." config.json sets num_loops: 2, so the 22 blocks run twice per forward pass over one shared set of weights, and each pass keeps its own keys and values. llama.cpp expands the layer count before it allocates anything: src/models/nanbeige.cpp sets hparams.n_layer_all = n_layer_phys * n_loops under the comment "Expand logical layer count before load_tensors() allocates layers / KV.", and src/llama-kv-cache.cpp sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure computed from 22 layers is exactly half the truth. The vendor confirms the design is deliberate — a Nanbeige team member on discussion #10: "We have also investigated KV-cache sharing across loop passes, but the performance gains were notably smaller than with the full looped setup."

Requirements

Component	Minimum	This recipe
GPU	16 GB VRAM, CUDA	RTX 4060 Ti 16GB — not measured; the budget below is derived from file bytes and llama.cpp's own allocation rules (/contribute)
RAM	8 GB system RAM	—
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	4,434,787,168 bytes, from the HF tree API
Software	llama.cpp b10153 or newer, CUDA toolkit, CMake	—

Ignore the model card's llama.cpp instructions. It still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release; mainline merged native nanbeige support in PR #25994 on 2026-07-27, and release b10153 is tagged at exactly that merge commit. The upstream feature request tracking it, issue #26086, was closed the same day. Mainline is also where the fixes land, so there is no reason to be on the fork any more.

The 16 GB question

Weights are not the constraint on this card at any tier — the largest GGUF on offer below bf16 is 4.130 GiB. The whole recipe is the cache.

KV cost per token

config.json gives num_key_value_heads: 8 and head_dim: 128, so each logical layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Over 44 logical layers that is 90,112 elements per token. The per-element cost comes from the block layouts in ggml/src/ggml-common.h — block_q8_0 is 34 bytes per 32 values, block_q4_0 is 18 bytes per 32 values:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

Multiplied out, this is the whole cache bill:

Context	f16 KV	q8_0 KV	q4_0 KV
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
98,304	16.500 GiB	8.766 GiB	4.641 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
196,608	33.000 GiB	17.531 GiB	9.281 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

What is not the cache, and how much VRAM to actually budget

Three things sit on top of weights + KV. Two are flat; the third is the one that decides this page, and it exists only if you quantize the cache.

The CUDA context is a fixed cost of a few hundred MiB that appears the moment the runtime touches the device.

The compute buffer is dominated, on this model, by a single reserved tensor: the logits. llama.cpp sizes its worst-case graph once at startup, and src/llama-context.cpp computes how many rows of logits that graph must hold as n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. With stock settings that is min(min(98304, 512), 2048) = 512 rows, each one a full f32 distribution over the vocabulary. This model's vocabulary is 166,144 tokens — several times a typical one — so:

512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB

That is the term worth budgeting, and the important property is that n_ubatch sets it, not -c. Doubling your context does not touch it; halving -ub halves it. Everything else in the buffer is per-microbatch activation working set — the widest single tensor is the FFN intermediate at 10752 × 512 × 4 B ≈ 21 MiB — which ggml's allocator reuses across the 44 unrolled layers rather than accumulating per layer.

The third term is the one nobody budgets for: a quantized KV cache is dequantized back to f16 before FlashAttention reads it, and the scratch for that lives in VRAM. ggml/src/ggml-cuda/fattn-common.cuh allocates it immediately past the destination tensor, and the guard is the whole story:

if (need_f16_K && K->type != GGML_TYPE_F16) {
    data.end = GGML_PAD(data.end, 128);
    data.K   = data.end;
    data.end += ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16);
}

with a near-identical block for V — its one extra branch is the second bullet below. Three details decide the size:

• It is skipped entirely for an f16 cache — K->type != GGML_TYPE_F16 is false, so the term is exactly zero. This is the only place on this page where the default cache is cheaper than a quantized one.
• V gets its own copy here. The code reuses K's buffer when V_is_K_view, but that branch is for architectures where K and V share storage. src/llama-kv-cache.cpp allocates this model's cache as two separate tensors — ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream) and a matching one for V — so both copies are charged.
• It is per-layer and transient, not ×44. ggml_nelements(K) is one layer's view over n_kv cells, and ggml's allocator reuses the space across layers.

So the cost is (n_embd_k_gqa + n_embd_v_gqa) × 2 B = (1024 + 1024) × 2 = 4,096 bytes per token of context, and unlike everything else in this section it grows with -c: 0.375 GiB at 98,304 tokens, 0.750 GiB at 196,608, 1.000 GiB at 262,144.

It is charged at the reserve, not just during prompt processing. On this card (ggml/src/ggml-cuda/fattn.cu) the vector kernel — the one that reads a quantized cache directly and needs no scratch — is selected only when the batch is at most 2 rows, so the worst-case graph llama.cpp sizes at startup always takes the dequantizing path. You cannot avoid this term by shrinking -ub; you avoid it only by not quantizing the cache.

The attention mask is not on this list, and that is not an oversight. It is the obvious candidate — it really is n_kv × n_ubatch × 2 B under Flash Attention, which really would be 0.094 GiB at 98,304 tokens and would really grow with context. But it does not live in VRAM. llm_graph_input_attn_kv::set_input fills it from the CPU and asserts as much — src/llama-graph.cpp carries GGML_ASSERT(ggml_backend_buffer_is_host(self_kq_mask->buffer)); — so on CUDA it is a pinned host allocation. You still pay for it, in system RAM: 96.0 MiB at 98,304 tokens, 256.0 MiB at 262,144. Budget it there and not here.

So the budget rule for this card: treat 14.5 GiB as the accounted ceiling on a 16 GiB card. That reserves roughly 1.5 GiB for the CUDA context, the activation working set and a desktop compositor. Run headless and you have more; run Windows with a browser open and you have less. The lead configuration below lands at 13.588 GiB rather than pressing that ceiling, because the terms nobody here has measured are the ones you find out about by crashing.

Putting the three together, the per-token cost that actually governs a quantized cache is not the number in the table above but that number plus 4,096:

Cache type	KV bytes/token	+ dequant scratch	Effective
f16	180,224	0	180,224
q8_0	95,744	4,096	99,840
q4_0	50,688	4,096	54,784

In relative terms the tax is modest — it eats 4.8% of what q8_0 saves against f16 and 3.2% of what q4_0 saves. In absolute terms at the long end it is a full gigabyte, which is exactly where a 16 GB card's decisions get made.

The answer: the largest context each combination buys

Every cell is the largest multiple of 16,384 tokens whose weights + KV + the 0.317 GiB logits reservation + the dequant scratch stay under 14.5 GiB accounted. That is what "round" means here and it is the only ladder used on this page. Each cell reads context · GiB accounted, so what the configuration leaves unspent against the 14.5 GiB ceiling is visible rather than implied.

Weights	f16 KV	q8_0 KV	q4_0 KV
Q4_K_M (2.398 GiB)	65,536 · 13.715	114,688 · 13.379	229,376 · 14.418
Q5_K_M (2.782 GiB)	65,536 · 14.099	114,688 · 13.763	212,992 · 13.966
Q6_K (3.190 GiB)	49,152 · 11.757	114,688 · 14.171	212,992 · 14.374
Q8_0 (4.130 GiB)	49,152 · 12.697	98,304 · 13.588	196,608 · 14.478

The one cell that looks like an error is not one. Q6_K with an f16 cache stops at 49,152 and leaves 2.743 GiB unspent, because the next rung up — 65,536 tokens of f16 — lands at 14.507 GiB, over the ceiling by 6.8 MiB. That is 40 tokens' worth of cache: the exact maximum there is 65,496. One weight rung lighter, Q5_K_M reaches that same window with 0.401 GiB to spare. That cliff is what an unquantized cache costs at 180,224 bytes per token: it turns a 0.408 GiB difference in weights into a 16,384-token difference in window.

Read the table and two things fall out.

Yes, f16 KV becomes affordable at 16 GB — at 65,536 tokens, and only with Q4_K_M or Q5_K_M weights. That is a real threshold and it is worth naming, because at 180,224 bytes per token the untouched default cache is what puts small cards out of the game — 32,768 tokens of f16 is already 5.500 GiB of cache on its own, before any weights. Sixteen gigabytes is the first capacity where the default reaches 65,536, which is exactly the max-new-tokens the model card recommends for agentic and tool-use work. A reader who wants to type -ngl 99 -c 65536 and nothing else can now do that on this card.

And you should still not do it — though the margin is much narrower than the KV table alone suggests. Set the two candidates side by side with every term counted:

	Weights	KV	Logits	Dequant	Total
Q4_K_M + f16 @ 65,536	2.398	11.000	0.317	0.000	13.715 GiB
Q8_0 + q8_0 @ 98,304	4.130	8.766	0.317	0.375	13.588 GiB

The quantized-cache configuration wins by 0.127 GiB — a rounding error on a 16 GB card, and small enough that the unmeasured CUDA context could swallow it. On the KV numbers alone the gap looks like 0.502 GiB; the dequant scratch takes three quarters of that back, because the f16 option pays none of it and the q8_0 option pays it on half again as many tokens.

So the recommendation stands, but not for the reason it first appears to. Quantizing the cache is not a free lunch and it does not really save you memory here. What it buys, at the same ~13.6 GiB, is 1.5× the context window and a near-lossless weight tier instead of a 4-bit one. That is a good trade — it is simply a trade, not a windfall, and anyone who budgets q8_0 at its sticker price of 95,744 bytes per token will be about 0.4 GiB short at this context and a full gigabyte short at 262,144.

And if you are willing to take 4-bit weights after all, the same q8_0 cache buys 114,688 tokens at 13.379 GiB — 1.75× the f16 window for less total memory than either row above. That is a weight-precision decision rather than a cache one, and it is the axis this page deliberately does not choose for you.

That is the tier's answer, and it is why the lead configuration below is the near-lossless one rather than the flag-free one.

What the extra memory over the 8 GB tier actually buys, in tokens

Concretely, since the cost per token is a constant: 8 GiB of additional budget is 47,662 more tokens at f16, 86,037 more at q8_0, and 156,796 more at q4_0 — all three computed at the effective per-token cost, dequant scratch included. (On the sticker KV price the last two would read 89,717 and 169,466; that difference is the tax, and it is the kind of number that turns a plan into an OOM at startup.) The cheaper the cache, the more the extra memory is worth — which is the same conclusion arriving from the other direction, and the reason the q4_0 column is where the tier's headline number lives if you want one: 196,608 tokens on any weight tier the card can hold.

What 16 GB does not buy

The model's full declared window. The card states that "The model supports a context length of up to 262,144 tokens (256K).", and at 44 logical layers the cheapest cache that reaches it is q4_0 at 12.375 GiB — plus 1.000 GiB of dequant scratch, which at this context is no longer a rounding term. With Q8_0 weights the total is 17.822 GiB and with Q4_K_M weights 16.090 GiB: both are over the raw 16 GiB card, before the CUDA context is counted at all. There is no margin argument to have here and no weight tier that rescues it.

The honest in-VRAM ceiling is 196,608 tokens, which every weight tier from Q4_K_M to Q8_0 reaches. Above that the ladder thins out and the accounted totals crowd the ceiling: Q5_K_M reaches 212,992 at 13.966 GiB, which is still a configuration with room in it; Q6_K reaches the same window at 14.374 and Q4_K_M reaches 229,376 at 14.418, leaving 0.126 and 0.082 GiB to spare under the 14.5 GiB ceiling. Treat those last two as the edge of the envelope rather than configurations to plan around — headless, and the first thing to give back when anything else on the card wants memory. 262,144 is reachable only by moving the cache to system RAM (see Troubleshooting).

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Confirm the checkout actually knows the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q8_0 GGUF

There is no first-party GGUF. Enumerating all ten repositories under the Nanbeige org shows safetensors in the model repos and, for quantized builds, only -FP8 and -GPTQ-Int8 variants — no GGUF anywhere. This recipe uses owao/Nanbeige4.2-3B-GGUF, a 13-rung community conversion re-uploaded after the vendor's 2026-07-27 chat-template fix.

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q8_0.gguf \
  --local-dir ./models

numpy is there for step 3, not for the download — it is not a dependency of huggingface_hub and the dump script will fail without it.

Do not mix quantizers when you redo this arithmetic. bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a 23-rung alternative and its files are not the same bytes: its Q4_K_M is 2,684,023,968 B (2.500 GiB) against owao's 2,574,807,904 B (2.398 GiB). Every weight figure on this page is owao's. The two repos' Q8_0 files happen to agree to within 320 bytes of metadata, but the K-quants do not.

3. Verify the artifact carries the loop parameter

This is the one check worth doing by hand. load_arch_hparams() reads num_loops as optional with a default of 1, so a GGUF converted by a path that omits the key loads without complaint, allocates 22 layers instead of 44, and runs a silently different model.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

You want two lines, ending = 22 and = 2 respectively:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case. (Index numbering will differ; only the names and values matter.)

Two commands that look like they should do this do not, so do not substitute them: llama-gguf <file> r n prints key names without their values, and llama-cli suppresses the loader's metadata dump at its default verbosity — that dump is emitted at trace level, so it needs -lv 4 — besides which llama-cli is an interactive chat client that waits for input rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 98304 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. The sampler values are the model card's recommendation for reasoning and chat; for agentic and tool-use work it recommends --temp 1.0.

Four flags carry weight here:

• -c 98304, set explicitly. Omitting -c is not neutral: fit_params defaults to true in common/common.h, so llama.cpp interpolates the context down until the model fits free memory with a 1 GiB margin — as low as fit_params_min_ctx, which is 4096. And -c 0 is the opposite trap: common/arg.cpp reads it as "give me the full trained context, do not shrink it" by setting fit_params_min_ctx = UINT32_MAX, which at the f16 default is a 44.000 GiB allocation and an immediate out-of-memory abort. Name the number you want.
• -fa on. A quantized V cache requires Flash Attention; the runtime will enable it for you and log that it did, but being explicit makes the failure legible. It also halves the attention mask, from F32 to F16 — which is a saving in host RAM rather than in VRAM, per the note above.
• -ctk q8_0 -ctv q8_0 — matched types. A stock CUDA build instantiates Flash Attention kernels only for identical K and V types: ggml/src/ggml-cuda/fattn.cu returns BEST_FATTN_KERNEL_NONE when K->type != V->type, guarded by #ifndef GGML_CUDA_FA_ALL_QUANTS. A clever-looking asymmetric cache such as -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. There is no ×4 multiplier hiding in these numbers. common/arg.cpp sets params.n_parallel = -1 for the server, and tools/server/server.cpp resolves the sentinel with params.n_parallel = 4; params.kv_unified = true; in the same branch — unified meaning one shared pool of -c cells that a single conversation may consume in full. Passing an explicit positive --parallel N skips that branch, leaves the cache non-unified, and quarters the per-conversation window for identical memory. If you want one slot, write -np 1, not -np 4.

If you want the round 128K number

The model card recommends 131,072 max-new-tokens for reasoning and chat, and that window is reachable — by dropping the cache a rung rather than the weights. Keeping Q8_0 weights and moving to a q4_0 cache costs 11.135 GiB accounted, which is the most comfortable configuration on this page by some distance:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 131072 -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

The obvious-looking alternative — keep the q8_0 cache and drop the weights to Q4_K_M — does not fit: 14.902 GiB, over the ceiling, because q8_0 at 131,072 tokens carries 0.500 GiB of dequant scratch on top of its 11.688 GiB of cache. The table above stops that pairing one rung earlier, at 114,688. Between a 4-bit cache with 8-bit weights and an 8-bit cache with 4-bit weights, this card can only afford the first.

If you want maximum context in VRAM

196,608 tokens is the ceiling every weight tier reaches. Q6_K weights get there at 13.538 GiB accounted, with proper headroom; Q8_0 weights also fit, at 14.478 GiB, but leave only 0.022 GiB to spare under the 14.5 GiB ceiling — so Q6_K is the better buy at this context and is what is shown here. q4_0 is the aggressive rung of the cache ladder — this is the configuration to reach for when the workload is long-document or long-trajectory and you can tolerate that.

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q6_K.gguf \
  -ngl 99 -c 196608 -fa on \
  -ctk q4_0 -ctv q4_0

That -c is deliberately one rung below Q6_K's own maximum in the table above — 212,992 at 14.374 GiB. The extra 16,384 tokens cost 0.836 GiB of exactly the headroom that covers the CUDA context, which is the term nobody here has measured. If you want that rung anyway, take it a weight tier lighter: Q5_K_M reaches 212,992 for 13.966 GiB, which is 0.408 GiB safer than Q6_K at the same window. Past that, Q4_K_M at 229,376 tokens for 14.418 GiB is the edge of the envelope, headless only.

preserve_thinking is a memory dial on this card

The chat template takes preserve_thinking, which controls whether reasoning from earlier assistant turns stays in the context — and therefore in the cache. A Nanbeige team member on discussion #9 gives both halves of the trade: "In our evaluations, enabling preserve_thinking generally provides better performance and better kv-cache reuse, so we recommend keeping it enabled when context length and memory allow.", and then "If you are working with a tight memory budget, disabling it can help reduce the KV-cache." At the lead configuration you are on the comfortable side of that line with 2.412 GiB unspent, so keep it on; at 196,608 and every rung above it you are not, and it becomes the first thing to turn off. Pass it through the OpenAI-compatible endpoint as "chat_template_kwargs": {"preserve_thinking": true}.

Results

• Speed: omitted. There is no throughput measurement of this model on a 16 GB NVIDIA card, and I could not find one on any surface I searched: all 27 discussion threads on the canonical repo (fetched individually, not searched), the discussions on both community GGUF repos, llama.cpp's issue tracker, and two web searches. The nearest thing is a configuration rather than a number — a user reporting a grammar bug on discussion #6 lists a 16 GB GPU running the Q8_0 GGUF at 65,536 context with Flash Attention and a unified cache, on a build predating mainline support, and states no cache type and no tok/s. It corroborates that a 16 GB card runs this model at that window; it measures nothing. If you run this pair, please contribute the numbers so /check/nanbeige4-2-3b/rtx-4060-ti-16gb stops being empty.
• VRAM usage: 13.588 GiB accounted at the lead configuration — 4.130 GiB of Q8_0 weights, 8.766 GiB of q8_0 cache, 0.317 GiB of reserved logits and 0.375 GiB of FlashAttention dequant scratch — leaving 2.412 GiB of the card's 16 GiB for the CUDA context, the activation working set and your desktop. Derived from measured file bytes and llama.cpp's own allocation rules, not measured on hardware. Separately, budget ~96 MiB of system RAM for the pinned attention mask.
• Quality notes: the model is ~4.17 B parameters in total, of which ~3.15 B are non-embedding; the "3B" in the name counts the non-embedding half, and the model card's own comparison table labels it that way. The total is HuggingFace's own tensor census of the repo — the model info API reports safetensors.total = 4,169,800,704 — and the extra billion is the untied 166,144-token vocabulary at both ends of the stack: 2 × 166,144 × 3,072 = 1,020,788,736 parameters, leaving 3,149,011,968. I found no evaluation of this model under a quantized KV cache at any tier in the spaces listed above — q8_0 is treated as near-lossless across the llama.cpp ecosystem generally, but that is an ecosystem prior, not a measurement of this model. The vendor publishes no throughput figures at all; the model card and the technical report carry quality benchmarks only.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-4060-ti-16gb.

Troubleshooting

Out of memory at startup

Check -ctk/-ctv first. At f16 the cache at 98,304 tokens is 16.500 GiB on its own — larger than the whole card — and that is the single most common way to be surprised by this model.

If the cache flags are already right and you are still a few hundred MiB short, reach for -ub before you touch -c. The compute buffer's dominant term is the logits reservation, and it is scaled by the micro-batch, not by the context:

-ub	reserved logits	vs. default
512 (default)	324.5 MiB	—
256	162.2 MiB	−162.3 MiB
128	81.1 MiB	−243.4 MiB
64	40.6 MiB	−283.9 MiB

./build/bin/llama-server -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 98304 -fa on -ctk q8_0 -ctv q8_0 -ub 256

This costs prompt-processing throughput — smaller micro-batches mean less work per kernel launch — and it costs nothing else. Decode speed and the context window are both untouched, which is why it is the right dial to turn first.

-ub does not touch the dequant scratch, though, so it tops out at about 284 MiB of relief. Past that the levers are the cache tier and the context, in that order — and note that dropping from q8_0 to q4_0 is worth more than the arithmetic in the KV table alone suggests, because the scratch is a flat 4,096 bytes per token on both, so going down a cache rung saves the full 45,056 bytes per token and adds nothing back. Cut -c last; it is the thing you came to this card for.

llama_model_load: error loading model architecture: unknown model architecture: 'nanbeige'

Your build predates b10153. Rebuild from mainline master or take a release tag at or above b10153 — that tag points at the merge commit of PR #25994. The same symptom is reported on the model's discussion #23 and on the GGUF repo's discussion #1, both from the pre-mainline period. Note that some distribution channels lag independently of your own build: the model card says LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into its backend directory.

Tool calls come back as plain text instead of executing

A llama.cpp parser bug, not a model bug and not your configuration. The model emits <tool_call> followed by a space rather than a newline for a minority of calls, and the auto-generated parser matches the marker with the newline attached. The reporter of PR #26324 puts the rate at roughly 25% and the consequence plainly: "All such tool calls currently fail and are displayed verbatim to the user instead of being executed." The PR was still open at the time of writing, so this is a community-proposed fix with no maintainer verdict — treat tool-call reliability as a known ceiling under llama.cpp and check the raw completion text when a call appears to vanish. The same behaviour is discussed on the model's discussion #17 and on the GGUF repo's discussion #2.

response_format: json_schema fails at sampler init

Grammar-constrained decoding fails before a token is generated, with Failed to initialize samplers: Unexpected empty grammar stack after accepting piece: assistant (13886). Two users reproduced it independently on discussion #6 — including the 16 GB report mentioned above — and both found --no-jinja clears it, while --chat-template chatml, strict: false and disabling reasoning all do not. Both reproductions ran builds predating mainline support, so read the status on a current build as untested rather than known-broken. The cost of --no-jinja is that you lose the model's own chat template and have to apply the prompt format yourself.

You need the full 262,144-token window on this card

It does not fit in 16 GiB at any cache quantization once the compute buffer is counted. Keep the weights on the GPU and move the cache to system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM for the cache and trades decode speed for window size. Treat it as a capability, not a default — on this card the in-VRAM edge at 229,376 tokens is close enough that the trade rarely pays.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads do exist under user namespaces, but check the generation before you pull one: of the fifteen namespaces an ollama.com search for "nanbeige" returns, twelve are Nanbeige4.1 and one is Nanbeige2-16B — different models that share the name prefix. Only two are 4.2. Use llama-server as above.

Anything else — or a real throughput measurement on this card — is welcome via the submission form.