Nanbeige4.2-3B on RTX 4080: 96K Context, and What Each Token Costs in Bytes

What You'll Build

A llama-server endpoint running Nanbeige4.2-3B on an RTX 4080 with a 98,304-token window and near-lossless quantization on both axes — Q8_0 weights and a q8_0 KV cache — for 13.588 GiB accounted, inside a card that has 16 GiB to give.

Hardware data: RTX 4080 (16 GB VRAM) · Q8_0 weights 4.130 GiB + q8_0 KV at 98,304 tokens 8.766 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.375 GiB = 13.588 GiB derived · no benchmark submitted yet · See benchmark data

NVIDIA's RTX 4080 family specification table lists this card at 9728 CUDA cores and 49 shader TFLOPS on the Ada Lovelace architecture, against 16 GB GDDR6X on a 256-bit interface. Compute is not what this card is short of. Memory is — and on a 3-billion-parameter model that sounds absurd until you count the cache, at which point the whole page becomes an argument about how to spend sixteen gigabytes. This is not a "will it fit" recipe. It fits several ways. It is a recipe about which axis to spend the memory on — weight precision, cache precision, or context length — and, because the card has the compute to make long windows worth opening, about what each token inside the window costs to produce.

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2 is a Looped Transformer — the model card says its "Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters." config.json sets num_loops: 2, so the 22 blocks run twice per forward pass over one shared set of weights, and each pass keeps its own keys and values. llama.cpp expands the layer count before it allocates anything: src/models/nanbeige.cpp sets hparams.n_layer_all = n_layer_phys * n_loops under the comment "Expand logical layer count before load_tensors() allocates layers / KV.", and src/llama-kv-cache.cpp sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure computed from 22 layers is exactly half the truth. The vendor confirms the design is deliberate — a Nanbeige team member on discussion #10: "We have also investigated KV-cache sharing across loop passes, but the performance gains were notably smaller than with the full looped setup."

Requirements

Component	Minimum	This recipe
GPU	16 GB VRAM, CUDA	RTX 4080 — not measured; the budget below is derived from file bytes and llama.cpp's own allocation rules (/contribute)
RAM	8 GB system RAM	—
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	4,434,787,168 bytes, from the HF tree API
Software	llama.cpp b10153 or newer, CUDA toolkit (11.8+ for a native sm_89 build), CMake 3.18+	—

Ignore the model card's llama.cpp instructions. It still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release; mainline merged native nanbeige support in PR #25994 on 2026-07-27, and release b10153 is tagged at exactly that merge commit. The upstream feature request tracking it, issue #26086, was closed the same day. Mainline is also where the fixes land, so there is no reason to be on the fork any more.

Build target: sm_89, and the toolkit version that emits it

NVIDIA's CUDA GPU Compute Capability list places the GeForce RTX 4080 at 8.9, alongside the rest of the Ada Lovelace consumer line. That number is not trivia here: llama.cpp's CUDA backend branches on it directly — ggml/src/ggml-cuda/common.cuh defines GGML_CUDA_CC_ADA_LOVELACE as 890, and the FlashAttention kernel selector gates a decode-path decision on cc >= GGML_CUDA_CC_ADA_LOVELACE. See Which FlashAttention kernel you actually get, below.

The toolkit floor is real too. ggml/src/ggml-cuda/CMakeLists.txt only appends 89-real to CMAKE_CUDA_ARCHITECTURES under if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8"). On an older toolkit the card still runs, from a virtual architecture that is JIT-compiled on first load rather than from a native sm_89 binary. A default build targets native and sidesteps the question, which that same file selects under GGML_NATIVE with CUDA ≥ 11.6 and CMake ≥ 3.24; if you build for distribution, docs/build.md gives the explicit form, -DCMAKE_CUDA_ARCHITECTURES="86;89".

The 16 GB question

Weights are not the constraint on this card at any tier — the largest GGUF on offer below bf16 is 4.130 GiB. The whole recipe is the cache.

KV cost per token

config.json gives num_key_value_heads: 8 and head_dim: 128, so each logical layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Over 44 logical layers that is 90,112 elements per token. The per-element cost comes from the block layouts in ggml/src/ggml-common.h — block_q8_0 is 34 bytes per 32 values, block_q4_0 is 18 bytes per 32 values:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

Multiplied out, this is the whole cache bill:

Context	f16 KV	q8_0 KV	q4_0 KV
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
98,304	16.500 GiB	8.766 GiB	4.641 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
196,608	33.000 GiB	17.531 GiB	9.281 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

What is not the cache, and how much VRAM to actually budget

Three things sit on top of weights + KV. Two are flat; the third is the one that decides this page, and it exists only if you quantize the cache.

The CUDA context is a fixed cost of a few hundred MiB that appears the moment the runtime touches the device.

The compute buffer is dominated, on this model, by a single reserved tensor: the logits. llama.cpp sizes its worst-case graph once at startup, and src/llama-context.cpp computes how many rows of logits that graph must hold as n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. With stock settings that is min(min(98304, 512), 2048) = 512 rows, each one a full f32 distribution over the vocabulary. This model's vocabulary is 166,144 tokens — several times a typical one — so:

512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB

That is the term worth budgeting, and the important property is that n_ubatch sets it, not -c. Doubling your context does not touch it; halving -ub halves it. Everything else in the buffer is per-microbatch activation working set — the widest single tensor is the FFN intermediate at 10752 × 512 × 4 B ≈ 21 MiB — which ggml's allocator reuses across the 44 unrolled layers rather than accumulating per layer.

The third term is the one nobody budgets for: a quantized KV cache is dequantized back to f16 before the prompt-processing FlashAttention kernel reads it, and the scratch for that lives in VRAM. ggml/src/ggml-cuda/fattn-common.cuh allocates it immediately past the destination tensor, and the guard is the whole story:

if (need_f16_K && K->type != GGML_TYPE_F16) {
    data.end = GGML_PAD(data.end, 128);
    data.K   = data.end;
    data.end += ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16);
}

with a near-identical block for V — its one extra branch is the second bullet below. Three details decide the size:

• It is skipped entirely for an f16 cache — K->type != GGML_TYPE_F16 is false, so the term is exactly zero. This is the only place on this page where the default cache is cheaper than a quantized one.
• V gets its own copy here. The code reuses K's buffer when V_is_K_view, but that branch is for architectures where K and V share storage. src/llama-kv-cache.cpp allocates this model's cache as two separate tensors — ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream) and a matching one for V — so both copies are charged.
• It is per-layer and transient, not ×44. ggml_nelements(K) is one layer's view over n_kv cells, and ggml's allocator reuses the space across layers.

So the cost is (n_embd_k_gqa + n_embd_v_gqa) × 2 B = (1024 + 1024) × 2 = 4,096 bytes per token of context, and unlike everything else in this section it grows with -c: 0.375 GiB at 98,304 tokens, 0.750 GiB at 196,608, 1.000 GiB at 262,144.

It is charged at the reserve, not just when prompt processing actually runs — llama.cpp sizes the worst-case graph at -ub rows, and anything wider than 2 rows goes to a kernel that needs the f16 copy. You cannot avoid this term by shrinking -ub; you avoid it only by not quantizing the cache.

The attention mask is not on this list, and that is not an oversight. It is the obvious candidate — it really is n_kv × n_ubatch × 2 B under Flash Attention, which really would be 0.094 GiB at 98,304 tokens and would really grow with context. But it does not live in VRAM. llm_graph_input_attn_kv::set_input fills it from the CPU and asserts as much — src/llama-graph.cpp carries GGML_ASSERT(ggml_backend_buffer_is_host(self_kq_mask->buffer)); — so on CUDA it is a pinned host allocation. You still pay for it, in system RAM: 96.0 MiB at 98,304 tokens, 256.0 MiB at 262,144. Budget it there and not here.

So the budget rule for this card: treat 14.5 GiB as the accounted ceiling on a 16 GiB card. That is not a measurement, and nothing on this page pretends otherwise — it is 16 GiB of physical memory minus roughly 1.5 GiB held back for the three terms above that no citation covers: the CUDA context, the activation working set, and whatever a desktop compositor holds on a card that is also driving your monitors. Run headless and the reserve is generous; run Windows with a browser open and it is about right. The lead configuration below lands at 13.588 GiB rather than pressing that ceiling, because the terms nobody here has measured are the ones you find out about by crashing.

Putting the three together, the per-token cost that actually governs a quantized cache is not the number in the table above but that number plus 4,096:

Cache type	KV bytes/token	+ dequant scratch	Effective
f16	180,224	0	180,224
q8_0	95,744	4,096	99,840
q4_0	50,688	4,096	54,784

In relative terms the tax is modest — it eats 4.8% of what q8_0 saves against f16 and 3.2% of what q4_0 saves. In absolute terms at the long end it is a full gigabyte, which is exactly where a 16 GB card's decisions get made.

The answer: the largest context each combination buys

Every cell is the largest multiple of 16,384 tokens whose weights + KV + the 0.317 GiB logits reservation + the dequant scratch stay under 14.5 GiB accounted. That is what "round" means here and it is the only ladder used on this page. Each cell reads context · GiB accounted, so what the configuration leaves unspent against the 14.5 GiB ceiling is visible rather than implied. Every one of the twelve cells is a true maximum, not merely a value that fits: the next rung of 16,384 tokens busts the ceiling in all twelve cases, by 0.007 GiB in the tightest (Q6_K with f16) and by 0.302 GiB or more everywhere else.

Weights	f16 KV	q8_0 KV	q4_0 KV
Q4_K_M (2.398 GiB)	65,536 · 13.715	114,688 · 13.379	229,376 · 14.418
Q5_K_M (2.782 GiB)	65,536 · 14.099	114,688 · 13.763	212,992 · 13.966
Q6_K (3.190 GiB)	49,152 · 11.757	114,688 · 14.171	212,992 · 14.374
Q8_0 (4.130 GiB)	49,152 · 12.697	98,304 · 13.588	196,608 · 14.478

The one cell that looks like an error is not one. Q6_K with an f16 cache stops at 49,152 and leaves 2.743 GiB unspent, because the next rung up — 65,536 tokens of f16 — lands at 14.507 GiB, over the ceiling by 6.8 MiB. That is 40 tokens' worth of cache: the exact maximum there is 65,496. One weight rung lighter, Q5_K_M reaches that same window with 0.401 GiB to spare. That cliff is what an unquantized cache costs at 180,224 bytes per token: it turns a 0.408 GiB difference in weights into a 16,384-token difference in window.

Read the table and two things fall out.

Yes, f16 KV becomes affordable at 16 GB — at 65,536 tokens, and only with Q4_K_M or Q5_K_M weights. That is a real threshold and it is worth naming, because at 180,224 bytes per token the untouched default cache is what puts smaller cards out of the game — 32,768 tokens of f16 is already 5.500 GiB of cache on its own, before any weights. Sixteen gigabytes is the first capacity where the default reaches 65,536, which is exactly the max-new-tokens the model card recommends for agentic and tool-use work. A reader who wants to type -ngl 99 -c 65536 and nothing else can now do that on this card.

And you should still not do it — though the margin is much narrower than the KV table alone suggests. Set the two candidates side by side with every term counted:

	Weights	KV	Logits	Dequant	Total
Q4_K_M + f16 @ 65,536	2.398	11.000	0.317	0.000	13.715 GiB
Q8_0 + q8_0 @ 98,304	4.130	8.766	0.317	0.375	13.588 GiB

The quantized-cache configuration wins by 0.127 GiB — a rounding error on a 16 GB card, and small enough that the unmeasured CUDA context could swallow it. On the KV numbers alone the gap looks like 0.502 GiB; the dequant scratch takes three quarters of that back, because the f16 option pays none of it and the q8_0 option pays it on half again as many tokens.

So the recommendation stands, but not for the reason it first appears to. Quantizing the cache is not a free lunch and it does not really save you memory here. What it buys, at the same ~13.6 GiB, is 1.5× the context window and a near-lossless weight tier instead of a 4-bit one. That is a good trade — it is simply a trade, not a windfall, and anyone who budgets q8_0 at its sticker price of 95,744 bytes per token will be about 0.4 GiB short at this context and a full gigabyte short at 262,144.

And if you are willing to take 4-bit weights after all, the same q8_0 cache buys 114,688 tokens at 13.379 GiB — 1.75× the f16 window for less total memory than either row above. That is a weight-precision decision rather than a cache one, and it is the axis this page deliberately does not choose for you.

What a token costs in bytes

Everything above is a capacity argument, and capacity arguments are the same on every 16 GB card — the twelve cells are built from this model's per-token costs, the GGUF file sizes and 16 GiB, and from nothing about the silicon. On a card with this much compute, the more interesting question is what happens inside a window once you have opened it.

Nanbeige4.2-3B uses full attention at every one of its 44 logical layers — config.json declares no sliding window and no attention sink — so a decode step reads the weights once and every filled cell of the cache once. Per-token memory traffic is therefore a sum of two numbers this page has already derived, and here it is for all twelve cells of the ladder:

Weights	f16 KV	q8_0 KV	q4_0 KV
Q4_K_M	13.398 GiB · 82.1%	12.625 GiB · 81.0%	13.226 GiB · 81.9%
Q5_K_M	13.782 GiB · 79.8%	13.008 GiB · 78.6%	12.837 GiB · 78.3%
Q6_K	11.440 GiB · 72.1%	13.416 GiB · 76.2%	13.244 GiB · 75.9%
Q8_0	12.380 GiB · 66.6%	12.896 GiB · 68.0%	13.411 GiB · 69.2%

Each cell is bytes read per decoded token at that cell's maximum window · the share of it that is cache. The result is the most useful thing on this page, and it is not what anyone expects:

Every configuration costs about the same per token. The twelve cells span 11.440 to 13.782 GiB — a 1.20× spread — while the windows they buy span 49,152 to 229,376 tokens, a 4.67× spread. That is not a coincidence, it is the definition: each cell spends the same 14.5 GiB ceiling, most of the ceiling is cache, and the cache is exactly what a decode step re-reads. Choosing a rung of this ladder is therefore not choosing a speed. It is choosing what one token's worth of memory traffic buys you — and on that basis Q8_0 weights with a q8_0 cache at 98,304 tokens is a strong cell, because it spends 68.0% of its traffic on cache rather than 82.1%, and it does it at the highest weight precision on the page.

The q4_0 cache is not "the cheap one" at its own maximum context. It looks cheap in the bytes-per-token table above — 50,688 against 95,744 — and that saving is entirely real at a fixed window: hold the window at 98,304 and Q8_0 weights, and dropping q8_0 to q4_0 cuts per-token traffic from 12.896 GiB to 8.771 GiB, a 32.0% reduction. But the ladder does not hold the window fixed; it spends the saving on more context, and more context is more cache to re-read. If you want fewer bytes per token, shorten the window — that is the only lever that actually reduces the traffic rather than reinvesting it.

The lightest cell on the page is the one that wastes capacity. Q6_K with an f16 cache at 49,152 tokens moves 11.440 GiB per token, less than anything else here, precisely because it leaves 2.743 GiB of the ceiling unspent. That is the honest shape of the trade-off, and it is worth seeing written down before you assume the top of the ladder is free.

What this page will not do is turn those byte counts into tokens per second. That conversion needs a measurement, nobody has published one for this pair, and estimating throughput from an interface width is how a plausible number becomes a wrong one. The bytes are arithmetic; the seconds are not. If you run any of these cells, contribute the numbers — /check/nanbeige4-2-3b/rtx-4080 is empty until someone does.

What 16 GB does not buy

The model's full declared window. The card states that "The model supports a context length of up to 262,144 tokens (256K).", and at 44 logical layers the cheapest cache that reaches it is q4_0 at 12.375 GiB — plus 1.000 GiB of dequant scratch, which at this context is no longer a rounding term. With Q8_0 weights the total is 17.822 GiB and with Q4_K_M weights 16.090 GiB: both are over the raw 16 GiB card, before the CUDA context is counted at all. There is no margin argument to have here and no weight tier that rescues it.

The honest in-VRAM ceiling is 196,608 tokens, which every weight tier from Q4_K_M to Q8_0 reaches. Above that the ladder thins out and the accounted totals crowd the ceiling: Q5_K_M reaches 212,992 at 13.966 GiB, which is still a configuration with room in it; Q6_K reaches the same window at 14.374 and Q4_K_M reaches 229,376 at 14.418, leaving 0.126 and 0.082 GiB to spare under the 14.5 GiB ceiling. Treat those last two as the edge of the envelope rather than configurations to plan around — headless, and the first thing to give back when anything else on the card wants memory. 262,144 is reachable only by moving the cache to system RAM (see Troubleshooting).

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Confirm the checkout actually knows the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q8_0 GGUF

There is no first-party GGUF. Enumerating all ten repositories under the Nanbeige org shows safetensors in the model repos and, for quantized builds, only -FP8 and -GPTQ-Int8 variants — no GGUF anywhere. This recipe uses owao/Nanbeige4.2-3B-GGUF, a 13-rung community conversion re-uploaded after the vendor's 2026-07-27 chat-template fix.

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q8_0.gguf \
  --local-dir ./models

numpy is there for step 3, not for the download — it is not a dependency of huggingface_hub and the dump script will fail without it.

Do not mix quantizers when you redo this arithmetic. bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a 23-rung alternative and its files are not the same bytes: its Q4_K_M is 2,684,023,968 B (2.500 GiB) against owao's 2,574,807,904 B (2.398 GiB). Every weight figure on this page is owao's. The two repos' Q8_0 files happen to agree to within 320 bytes of metadata, but the K-quants do not.

3. Verify the artifact carries the loop parameter

This is the one check worth doing by hand. load_arch_hparams() reads num_loops as optional with a default of 1, so a GGUF converted by a path that omits the key loads without complaint, allocates 22 layers instead of 44, and runs a silently different model.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

You want two lines, ending = 22 and = 2 respectively:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case. (Index numbering will differ; only the names and values matter.)

Two commands that look like they should do this do not, so do not substitute them: llama-gguf <file> r n prints key names without their values, and llama-cli suppresses the loader's metadata dump at its default verbosity — that dump is emitted at trace level, so it needs -lv 4 — besides which llama-cli is an interactive chat client that waits for input rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 98304 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. The sampler values are the model card's recommendation for reasoning and chat; for agentic and tool-use work it recommends --temp 1.0.

Four flags carry weight here:

• -c 98304, set explicitly. Omitting -c is not neutral: fit_params defaults to true in common/common.h, so llama.cpp interpolates the context down until the model fits free memory with a 1 GiB margin — as low as fit_params_min_ctx, which is 4096. And -c 0 is the opposite trap: common/arg.cpp reads it as "give me the full trained context, do not shrink it" by setting fit_params_min_ctx = UINT32_MAX, which at the f16 default is a 44.000 GiB allocation and an immediate out-of-memory abort. Name the number you want.
• -fa on. A quantized V cache requires Flash Attention; the runtime will enable it for you and log that it did, but being explicit makes the failure legible. It also halves the attention mask, from F32 to F16 — which is a saving in host RAM rather than in VRAM, per the note above.
• -ctk q8_0 -ctv q8_0 — matched types. A stock CUDA build instantiates Flash Attention kernels only for identical K and V types: ggml/src/ggml-cuda/fattn.cu returns BEST_FATTN_KERNEL_NONE when K->type != V->type, guarded by #ifndef GGML_CUDA_FA_ALL_QUANTS. A clever-looking asymmetric cache such as -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. There is no ×4 multiplier hiding in these numbers. common/arg.cpp sets params.n_parallel = -1 for the server, and tools/server/server.cpp resolves the sentinel with params.n_parallel = 4; params.kv_unified = true; in the same branch — unified meaning one shared pool of -c cells that a single conversation may consume in full. Passing an explicit positive --parallel N skips that branch, leaves the cache non-unified, and quarters the per-conversation window for identical memory. If you want one slot, write -np 1, not -np 4.

Which FlashAttention kernel you actually get

Worth knowing, because it runs the opposite way from what "quantized cache" suggests and it is where this card's compute capability stops being trivia.

ggml/src/ggml-cuda/fattn.cu picks a kernel per attention call from the shape of Q. For a quantized K and V, the branch reads if (cc >= GGML_CUDA_CC_ADA_LOVELACE) { if (Q->ne[1] <= 2) return BEST_FATTN_KERNEL_VEC; } with an else giving pre-Ada cards the same path only at exactly one row. This card clears that gate. And for the vector kernel the scratch guard is need_f16_K = K->type == GGML_TYPE_F32 — false for q8_0 and for q4_0.

Put together: at decode the quantized cache is read in its quantized form. The dequantization discussed above is a prompt-processing cost — real, and permanently reserved because the worst-case graph is sized at -ub = 512 rows, which lands on BEST_FATTN_KERNEL_MMA_F16 where need_f16_K and need_f16_V are both set unconditionally — but it is not a toll on every generated token.

That distinction is live rather than settled. A community thread opened on 2026-08-10 argues the opposite for long-context inference — "That increases DRAM transfers by a factor of 4. That makes long context inference incredibly slow." — with no measurement, no named card, and no maintainer response at the time of writing; its author reports from an 8 GB card elsewhere in the same repo. Read the selector before you act on either version, and treat the question as open.

If you want the round 128K number

The model card recommends 131,072 max-new-tokens for reasoning and chat, and that window is reachable — by dropping the cache a rung rather than the weights. Keeping Q8_0 weights and moving to a q4_0 cache costs 11.135 GiB accounted, which is the most comfortable configuration on this page by some distance:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 131072 -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

The obvious-looking alternative — keep the q8_0 cache and drop the weights to Q4_K_M — does not fit: 14.902 GiB, over the ceiling, because q8_0 at 131,072 tokens carries 0.500 GiB of dequant scratch on top of its 11.688 GiB of cache. The table above stops that pairing one rung earlier, at 114,688. Between a 4-bit cache with 8-bit weights and an 8-bit cache with 4-bit weights, this card can only afford the first.

If you want maximum context in VRAM

196,608 tokens is the ceiling every weight tier reaches. Q6_K weights get there at 13.538 GiB accounted, with proper headroom; Q8_0 weights also fit, at 14.478 GiB, but leave only 0.022 GiB to spare under the 14.5 GiB ceiling — so Q6_K is the better buy at this context and is what is shown here. q4_0 is the aggressive rung of the cache ladder — this is the configuration to reach for when the workload is long-document or long-trajectory and you can tolerate that.

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q6_K.gguf \
  -ngl 99 -c 196608 -fa on \
  -ctk q4_0 -ctv q4_0

That -c is deliberately one rung below Q6_K's own maximum in the table above — 212,992 at 14.374 GiB. The extra 16,384 tokens cost 0.836 GiB of exactly the headroom that covers the CUDA context, which is the term nobody here has measured. If you want that rung anyway, take it a weight tier lighter: Q5_K_M reaches 212,992 for 13.966 GiB, which is 0.408 GiB safer than Q6_K at the same window. Past that, Q4_K_M at 229,376 tokens for 14.418 GiB is the edge of the envelope, headless only.

preserve_thinking is a memory dial on this card

The chat template takes preserve_thinking, which controls whether reasoning from earlier assistant turns stays in the context — and therefore in the cache, and therefore in the per-token traffic table above. A Nanbeige team member on discussion #9 gives both halves of the trade: "In our evaluations, enabling preserve_thinking generally provides better performance and better kv-cache reuse, so we recommend keeping it enabled when context length and memory allow.", and then "If you are working with a tight memory budget, disabling it can help reduce the KV-cache." At the lead configuration you are on the comfortable side of that line with 2.412 GiB unspent, so keep it on; at 196,608 and every rung above it you are not, and it becomes the first thing to turn off. Pass it through the OpenAI-compatible endpoint as "chat_template_kwargs": {"preserve_thinking": true}.

Results

• Speed: omitted. There is no throughput measurement of this model on this card, or on any 16 GB Ada card, on any surface searched: all 28 discussion threads on the canonical repo (each fetched individually through the JSON API rather than searched), the discussions on both community GGUF repos, and llama.cpp's issue tracker. The only throughput figures anywhere in that corpus are on an RTX 5060 with 8 GB in discussion #18 — a different capacity and a newer generation, so it is neither a floor nor a ceiling for this card and is deliberately not carried onto this page. If you run this pair, please contribute the numbers so /check/nanbeige4-2-3b/rtx-4080 stops being empty.
• VRAM usage: 13.588 GiB accounted at the lead configuration — 4.130 GiB of Q8_0 weights, 8.766 GiB of q8_0 cache, 0.317 GiB of reserved logits and 0.375 GiB of FlashAttention dequant scratch — leaving 2.412 GiB of the card's 16 GiB for the CUDA context, the activation working set and your desktop. Derived from measured file bytes and llama.cpp's own allocation rules, not measured on hardware. Separately, budget ~96 MiB of system RAM for the pinned attention mask.
• Quality notes: the model is ~4.17 B parameters in total, of which ~3.15 B are non-embedding; the "3B" in the name counts the non-embedding half, and the model card's own comparison table labels it that way. The total is HuggingFace's own tensor census of the repo — the model info API reports safetensors.total = 4,169,800,704 — and the extra billion is the untied 166,144-token vocabulary at both ends of the stack: 2 × 166,144 × 3,072 = 1,020,788,736 parameters, leaving 3,149,011,968. I found no evaluation of this model under a quantized KV cache at any tier in the spaces listed above — q8_0 is treated as near-lossless across the llama.cpp ecosystem generally, but that is an ecosystem prior, not a measurement of this model. The vendor publishes no throughput figures at all; the model card and the technical report carry quality benchmarks only.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-4080.

Troubleshooting

Out of memory at startup

Check -ctk/-ctv first. At f16 the cache at 98,304 tokens is 16.500 GiB on its own — larger than the whole card — and that is the single most common way to be surprised by this model.

If the cache flags are already right and you are still a few hundred MiB short, reach for -ub before you touch -c. The compute buffer's dominant term is the logits reservation, and it is scaled by the micro-batch, not by the context:

-ub	reserved logits	vs. default
512 (default)	324.5 MiB	—
256	162.2 MiB	−162.3 MiB
128	81.1 MiB	−243.4 MiB
64	40.6 MiB	−283.9 MiB

./build/bin/llama-server -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 98304 -fa on -ctk q8_0 -ctv q8_0 -ub 256

This costs prompt-processing throughput — smaller micro-batches mean less work per kernel launch, and on a card with 9728 shaders that is the one place the silicon is genuinely being used — and it costs nothing else. Decode speed and the context window are both untouched, which is why it is still the right dial to turn first: it trades a resource this card has for one it does not.

-ub does not touch the dequant scratch, though, so it tops out at about 284 MiB of relief. Past that the levers are the cache tier and the context, in that order — and note that dropping from q8_0 to q4_0 is worth more than the arithmetic in the KV table alone suggests, because the scratch is a flat 4,096 bytes per token on both, so going down a cache rung saves the full 45,056 bytes per token and adds nothing back. Cut -c last; it is the thing you came to this card for.

llama_model_load: error loading model architecture: unknown model architecture: 'nanbeige'

Your build predates b10153. Rebuild from mainline master or take a release tag at or above b10153 — that tag points at the merge commit of PR #25994. The same symptom is reported on the model's discussion #23 and on the GGUF repo's discussion #1. Only the latter is from the pre-mainline period — it dates from 2026-07-21; #23 was opened 2026-07-30, three days after the merge, by someone whose own build was older. Note that some distribution channels lag independently of your own build: the model card says LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into its backend directory.

Tool calls come back as plain text instead of executing

Fixed in mainline as of release tag b10227. The bug was llama.cpp's, not the model's and not your configuration: for a minority of calls the model emits <tool_call> followed by a space rather than a newline, and the auto-generated parser matched the marker with the newline attached, so the call was never recognised and the text came back as content. The reporter of PR #26324 put that at roughly 25% of calls; the model's discussion #17 and the GGUF repo's discussion #2 report the same defect independently. All of it is now pre-fix history.

What is still worth knowing is that the fix is not the pull request a search turns up first. #26324 was closed unmerged on 2026-08-10, a maintainer having objected that blanket whitespace trimming was reverted once before for degrading other models' output. The behaviour was repaired instead by PR #26252, a specialized parser merged on 2026-08-02 and shipped as tag b10227, and #26324's author closed his own PR on the strength of it: "Tried b10335 (#26252 was merged in b10227) and nanbeige tool calls work 100% of the times, thank you". This page therefore carries two build floors — b10153 for the architecture, b10227 for tool calls — and if calls still arrive as text the second one is what your build sits under. Rebuild rather than lowering the temperature.

response_format: json_schema fails at sampler init

Grammar-constrained decoding fails before a token is generated, with Failed to initialize samplers: Unexpected empty grammar stack after accepting piece: assistant (13886). Two users reproduced it independently on discussion #6, and both found --no-jinja clears it, while --chat-template chatml, strict: false and disabling reasoning all do not. Both reproductions ran builds predating mainline support, so read the status on a current build as untested rather than known-broken. The cost of --no-jinja is that you lose the model's own chat template and have to apply the prompt format yourself.

You need the full 262,144-token window on this card

It does not fit in 16 GiB at any cache quantization once the compute buffer is counted. Keep the weights on the GPU and move the cache to system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM for the cache — 12.375 GiB at q4_0, plus 256.0 MiB for the pinned mask. Read it against the traffic table above before you reach for it: the same 12.375 GiB now crosses PCIe on every decoded token instead of being read from GDDR6X, which is a different bus and a different order of magnitude. Treat it as a capability, not a default — on this card the in-VRAM edge at 229,376 tokens is close enough to 262,144 that the trade rarely pays.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads do exist under user namespaces, but check the generation before you pull one: of the fifteen namespaces an ollama.com search for "nanbeige" returns, ten are Nanbeige4.1, two carry no version in their slug at all, and one is Nanbeige2-16B — different models that share the name prefix. Only two are 4.2. Use llama-server as above.

Anything else — or a real throughput measurement on this card — is welcome via the submission form.