Nanbeige4.2-3B on RTX 5060 Ti: the whole 16 GB context ladder on a 128-bit bus

What You'll Build

A llama-server endpoint running Nanbeige4.2-3B on an RTX 5060 Ti 16GB with a 98,304-token window and near-lossless quantization on both axes — Q8_0 weights and a q8_0 KV cache — for 13.588 GiB accounted on a card that has 16 GiB to give.

Hardware data: RTX 5060 Ti 16GB (16 GB VRAM, sm_120) · Q8_0 weights 4.130 GiB + q8_0 KV at 98,304 tokens 8.766 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.375 GiB = 13.588 GiB derived · no benchmark submitted yet · See benchmark data

This is the cheapest card in the catalogue that holds the whole 16 GB ladder, and the ladder is the point. NVIDIA's own comparison table gives this card 16 GB of GDDR7 on a 128-bit interface at 448 GB/sec, against 256-bit and 896–960 GB/sec on the RTX 5070 Ti and RTX 5080. That is a 2.14× spread in bandwidth over an identical memory budget — and every number on this page is a memory number, so every configuration below is available on this card at exactly the byte counts the more expensive 16 GB Blackwell cards get. What the narrow bus costs you is throughput, which nobody has measured for this model on any of the three.

Two things then decide whether the reader gets there: the KV cache, which is twice the size the model's config.json implies, and the CUDA toolkit, which has a hard floor on this generation that it does not have on Ada.

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2 is a Looped Transformer — the model card says its "Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters." config.json sets num_loops: 2, so the 22 blocks run twice per forward pass over one shared set of weights, and each pass keeps its own keys and values. llama.cpp expands the layer count before it allocates anything: src/models/nanbeige.cpp sets hparams.n_layer_all = n_layer_phys * n_loops under the comment "Expand logical layer count before load_tensors() allocates layers / KV.", and src/llama-kv-cache.cpp sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure computed from 22 layers is exactly half the truth. The vendor confirms the design is deliberate — team member leran1995 on discussion #10: "We have also investigated KV-cache sharing across loop passes, but the performance gains were notably smaller than with the full looped setup."

Requirements

Component	Minimum	This recipe
GPU	16 GB VRAM, CUDA compute capability 12.0	RTX 5060 Ti 16GB — not measured; the budget below is derived from file bytes and llama.cpp's own allocation rules (/contribute)
Toolkit	CUDA 12.8 or newer — see below	—
RAM	8 GB system RAM	—
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	4,434,787,168 bytes, from the HF tree API
Software	llama.cpp b10153 or newer, CMake 3.18+	the CUDA backend raises the floor: ggml/src/ggml-cuda/CMakeLists.txt opens with cmake_minimum_required(VERSION 3.18) # for CMAKE_CUDA_ARCHITECTURES, while the root project asks only 3.14

Ignore the model card's llama.cpp instructions. It still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release; mainline merged native nanbeige support in PR #25994 on 2026-07-27, and release b10153 is tagged at exactly that merge commit. The upstream feature request tracking it, issue #26086, was closed the same day. Mainline is also where the fixes land, so there is no reason to be on the fork any more.

The Blackwell part: your toolkit, not your llama.cpp

The RTX 5060 Ti is compute capability 12.0, and that is worth reading off a source rather than inferring from the model number: NVIDIA's CUDA GPUs table names this card by name, alongside the RTX 5070 Ti, 5080 and 5090, all at 12.0. NVIDIA's comparison table agrees from the other direction, giving its Shader Cores row as Blackwell for the whole 50-series column set. sm_120 is the one thing on this page that an Ada or Ampere reader never has to think about.

The runtime is not the gap. b10153 already treats Blackwell as a first-class target: ggml/src/ggml-cuda/common.cuh defines GGML_CUDA_CC_BLACKWELL 1200 at that commit, which clears turing_mma_available and enables the tensor-core Flash-Attention path this recipe depends on. You do not need a build newer than b10153 for this card — the same tag that first loads nanbeige already knows sm_120.

The toolkit is the gap. CUDA 12.8 is where nvcc learned this architecture: NVIDIA's 12.8 release notes open their New Features list with "This release adds compiler support for the following Nvidia Blackwell GPU architectures:" and then name SM_100, SM_101 and SM_120. llama.cpp encodes the same floor twice in ggml/src/ggml-cuda/CMakeLists.txt — once as a comment, "Blackwell, needs CUDA v12.8, FP4 tensor cores", and once as the guard that matters:

if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real)
endif()

Below 12.8 that list stops at 90-virtual and no sm_120 device code is produced at all. Check the toolkit before you check anything else:

nvcc --version   # must report release 12.8 or newer

This has one consequence that reaches readers who never build anything. llama.cpp's release job compiles the Windows CUDA binaries from a 12.4 / 13.3 matrix with -DGGML_NATIVE=OFF and no architecture override (.github/workflows/release.yml), so the default list above is exactly what ships: the CUDA 12.4 archive contains no Blackwell device code and the CUDA 13.3 archive does. Take the CUDA 13 asset.

The 16 GB question

Weights are not the constraint on this card at any tier — the largest GGUF on offer below bf16 is 4.130 GiB. The whole recipe is the cache.

KV cost per token

config.json gives num_key_value_heads: 8 and head_dim: 128, so each logical layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Over 44 logical layers that is 90,112 elements per token. The per-element cost comes from the block layouts in ggml/src/ggml-common.h — block_q8_0 is 34 bytes per 32 values, block_q4_0 is 18 bytes per 32 values:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

Multiplied out, this is the whole cache bill:

Context	f16 KV	q8_0 KV	q4_0 KV
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
98,304	16.500 GiB	8.766 GiB	4.641 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
196,608	33.000 GiB	17.531 GiB	9.281 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

What is not the cache, and how much VRAM to actually budget

Three things sit on top of weights + KV. Two are flat; the third is the one that decides this page, and it exists only if you quantize the cache.

The CUDA context is a fixed cost of a few hundred MiB that appears the moment the runtime touches the device.

The compute buffer is dominated, on this model, by a single reserved tensor: the logits. llama.cpp sizes its worst-case graph once at startup, and src/llama-context.cpp computes how many rows of logits that graph must hold as n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. With stock settings that is min(min(98304, 512), 2048) = 512 rows, each one a full f32 distribution over the vocabulary. This model's vocabulary is 166,144 tokens — several times a typical one — so:

512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB

That is the term worth budgeting, and the important property is that n_ubatch sets it, not -c. Doubling your context does not touch it; halving -ub halves it. Everything else in the buffer is per-microbatch activation working set — the widest single tensor is the FFN intermediate at 10752 × 512 × 4 B ≈ 21 MiB — which ggml's allocator reuses across the 44 unrolled layers rather than accumulating per layer.

The third term is the one nobody budgets for: a quantized KV cache is dequantized back to f16 before the prompt-processing FlashAttention kernel reads it, and the scratch for that lives in VRAM — permanently, even though only prefill uses it. ggml/src/ggml-cuda/fattn-common.cuh allocates it immediately past the destination tensor, and the guard is the whole story:

if (need_f16_K && K->type != GGML_TYPE_F16) {
    data.end = GGML_PAD(data.end, 128);
    data.K   = data.end;
    data.end += ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16);
}

with a near-identical block for V — its one extra branch is the second bullet below. Three details decide the size:

• It is skipped entirely for an f16 cache — K->type != GGML_TYPE_F16 is false, so the term is exactly zero. This is the only place on this page where the default cache is cheaper than a quantized one.
• V gets its own copy here. The code reuses K's buffer when V_is_K_view, but that branch is for architectures where K and V share storage. src/llama-kv-cache.cpp allocates this model's cache as two separate tensors — ggml_new_tensor_3d(ctx, type_k, n_embd_k_gqa, kv_size, n_stream) and a matching one for V — so both copies are charged.
• It is per-layer and transient, not ×44. ggml_nelements(K) is one layer's view over n_kv cells, and ggml's allocator reuses the space across layers.

So the cost is (n_embd_k_gqa + n_embd_v_gqa) × 2 B = (1024 + 1024) × 2 = 4,096 bytes per token of context, and unlike everything else in this section it grows with -c: 0.375 GiB at 98,304 tokens, 0.750 GiB at 196,608, 1.000 GiB at 262,144.

It is charged at the reserve, and Blackwell does not exempt you. The graph llama.cpp sizes at startup is the prompt-processing graph, with Q->ne[1] equal to n_ubatch — 512 by default. Every batch test in ggml/src/ggml-cuda/fattn.cu that would select the no-scratch vector kernel is a test for one or two rows, so at 512 the selector falls through to BEST_FATTN_KERNEL_MMA_F16, which sets need_f16_K and need_f16_V true. That is true on every NVIDIA compute capability with tensor cores, this one included; you avoid this term by not quantizing the cache, not by shrinking -ub.

Decode never touches it. At one row per token the same selector returns BEST_FATTN_KERNEL_VEC, whose flag is computed as need_f16_K = K->type == GGML_TYPE_F32 — false for a q8_0 or q4_0 cache — so token generation reads the quantized cache directly. The allocation stands anyway: llama.cpp reserves against the prompt-processing graph and the compute buffer does not shrink afterwards. The scratch is a prefill cost that you pay for in resident memory the whole time the server is up.

The attention mask is not on this list, and that is not an oversight. It is the obvious candidate — it really is n_kv × n_ubatch × 2 B under Flash Attention, which really would be 0.094 GiB at 98,304 tokens and would really grow with context. But it does not live in VRAM. llm_graph_input_attn_kv::set_input fills it from the CPU and asserts as much — src/llama-graph.cpp carries GGML_ASSERT(ggml_backend_buffer_is_host(self_kq_mask->buffer)); — so on CUDA it is a pinned host allocation. You still pay for it, in system RAM: 96.0 MiB at 98,304 tokens, 256.0 MiB at 262,144. Budget it there and not here.

So the budget rule for this card: the RTX 5060 Ti has 16 GiB of physical VRAM, and three terms this page cannot quantify sit on top of everything it can — the CUDA context, the per-microbatch activation working set, and whatever a desktop compositor is holding. Reserving 1.5 GiB for that trio leaves an accounted ceiling of 14.5 GiB, and that is the number every cell below is measured against. It is a chosen margin, not a measurement: run headless and you have more, run Windows with a browser open and you have less. The lead configuration lands at 13.588 GiB rather than pressing the ceiling, because the terms nobody here has measured are the ones you find out about by crashing.

Putting the three together, the per-token cost that actually governs a quantized cache is not the number in the table above but that number plus 4,096:

Cache type	KV bytes/token	+ dequant scratch	Effective
f16	180,224	0	180,224
q8_0	95,744	4,096	99,840
q4_0	50,688	4,096	54,784

In relative terms the tax is modest — it eats 4.8% of what q8_0 saves against f16 and 3.2% of what q4_0 saves. In absolute terms at the long end it is a full gigabyte, which is exactly where a 16 GB card's decisions get made.

The answer: the largest context each combination buys

Every cell is the largest multiple of 16,384 tokens whose weights + KV + the 0.317 GiB logits reservation + the dequant scratch stay under 14.5 GiB accounted. That is what "round" means here and it is the only ladder used on this page. Each cell reads context · GiB accounted, so what the configuration leaves unspent against the ceiling is visible rather than implied.

Weights	f16 KV	q8_0 KV	q4_0 KV
Q4_K_M (2.398 GiB)	65,536 · 13.715	114,688 · 13.379	229,376 · 14.418
Q5_K_M (2.782 GiB)	65,536 · 14.099	114,688 · 13.763	212,992 · 13.966
Q6_K (3.190 GiB)	49,152 · 11.757	114,688 · 14.171	212,992 · 14.374
Q8_0 (4.130 GiB)	49,152 · 12.697	98,304 · 13.588	196,608 · 14.478

The one cell that looks like an error is not one. Q6_K with an f16 cache stops at 49,152 and leaves 2.743 GiB unspent, because the next rung up — 65,536 tokens of f16 — lands at 14.507 GiB, over the ceiling by 6.8 MiB. That is 40 tokens' worth of cache: the exact maximum there is 65,496. One weight rung lighter, Q5_K_M reaches that same window with 0.401 GiB to spare. That cliff is what an unquantized cache costs at 180,224 bytes per token: it turns a 0.408 GiB difference in weights into a 16,384-token difference in window.

Read the table and two things fall out.

Yes, f16 KV becomes affordable at 16 GB — at 65,536 tokens, and only with Q4_K_M or Q5_K_M weights. That is a real threshold and it is worth naming, because at 180,224 bytes per token the untouched default cache is what puts small cards out of the game — 32,768 tokens of f16 is already 5.500 GiB of cache on its own, before any weights. Sixteen gigabytes is the first capacity where the default reaches 65,536, which is exactly the max-new-tokens the model card recommends for agentic and tool-use work. A reader who wants to type -ngl 99 -c 65536 and nothing else can now do that on this card.

And you should still not do it — though the margin is much narrower than the KV table alone suggests. Set the two candidates side by side with every term counted:

	Weights	KV	Logits	Dequant	Total
Q4_K_M + f16 @ 65,536	2.398	11.000	0.317	0.000	13.715 GiB
Q8_0 + q8_0 @ 98,304	4.130	8.766	0.317	0.375	13.588 GiB

The quantized-cache configuration wins by 0.127 GiB — a rounding error on a 16 GB card, and small enough that the unmeasured CUDA context could swallow it. On the KV numbers alone the gap looks like 0.502 GiB; the dequant scratch takes three quarters of that back, because the f16 option pays none of it and the q8_0 option pays it on half again as many tokens.

So the recommendation stands, but not for the reason it first appears to. Quantizing the cache is not a free lunch and it does not really save you memory here. What it buys, at the same ~13.6 GiB, is 1.5× the context window and a near-lossless weight tier instead of a 4-bit one. That is a good trade — it is simply a trade, not a windfall, and anyone who budgets q8_0 at its sticker price of 95,744 bytes per token will be about 0.4 GiB short at this context and a full gigabyte short at 262,144.

And if you are willing to take 4-bit weights after all, the same q8_0 cache buys 114,688 tokens at 13.379 GiB — 1.75× the f16 window for less total memory than either row above. That is a weight-precision decision rather than a cache one, and it is the axis this page deliberately does not choose for you.

What the 128-bit bus does not change, and what it might

Nothing above moves. A byte is a byte at 448 GB/sec, so the ladder on this page is the same ladder the RTX 5070 Ti and RTX 5080 get, cell for cell — those cards carry the same 16 GB of GDDR7 and reach the same 196,608-token in-VRAM ceiling. You are not buying a smaller window here; you are buying the same window on a narrower bus.

What the bus plausibly changes is how long each decoded token takes, and this is where the page has to stop asserting. Two facts are worth putting next to each other before you plan around either:

• The cache is read, not just stored. At the lead configuration there are 8.766 GiB of q8_0 cache resident, and a decoded token attends over all of it once the window is full. Halving the cache tier from q8_0 to q4_0 halves that traffic as well as the footprint — so on this card, unlike on the memory-only argument above, the cheaper cache rung has a second thing going for it.
• Nobody has measured any of this. There is no throughput figure for Nanbeige4.2-3B on the RTX 5060 Ti, the RTX 5070 Ti or the RTX 5080, so the 2.14× bandwidth spread between them has no published consequence. I am not going to convert bandwidth into tok/s for you; see Results for the one adjacent measurement that exists and why it is not this card's.

If you run this pair, that gap is one submission wide — contribute the numbers and /check/nanbeige4-2-3b/rtx-5060-ti stops being empty.

What 16 GB does not buy

The model's full declared window. The card states that "The model supports a context length of up to 262,144 tokens (256K).", and at 44 logical layers the cheapest cache that reaches it is q4_0 at 12.375 GiB — plus 1.000 GiB of dequant scratch, which at this context is no longer a rounding term. With Q8_0 weights the total is 17.822 GiB and with Q4_K_M weights 16.090 GiB: both are over the raw 16 GiB card, before the CUDA context is counted at all. There is no margin argument to have here and no weight tier that rescues it, and — since the memory budget is identical — no 16 GB Blackwell card rescues it either. The 32 GB RTX 5090 recipe is where that window lives.

The honest in-VRAM ceiling is 196,608 tokens, which every weight tier from Q4_K_M to Q8_0 reaches. Above that the ladder thins out and the accounted totals crowd the ceiling: Q5_K_M reaches 212,992 at 13.966 GiB, which is still a configuration with room in it; Q6_K reaches the same window at 14.374 and Q4_K_M reaches 229,376 at 14.418, leaving 0.126 and 0.082 GiB to spare under the 14.5 GiB ceiling. Treat those last two as the edge of the envelope rather than configurations to plan around — headless, and the first thing to give back when anything else on the card wants memory. 262,144 is reachable only by moving the cache to system RAM (see Troubleshooting).

Installation

1. Build llama.cpp with CUDA

nvcc --version   # must report release 12.8 or newer

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

The invocation is unchanged from any other CUDA card. With the default GGML_NATIVE=ON, CMake sets CMAKE_CUDA_ARCHITECTURES to native and compiles for the card in the machine, which is correct and fastest here. If you build in a container without the GPU visible, name the architecture instead:

cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120

Write 120, not 120a — the CUDA CMakeLists rewrites plain 12X entries to 12Xa itself, explaining that "Notably the Blackwell FP4 tensor core instructions are not forwards compatible and therefore need 12Xa." Compiling for a bare sm_120 target is what produces the ptxas failures reported in issue #19662.

Confirm the checkout actually knows the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q8_0 GGUF

There is no first-party GGUF. Enumerating all ten repositories under the Nanbeige org shows safetensors in the model repos and, for quantized builds, only -FP8 and -GPTQ-Int8 variants — no GGUF anywhere. This recipe uses owao/Nanbeige4.2-3B-GGUF, a 13-rung community conversion re-uploaded after the vendor's 2026-07-27 chat-template fix.

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q8_0.gguf \
  --local-dir ./models

numpy is there for step 3, not for the download — it is not a dependency of huggingface_hub and the dump script will fail without it.

Do not mix quantizers when you redo this arithmetic. bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a 23-rung alternative — its tree holds 25 .gguf files, but bf16 and imatrix are not quant rungs — and its files are not the same bytes: its Q4_K_M is 2,684,023,968 B (2.500 GiB) against owao's 2,574,807,904 B (2.398 GiB). Every weight figure on this page is owao's. The two repos' Q8_0 files happen to agree to within 320 bytes of metadata, but the K-quants do not.

3. Verify the artifact carries the loop parameter

This is the one check worth doing by hand. load_arch_hparams() reads num_loops as optional with a default of 1, so a GGUF converted by a path that omits the key loads without complaint, allocates 22 layers instead of 44, and runs a silently different model.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

You want two lines, ending = 22 and = 2 respectively:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case. (Index numbering will differ; only the names and values matter.)

Two commands that look like they should do this do not, so do not substitute them: llama-gguf <file> r n prints key names without their values, and llama-cli suppresses the loader's metadata dump at its default verbosity — that dump is emitted at trace level, so it needs -lv 4 — besides which llama-cli is an interactive chat client that waits for input rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 98304 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. The temperature is the model card's reasoning-and-chat row; top_p and top_k are the values in its own quickstart snippet and generation_config.json. For agentic and tool-use work the card recommends --temp 1.0.

Confirm the startup line reports 44 layers before trusting any of the arithmetic above:

llama_kv_cache: size = ... MiB (98304 cells,  44 layers,  4/1 seqs), K (q8_0): ..., V (q8_0): ...

Four flags carry weight here:

• -c 98304, set explicitly. Omitting -c is not neutral: fit_params defaults to true in common/common.h, so llama.cpp interpolates the context down until the model fits free memory with a 1 GiB margin — as low as fit_params_min_ctx, which is 4096. And -c 0 is the opposite trap: common/arg.cpp reads it as "give me the full trained context, do not shrink it" by setting fit_params_min_ctx = UINT32_MAX, which at the f16 default is a 44.000 GiB allocation and an immediate out-of-memory abort. Name the number you want.
• -fa on. A quantized V cache requires Flash Attention — src/llama-context.cpp throws quantized V cache was requested, but this requires Flash Attention otherwise. The runtime would enable it for you, but being explicit makes the failure legible. It also halves the attention mask, from F32 to F16 — which is a saving in host RAM rather than in VRAM, per the note above.
• -ctk q8_0 -ctv q8_0 — matched types. A stock CUDA build instantiates Flash Attention kernels only for identical K and V types: ggml/src/ggml-cuda/fattn.cu returns BEST_FATTN_KERNEL_NONE when K->type != V->type, guarded by #ifndef GGML_CUDA_FA_ALL_QUANTS, and the same file's vector-kernel table carries only f16/f16, q4_0/q4_0, q8_0/q8_0 and bf16/bf16 by default. A clever-looking asymmetric cache such as -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. There is no ×4 multiplier hiding in these numbers. common/arg.cpp sets params.n_parallel = -1 for the server, and tools/server/server.cpp resolves the sentinel with params.n_parallel = 4; params.kv_unified = true; in the same branch — unified meaning one shared pool of -c cells that a single conversation may consume in full. Passing an explicit positive --parallel N skips that branch, leaves the cache non-unified, and quarters the per-conversation window for identical memory. If you want one slot, write -np 1, not -np 4.

If you want the round 128K number

The model card recommends 131,072 max-new-tokens for reasoning and chat, and that window is reachable — by dropping the cache a rung rather than the weights. Keeping Q8_0 weights and moving to a q4_0 cache costs 11.135 GiB accounted, which is the most comfortable configuration on this page by some distance, and on a 128-bit card it is also the one that reads the least memory per token:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 131072 -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

The obvious-looking alternative — keep the q8_0 cache and drop the weights to Q4_K_M — does not fit: 14.902 GiB, over the ceiling, because q8_0 at 131,072 tokens carries 0.500 GiB of dequant scratch on top of its 11.688 GiB of cache. The table above stops that pairing one rung earlier, at 114,688. Between a 4-bit cache with 8-bit weights and an 8-bit cache with 4-bit weights, this card can only afford the first.

If you want maximum context in VRAM

196,608 tokens is the ceiling every weight tier reaches. Q6_K weights get there at 13.538 GiB accounted, with proper headroom; Q8_0 weights also fit, at 14.478 GiB, but leave only 0.022 GiB to spare under the 14.5 GiB ceiling — so Q6_K is the better buy at this context and is what is shown here.

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q6_K.gguf \
  -ngl 99 -c 196608 -fa on \
  -ctk q4_0 -ctv q4_0

That -c is deliberately one rung below Q6_K's own maximum in the table above — 212,992 at 14.374 GiB. The extra 16,384 tokens cost 0.836 GiB of exactly the headroom that covers the CUDA context, which is the term nobody here has measured. If you want that rung anyway, take it a weight tier lighter: Q5_K_M reaches 212,992 for 13.966 GiB, which is 0.408 GiB safer than Q6_K at the same window. Past that, Q4_K_M at 229,376 tokens for 14.418 GiB is the edge of the envelope, headless only.

preserve_thinking is a memory dial on this card

The chat template takes preserve_thinking, which controls whether reasoning from earlier assistant turns stays in the context — and therefore in the cache. leran1995 on discussion #9 gives both halves of the trade: "In our evaluations, enabling preserve_thinking generally provides better performance and better kv-cache reuse, so we recommend keeping it enabled when context length and memory allow.", and then "If you are working with a tight memory budget, disabling it can help reduce the KV-cache." At the lead configuration you are on the comfortable side of that line with 2.412 GiB unspent, so keep it on; at 196,608 and every rung above it you are not, and it becomes the first thing to turn off. Pass it through the OpenAI-compatible endpoint as "chat_template_kwargs": {"preserve_thinking": true}.

Results

• Speed: omitted. There is no throughput measurement of this model on an RTX 5060 Ti. The nearest thing in existence is on a different card, and the resemblance is close enough to be worth defusing explicitly: community user thermi6 posted llama-server logs on discussion #18, prefaced "For information, this is what I get with an RTX 5060 that has 8 GB of VRAM:" — 63.07 t/s decoding on a short prompt, 20.56 and then 16.74 t/s on two successively longer ones, running Q4_K_M weights with a q4_0 cache at 65,536 context and --parallel 1. That is the 8 GB RTX 5060, one character away from this page's card and one row away from it in NVIDIA's table, where it carries the same published 448 GB/sec on half the memory. It is still not a datapoint for the RTX 5060 Ti: different SKU, different die configuration, a lighter weight tier, a cheaper cache tier and a smaller window than anything recommended here. Nothing on this page is derived from it. If you run the configuration above, please contribute the numbers so /check/nanbeige4-2-3b/rtx-5060-ti stops being empty.
• VRAM usage: 13.588 GiB accounted at the lead configuration — 4.130 GiB of Q8_0 weights, 8.766 GiB of q8_0 cache, 0.317 GiB of reserved logits and 0.375 GiB of FlashAttention dequant scratch — leaving 2.412 GiB of the card's 16 GiB for the CUDA context, the activation working set and your desktop. Derived from measured file bytes and llama.cpp's own allocation rules, not measured on hardware. Separately, budget ~96 MiB of system RAM for the pinned attention mask.
• Quality notes: the model is ~4.17 B parameters in total, of which ~3.15 B are non-embedding; the "3B" in the name counts the non-embedding half, and the model card's own comparison table labels it that way. The total is HuggingFace's own tensor census of the repo — the model info API reports safetensors.total = 4,169,800,704 — and the extra billion is the untied 166,144-token vocabulary at both ends of the stack: 2 × 166,144 × 3,072 = 1,020,788,736 parameters, leaving 3,149,011,968. I found no evaluation of this model under a quantized KV cache at any tier — q8_0 is treated as near-lossless across the llama.cpp ecosystem generally, but that is an ecosystem prior, not a measurement of this model. The vendor publishes no throughput figures at all; the model card and the technical report carry quality benchmarks only.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-5060-ti.

Troubleshooting

Out of memory at startup

Check -ctk/-ctv first. At f16 the cache at 98,304 tokens is 16.500 GiB on its own — larger than the whole card — and that is the single most common way to be surprised by this model.

If the cache flags are already right and you are still a few hundred MiB short, reach for -ub before you touch -c. The compute buffer's dominant term is the logits reservation, and it is scaled by the micro-batch, not by the context:

-ub	reserved logits	vs. default
512 (default)	324.5 MiB	—
256	162.2 MiB	−162.3 MiB
128	81.1 MiB	−243.4 MiB
64	40.6 MiB	−283.9 MiB

./build/bin/llama-server -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 98304 -fa on -ctk q8_0 -ctv q8_0 -ub 256

This costs prompt-processing throughput — smaller micro-batches mean less work per kernel launch — and it costs nothing else. Decode speed and the context window are both untouched, which is why it is the right dial to turn first.

-ub does not touch the dequant scratch, though, so it tops out at about 284 MiB of relief. Past that the levers are the cache tier and the context, in that order — and note that dropping from q8_0 to q4_0 is worth more than the arithmetic in the KV table alone suggests, because the scratch is a flat 4,096 bytes per token on both, so going down a cache rung saves the full 45,056 bytes per token and adds nothing back. Cut -c last; it is the thing you came to this card for.

It builds, or it downloads, but the GPU sits idle

Check the toolkit before anything else. A build made with CUDA older than 12.8 emits no sm_120 code for this card, because 120a-real is appended only inside if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8"). Run nvcc --version; if you took a prebuilt Windows archive, make sure it was the CUDA 13 one and not the CUDA 12.4 one. A community report on a different model, issue #26205 — unconfirmed, no maintainer response, filed against an RTX 5060 Laptop — describes exactly that shape on the official 12.4 build: "the dedicated GPU utilization remains at 0, and the large model runs successfully entirely in CPU mode". Read it as corroboration of the build gate, not as a measurement of anything.

llama_model_load: error loading model architecture: unknown model architecture: 'nanbeige'

Your build predates b10153. Rebuild from mainline master or take a release tag at or above b10153 — that tag points at the merge commit of PR #25994. A user on the model's discussion #23 reports the same failure to load the architecture, and is answered by another who fixed it simply by taking a current release tag; the identical error text appears on the GGUF repo's discussion #1, from the pre-mainline period. Note that some distribution channels lag independently of your own build: the model card says LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into its backend directory.

Long-context decode feels slower than the cache size suggests

There is an open, unanswered community report worth knowing about before you plan a long-context workload here. thermi6 opened discussion #28 on 2026-08-10 claiming that "Currently any implementation with quantized kv cache still reads quantized fp16 instead of the quantized kv cache format." and that "That increases DRAM transfers by a factor of 4." — i.e. that a q4_0 cache buys footprint but not traffic. No maintainer has responded and no measurement accompanies it.

Read against mainline, the claim has a scope. The f16 expansion is real — it is the dequant scratch this page budgets — but the kernel that uses it is selected by batch width. At prompt processing the batch is n_ubatch rows and fattn.cu picks BEST_FATTN_KERNEL_MMA_F16, which does expand the cache to f16 first. At decode the batch is one row, and on Ada-and-newer the same selector returns BEST_FATTN_KERNEL_VEC for a quantized cache at Q->ne[1] <= 2; the vector kernel reads q4_0/q8_0 directly and sets no need_f16 flag. The prerequisite it needs is met here — src/llama-kv-cache.cpp pads n_kv to a multiple of 256, which is FATTN_KQ_STRIDE. So on this card the reported effect should land on prefill rather than on token generation. That is a reading of the selector, not a measurement; if your own numbers say otherwise, the thread and /contribute both want them.

Tool calls come back as plain text instead of executing

Fixed in mainline as of release tag b10227. The bug was llama.cpp's, not the model's and not your configuration: for a minority of calls the model emits <tool_call> followed by a space rather than a newline, and the auto-generated parser matched the marker with the newline attached, so the call was never recognised and the text came back as content. The reporter of PR #26324 put that at roughly 25% of calls; the model's discussion #17 and the GGUF repo's discussion #2 report the same defect independently. All of it is now pre-fix history.

What is still worth knowing is that the fix is not the pull request a search turns up first. #26324 was closed unmerged on 2026-08-10, a maintainer having objected that blanket whitespace trimming was reverted once before for degrading other models' output. The behaviour was repaired instead by PR #26252, a specialized parser merged on 2026-08-02 and shipped as tag b10227, and #26324's author closed his own PR on the strength of it: "Tried b10335 (#26252 was merged in b10227) and nanbeige tool calls work 100% of the times, thank you". This page therefore carries two build floors — b10153 for the architecture, b10227 for tool calls — and if calls still arrive as text the second one is what your build sits under. Rebuild rather than lowering the temperature.

response_format: json_schema fails at sampler init

Grammar-constrained decoding fails before a token is generated, with Failed to initialize samplers: Unexpected empty grammar stack after accepting piece: assistant (13886). Two users reproduced it independently on discussion #6, and both found --no-jinja clears it, while --chat-template chatml, strict: false and disabling reasoning all do not. Both reproductions ran builds predating mainline support, so read the status on a current build as untested rather than known-broken. The cost of --no-jinja is that you lose the model's own chat template and have to apply the prompt format yourself.

You need the full 262,144-token window on this card

It does not fit in 16 GiB at any cache quantization once the compute buffer is counted, and no other 16 GB Blackwell card changes that. Keep the weights on the GPU and move the cache to system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM for the cache and trades decode speed for window size — on a 128-bit card, where every token already crosses the narrowest bus of the three, expect that trade to bite harder than it would on the 256-bit siblings. Treat it as a capability, not a default: the in-VRAM edge at 229,376 tokens is close enough that it rarely pays.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads do exist under user namespaces, but check the generation before you pull one: of the fifteen entries an ollama.com search for "nanbeige" returns, only two name 4.2 — ten name Nanbeige4.1, two are unversioned nanbeige3b builds and one is Nanbeige2-16B, all different models that share the name prefix. Use llama-server as above.

Anything else — or a real throughput measurement on this card — is welcome via the submission form.