Nanbeige4.2-3B on RTX 5060: the One 8 GB Card With a Real 65,536-Token Run Behind It

What You'll Build

A local llama-server endpoint running Nanbeige4.2-3B on one RTX 5060 with a 65,536-token window on a 4-bit KV cache — the configuration a community user has actually run on this exact card and posted the server log for. Every other 8 GB page for this model derives its budget and stops there. This one derives the same budget and then shows what the card did with it, including the part nobody advertises: decode throughput on this model falls sharply as the conversation grows, and the fall is measured, not theorised.

Hardware data: RTX 5060 (8 GB VRAM) · Q4_K_M weights 2.398 GiB + q4_0 KV at 65,536 tokens 3.094 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.250 GiB = 6.059 GiB accounted · community-reported 53.27 tok/s decode on a short conversation, 16.00 tok/s deep in context · See benchmark data

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2-3B is a Looped Transformer: the model card says "Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters.", and config.json sets num_loops: 2 next to num_hidden_layers: 22. The 22 blocks run twice per forward pass over one shared set of weights, and each pass keeps its own keys and values. llama.cpp expands the count before allocating anything — src/models/nanbeige.cpp sets hparams.n_layer_all = n_layer_phys * n_loops under the comment "Expand logical layer count before load_tensors() allocates layers / KV." and aliases the weights with "Share physical weights across loops; each slot still has its own KV index." — and src/llama-kv-cache.cpp then sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure computed from 22 layers is exactly half the truth.

The user who ran this card reached the same number independently: the entrypoint script he posted carries the comment "Nanbeige4.2-3B loops layers (num_loops=2 -> 44 KV layers),". The cost is deliberate on the vendor's side — on discussion #18 a Nanbeige team member writes: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2".

Requirements

Component	Minimum	This recipe
GPU	8 GB VRAM, CUDA compute capability 7.5+	RTX 5060 (8 GB) — the budget below is derived from file bytes and llama.cpp's allocation rules; the throughput figures are community-reported, not ours (/contribute)
RAM	8 GB system RAM	— (16 GB+ for the --no-kv-offload long-context mode)
Storage	2.57 GB for the Q4_K_M GGUF (decimal, as HuggingFace lists it)	2,574,807,904 bytes, from the HF tree API
Software	llama.cpp b10153+, CUDA toolkit 12.8+, CMake 3.18+	—

The toolkit floor is the strictest of any consumer card that runs this model, and it is not optional. NVIDIA's CUDA GPU Compute Capability table lists the GeForce RTX 5060 at 12.0, and llama.cpp's own ggml/src/ggml-cuda/CMakeLists.txt annotates that architecture with its requirement:

# 86     == RTX 3000, needs CUDA v11.1
# 89     == RTX 4000, needs CUDA v11.8
# 120    == Blackwell, needs CUDA v12.8, FP4 tensor cores

The same file only appends the 120a-real target when CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8", and a CUDA 12.4 toolkit that builds llama.cpp perfectly for older cards will not produce code for this one by either route. With the default GGML_NATIVE=ON — and CMake 3.24 or newer, which is the other half of the gate on native detection — the build hands the detected architecture straight to nvcc, and an old toolkit fails at compile time rather than degrading quietly. Below CMake 3.24 the fallback list runs instead, and it simply omits the Blackwell target. Install a 12.8-or-newer toolkit before anything else.

Do not follow the model card's llama.cpp instructions. The card still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release. Mainline merged native nanbeige support in PR #25994 on 2026-07-27, and issue #26086 was closed the same day. Release tag b10153 is that merge commit, so it is the earliest tag that works.

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Confirm the checkout carries the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q4_K_M GGUF

Nanbeige publishes no first-party GGUF: the org's ten public repositories include -FP8 and -GPTQ-Int8 builds of this model and nothing in GGUF form. This recipe uses owao/Nanbeige4.2-3B-GGUF; bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a wider alternative ladder. Pick one repository and stay inside it — the two quantizers' Q4_K_M builds differ by 109 MB, and mixing their numbers into one budget is how a sum goes quietly wrong.

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q4_K_M.gguf \
  --local-dir ./models

numpy is not a dependency of huggingface_hub, and the next step needs it.

3. Verify the artifact carries the loop parameter

Worth doing by hand exactly once. If a GGUF was produced without num_loops, llama.cpp defaults the key to 1, runs 22 layers instead of 44, and you get a silently different model — no error, no warning, half the depth.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q4_K_M.gguf | grep -E "num_loops|block_count"

You want two lines, ending = 22 and = 2 respectively:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case. (The leading index numbers vary from file to file; only the key names and the values matter.)

Two commands that look like they should do this job do not, so do not substitute them: llama-gguf <file> r n prints key names only, never their values, and llama-cli suppresses the loader's metadata dump at its default verbosity besides sitting in an interactive chat loop rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 65536 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  --parallel 1 \
  --temp 1.0 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. The sampler values are the model card's recommendation for agentic and tool-use tasks — the same row of its scenario table that asks for 65,536 new tokens. For reasoning and chat it recommends --temp 0.6.

Four flags are load-bearing:

• -c 65536, set explicitly. Omitting it is not neutral — llama.cpp's context-fitting default interpolates the window downward until the model fits free memory, so you silently get some other number and the budget below stops describing your process.
• -fa on. A quantized V cache requires Flash Attention; without it llama.cpp refuses outright.
• -ctk q4_0 -ctv q4_0 — matched types. A stock CUDA build instantiates FlashAttention kernels only for identical K and V types; ggml/src/ggml-cuda/fattn.cu returns BEST_FATTN_KERNEL_NONE when K->type != V->type outside GGML_CUDA_FA_ALL_QUANTS. A clever-looking -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• --parallel 1, deliberately. This matches the reported run below. It is not the default: with no flag, tools/server/server.cpp tests params.n_parallel < 0 and resolves it with params.n_parallel = 4; params.kv_unified = true; in the same branch, giving four slots over one shared pool of -c cells. Both shapes cost the same memory and both give a single conversation the full 65,536 tokens; --parallel 1 simply says out loud that you want one slot, and it is what the log below was produced with.

The run this page is built on

On discussion #18, community user thermi6 writes "For information, this is what I get with an RTX 5060 that has 8 GB of VRAM:" and posts his entrypoint plus a complete llama-server log. His configuration:

KV_TYPE_K=q4_0
KV_TYPE_V=q4_0
KV_OFFLOAD=1
MAX_CTX=65536

driving llama-server with --gpu-layers 99 --ctx-size 65536 --cache-type-k q4_0 --cache-type-v q4_0 --flash-attn on --no-mmap --parallel 1 --temp 0.7 --min-p 0.05. The server comes up and reports n_slots = 1, n_ctx_slot = 65536, kv_unified = 'false' — a full window in one slot, exactly as the --parallel 1 note above describes. His script's own annotation for this mode reads "KV_OFFLOAD=1 (default) keeps KV in VRAM: fastest, ctx up to ~64k."

Two honest caveats about using it. First, he does not say which repository his Q4_K_M came from (his file is named nanbeige-3b-q4_k_m.gguf), so the artifact's provenance is unstated. That does not change the conclusion: the two Q4_K_M builds in circulation are 2,574,807,904 and 2,684,023,968 bytes, and this configuration accounts to 6.059 GiB with the first and 6.160 GiB with the second — both inside the card. Second, his sampler settings are his own rather than the model card's, and the run is containerised — the /models/target and /opt/llama/build/bin paths, the 0.0.0.0 bind and the timestamp-prefixed log lines all say so — but he never states whether that card also drives a display, and his log reports no memory figure at all. Read the numbers as an existence proof plus a shape, not as a controlled benchmark, and read the headroom below as unclaimed rather than measured.

Results

• Speed — community-reported, on this exact card. From thermi6's log, three generations in one session, each with more of the conversation resident than the last:

  Generation	Resident tokens at release	Decode, first sample → last	Slot summary
  first	2,317	63.07 → 53.67 tok/s over 1,939 tokens	53.27 tokens per second
  second	10,922	20.56 → 20.20 tok/s over 465 tokens	20.14 tokens per second
  third	longer still	16.74 → 16.00 tok/s over 2,325 tokens	— (still generating when the log ends)

  Prompt processing in the same session is reported three times — 1991.19 tokens per second over 9,221 tokens, 2457.10 tokens per second over 302 tokens and 1348.39 tokens per second over 1,184 tokens — and those three do not line up into a trend: the fastest of them is the one issued against the most resident context. Prompt length and prefix reuse differ too much across the three to read anything out of them. Decode is the opposite, and there the pattern is unmistakable — decode throughput tracked the size of the resident conversation, not the configured window — and it is the thing to plan around on this model, because a 44-layer cache is re-read on every decoded token. This page reports the shape and does not assert a cause: the same author has since opened discussion #28 proposing a mechanism for the long-context slowdown, and that thread is open, unanswered by the team, and carries no measurement. If you can measure this pair under controlled conditions, please contribute the numbers so /check/nanbeige4-2-3b/rtx-5060 carries something better than one user's session.

• VRAM usage: 6.059 GiB accounted at the lead configuration — 2.398 GiB of Q4_K_M weights, 3.094 GiB of q4_0 cache at 65,536 tokens, 0.317 GiB of reserved logits and 0.250 GiB of FlashAttention dequant scratch — leaving 1.941 GiB of the card's 8 GiB for the CUDA context, the activation working set and any display attached to it. Derived from measured file bytes and llama.cpp's own allocation rules; the reported run corroborates that this configuration loads and serves on an 8 GB card, and reports no VRAM figure of its own.

• Quality notes: the model is ~4.17 B parameters total and ~3.15 B non-embedding — the "3B" in the name counts the non-embedding half, and the extra billion is the untied 166,144-token vocabulary at both ends of the stack. There is no published evaluation of this model under a quantized KV cache at any tier, so q4_0 is the aggressive choice on the ecosystem's general reputation, not on a measured quality difference for this architecture.

Where the budget comes from

llama.cpp sizes the cache as n_layer × (n_embd_k_gqa + n_embd_v_gqa) × bytes_per_element per token. config.json gives num_key_value_heads: 8 and head_dim: 128, so each layer stores 8 × 128 = 1024 elements for K and 1024 for V. Over 44 logical layers that is 90,112 elements per token:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

The block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 elements, from ggml/src/ggml-common.h. At 65,536 tokens that is 11.000 GiB of f16 cache, 5.844 GiB at q8_0 and 3.094 GiB at q4_0 — which is why the reported run is on q4_0 and could not have been on anything else.

Two further terms complete the bill.

A flat 0.317 GiB of reserved logits. At startup llama.cpp reserves one worst-case prompt-processing graph. src/llama-context.cpp builds it at const uint32_t n_tokens = std::min(cparams.n_ctx, cparams.n_ubatch); — 512 rows at stock settings — and sets the logit rows it must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), with n_outputs_max defaulting to n_batch. That is 512 rows, each a full f32 distribution over the 166,144-token vocabulary: 512 × 166,144 × 4 B = 340,262,912 B = 324.5 MiB. It does not move when you change -c; it halves when you halve -ub.

4,096 bytes per context token of FlashAttention dequant scratch — only because the cache is quantized. In ggml_cuda_flash_attn_ext_get_alloc_size the TILE and MMA_F16 kernels set need_f16_K = need_f16_V = true, and fattn-common.cuh then adds ggml_nelements(K) * ggml_type_size(GGML_TYPE_F16) for K and again for V, each behind if (need_f16_K && K->type != GGML_TYPE_F16). So an f16 cache pays none of it, and a quantized one pays (1024 + 1024) × 2 B per token — 0.250 GiB at 65,536. Note what this term is not: it is not a Blackwell tax. The reserve is 512 Q rows wide, which fails the compute-capability-8.9 test for the cheap vector kernel (Q->ne[1] <= 2), fails the pre-Ada test (Q->ne[1] == 1) and fails the trailing single-row fallback, so MMA_F16 is returned on every NVIDIA architecture alike. The term is a property of the reserved graph's shape, not of compute capability 12.0.

Adding all four terms, the practical envelope on 8 GiB:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Unallocated on 8 GiB
Q4_K_M · q4_0 · 65,536 (this recipe, as reported)	2.398	3.094	0.317	0.250	6.059 GiB	1.941 GiB
Q4_K_M · q8_0 · 32,768	2.398	2.922	0.317	0.125	5.762 GiB	2.238 GiB
Q4_K_M · q4_0 · 32,768	2.398	1.547	0.317	0.125	4.387 GiB	3.613 GiB
Q5_K_M · q4_0 · 65,536	2.782	3.094	0.317	0.250	6.443 GiB	1.557 GiB
Q4_K_M · q4_0 · 98,304	2.398	4.641	0.317	0.375	7.730 GiB	0.270 GiB — no room for the CUDA context
Q4_K_M · q8_0 · 65,536	2.398	5.844	0.317	0.250	8.809 GiB	will not fit
Q4_K_M · f16 · 32,768	2.398	5.500	0.317	0.000	8.215 GiB	will not fit

The last row is the trap: a reader who budgets from num_hidden_layers: 22 predicts 2.750 GiB of f16 cache at 32,768, concludes there is room to spare, and is wrong by a factor of two.

Two consumers sit on the same 8 GiB and are deliberately not in the accounted total, because nothing here measures them: the CUDA runtime's own per-process context, and your desktop compositor if this card drives a monitor. The reported run does not settle either of them: it proves this configuration loads and serves on an 8 GB card, but it publishes no memory figure and never says whether that card was also driving a display. So the 1.941 GiB is what those two consumers have to fit into, not confirmed slack — and on a desktop the compositor is the one that eats it. One further consumer lives outside VRAM entirely: the attention mask is n_kv × n_ubatch in f16, and src/llama-graph.cpp asserts ggml_backend_buffer_is_host(self_kq_mask->buffer) on it, so it is pinned system RAM.

Reaching the full 256K context

The card states that "The model supports a context length of up to 262,144 tokens (256K)." At 44 logical layers that cache is 12.375 GiB even at q4_0 — more than this entire card, before a single weight byte. The same user's script reaches the same conclusion in its comments: "so full 262k KV is ~18 GiB even quantized and cannot fit 8 GB VRAM." (his 18 GiB is an estimate in a shell comment; the 12.375 GiB here is computed from q4_0's exact 18-bytes-per-32-values block layout, and the two are an estimate and a derivation rather than a disagreement about the architecture). It stays reachable by keeping the weights on the GPU and the cache in system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q4_K_M.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM and trades decode speed for window size. Treat it as a capability, not a default.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-5060.

Troubleshooting

nvcc fatal: Unsupported gpu architecture 'compute_120'

Your CUDA toolkit predates 12.8. This is the one failure on this card with no workaround at the llama.cpp level: ggml/src/ggml-cuda/CMakeLists.txt only emits the 120a-real target when the detected toolkit is 12.8 or newer, and a native build asks nvcc to compile for the card it found. Install a 12.8-or-newer toolkit.

llama-server OOMs at 65,536 tokens

Check the cache flags first. At this context the default f16 cache is 11.000 GiB and q8_0 is 5.844 GiB — both exceed the card before any weights. Only -ctk q4_0 -ctv q4_0 fits.

If the flags are already right and you are a couple of hundred MiB short — the usual cause on a desktop being that this card is also driving your monitor, which the accounted total deliberately does not budget for — reach for -ub before you touch -c, because the logits reservation scales with the micro-batch and not with the window:

-ub	reserved logits	vs. default
512 (default)	0.317 GiB	—
256	0.158 GiB	−0.158 GiB
128	0.079 GiB	−0.238 GiB

It costs prompt-processing throughput, not context and not quality. It does not touch the 0.250 GiB of dequant scratch, which is sized by -c. Past that, every 16,384 tokens you give back returns 0.836 GiB between cache and scratch.

unknown model architecture: 'nanbeige'

The build predates b10153. Rebuild from mainline master, or take a release tag at or above b10153. Distribution channels lag: the model card notes that LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into the LM Studio backend directory.

Tool calls come back as plain text instead of executing

Fixed in mainline as of tag b10227 — check your build before debugging anything else. It was a llama.cpp parser bug, not a model bug and not your configuration: the model sometimes emits <tool_call> followed by a space rather than a newline, and the auto-generated parser required that newline verbatim, so the whole reply was consumed as content and the tool-call rule never ran. PR #26324 measured the loss at roughly a quarter of calls across 240 markers; the model's own discussion #17 reported roughly one call in six, though on the vendor nanbeige42 fork rather than on mainline. Both are community reports rather than a maintainer measurement, so read them as "some calls" rather than a constant. The fix arrived by a different route — PR #26252, a specialized parser merged on 2026-08-02 and released as tag b10227 — after which the author of #26324 reported tool calls working every time on a newer build and closed his own pull request unmerged on 2026-08-10. If calls still come back as text, your build predates b10227; rebuild. The failure bit hardest at the --temp 1.0 this recipe recommends — #17's small probe found every generation parsing at temperature 0.

response_format: json_schema fails before any token is generated

Grammar-constrained decoding aborts at sampler init with Failed to initialize samplers: Unexpected empty grammar stack. Two users reproduced it independently on discussion #6 — but both were running the vendor fork rather than the mainline build installed here, so treat the status on mainline as untested rather than known-broken. If you hit it, restart the server with --no-jinja; plain generation is unaffected either way.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads exist under user namespaces, but most search hits for "nanbeige" are 4.1 builds — a different model generation. Use llama-server as above.

The model loads but the output is subtly poor

Re-run step 3 and confirm the GGUF reports nanbeige.num_loops = 2. A file converted by a path that omits the key loads happily and runs at half depth. Also confirm the build post-dates 2026-07-27; older mainline builds refuse the architecture outright rather than misbehave.