Nanbeige4.2-3B on RTX 4060 Ti 8GB: an 8 GB Budget for 65,536 New Tokens

What You'll Build

A local llama-server endpoint running Nanbeige4.2-3B on one RTX 4060 Ti 8GB with a 65,536-token context window, weights and cache both resident in the card's 8 GB. The number comes from the model card's agentic row, where it is a Max New Tokens figure rather than a window — so 65,536 is the floor a full agentic generation needs, and the prompt has to share the window with it. Reaching that number on 8 GB is not free and not automatic: this model bills its KV cache at twice the rate its config file implies, so 65,536 tokens fits only if you also drop the cache to 4-bit. This page is the arithmetic behind that trade, done in bytes.

Hardware data: RTX 4060 Ti 8GB (8 GB VRAM) · Q4_K_M weights 2.398 GiB + q4_0 KV at 65,536 tokens 3.094 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.250 GiB = 6.059 GiB accounted · no benchmark submitted yet · See benchmark data

⚠️ Check which SKU you have before you read any number on this page. NVIDIA sells the RTX 4060 Ti in two memory capacities. Everything here assumes the 8 GB card. Run nvidia-smi --query-gpu=name,memory.total --format=csv first; if it does not report roughly 8192 MiB, this budget is not yours — the 16 GB variant is a separate page at /check/nanbeige4-2-3b/rtx-4060-ti-16gb. On this model the difference is not cosmetic, because context is the only thing the extra capacity buys and context is the expensive axis.

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2-3B is a Looped Transformer: the model card says "Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters.", and config.json sets num_loops: 2 next to num_hidden_layers: 22. The 22 blocks run twice per forward pass over one shared set of weights, and each pass keeps its own keys and values. llama.cpp expands the count before allocating anything — src/models/nanbeige.cpp sets hparams.n_layer_all = n_layer_phys * n_loops under the comment "Expand logical layer count before load_tensors() allocates layers / KV." and aliases the weights with "Share physical weights across loops; each slot still has its own KV index." — and src/llama-kv-cache.cpp then sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure computed from 22 layers is exactly half the truth.

The cost is deliberate. On discussion #18 a Nanbeige team member writes: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2", and on discussion #10 the same member adds that "In future versions, we plan to mitigate the KV-cache overhead through linear and sparse attention mechanisms." — i.e. the fix is a future architecture, not a flag you are missing.

Requirements

Component	Minimum	This recipe
GPU	8 GB VRAM, CUDA compute capability 7.5+	RTX 4060 Ti 8GB — not measured; the budget below is derived from file bytes and llama.cpp's own allocation rules (/contribute)
RAM	8 GB system RAM	— (16 GB+ for the --no-kv-offload long-context mode)
Storage	2.57 GB for the Q4_K_M GGUF (decimal, as HuggingFace lists it)	2,574,807,904 bytes, from the HF tree API
Software	llama.cpp b10153+, CUDA toolkit 11.8+, CMake 3.18+	—

NVIDIA's CUDA GPU Compute Capability table lists the GeForce RTX 4060 Ti at 8.9, and llama.cpp's own ggml/src/ggml-cuda/CMakeLists.txt annotates that architecture with its toolkit floor:

# 86     == RTX 3000, needs CUDA v11.1
# 89     == RTX 4000, needs CUDA v11.8
# 120    == Blackwell, needs CUDA v12.8, FP4 tensor cores

So 89 is your row, and the same file gates 89-real behind CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8". A CUDA 11.1 toolkit that is fine for the previous generation is not enough here. You never name the architecture yourself, but note which mechanism does it for you: the native path is gated on CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24", so on the CMake 3.18 this recipe allows it does not run — the fallback list runs instead, and that list appends 89-real for any 11.8-or-newer toolkit. Either path covers this card.

Do not follow the model card's llama.cpp instructions. The card still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release. Mainline merged native nanbeige support in PR #25994 on 2026-07-27, and issue #26086 was closed the same day. Release tag b10153 is that merge commit, so it is the earliest tag that works.

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Confirm the checkout carries the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q4_K_M GGUF

Nanbeige publishes no first-party GGUF: the org's ten public repositories include -FP8 and -GPTQ-Int8 builds of this model and nothing in GGUF form. This recipe uses owao/Nanbeige4.2-3B-GGUF; bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a wider alternative ladder. Pick one repository and stay inside it — the two quantizers' Q4_K_M builds differ by 109 MB, and mixing their numbers into one budget is how a sum goes quietly wrong.

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q4_K_M.gguf \
  --local-dir ./models

numpy is not a dependency of huggingface_hub, and the next step needs it.

3. Verify the artifact carries the loop parameter

Worth doing by hand exactly once. If a GGUF was produced without num_loops, llama.cpp defaults the key to 1, runs 22 layers instead of 44, and you get a silently different model — no error, no warning, half the depth.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q4_K_M.gguf | grep -E "num_loops|block_count"

You want two lines, ending = 22 and = 2 respectively:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case. (The leading index numbers vary from file to file; only the key names and the values matter.)

Two commands that look like they should do this job do not, so do not substitute them: llama-gguf <file> r n prints key names only, never their values, and llama-cli suppresses the loader's metadata dump at its default verbosity besides sitting in an interactive chat loop rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 65536 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 1.0 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1. The sampler values are the model card's recommendation for agentic and tool-use tasks — the same row of its scenario table that asks for 65,536 new tokens. For reasoning and chat it recommends --temp 0.6.

Four flags are load-bearing:

• -c 65536, set explicitly. Omitting it is not neutral — llama.cpp's context-fitting default interpolates the window downward until the model fits free memory, so you silently get some other number and the budget below stops describing your process.
• -fa on. A quantized V cache requires Flash Attention; without it llama.cpp refuses outright. The auto default would enable it anyway, but being explicit makes the failure mode legible.
• -ctk q4_0 -ctv q4_0 — matched types. A stock CUDA build instantiates FlashAttention kernels only for identical K and V types; ggml/src/ggml-cuda/fattn.cu returns BEST_FATTN_KERNEL_NONE when K->type != V->type outside GGML_CUDA_FA_ALL_QUANTS. A clever-looking -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. With no flag, tools/server/server.cpp resolves the default with params.n_parallel = 4; params.kv_unified = true; in the same branch (it tests params.n_parallel < 0). Unified means one shared pool of -c cells that a single conversation may consume in full — which is what makes 65,536 an actual window rather than a quarter of one. Passing an explicit positive --parallel N skips that branch, leaves the cache non-unified, and gives each conversation -c / N for identical memory. If you want exactly one slot with the full window, --parallel 1 is the honest way to say so.

Why 65,536 needs the 4-bit cache, in bytes

llama.cpp sizes the cache as n_layer × (n_embd_k_gqa + n_embd_v_gqa) × bytes_per_element per token. config.json gives num_key_value_heads: 8 and head_dim: 128, so each layer stores 8 × 128 = 1024 elements for K and 1024 for V. Over 44 logical layers that is 90,112 elements per token:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

The block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 elements, from ggml/src/ggml-common.h. At 65,536 tokens that is 11.000 GiB of f16 cache, 5.844 GiB at q8_0, and 3.094 GiB at q4_0 — the first two exceed the card on their own, before a byte of weights.

Two further terms complete the bill, and both are large enough that leaving them out would flip rows.

A flat 0.317 GiB of reserved logits. At startup llama.cpp reserves one worst-case prompt-processing graph. src/llama-context.cpp builds it at const uint32_t n_tokens = std::min(cparams.n_ctx, cparams.n_ubatch); — 512 rows at stock settings — and sets the logit rows it must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), with n_outputs_max defaulting to n_batch. That is 512 rows, each a full f32 distribution over this model's unusually large 166,144-token vocabulary: 512 × 166,144 × 4 B = 340,262,912 B = 324.5 MiB. It does not move when you change -c; it halves when you halve -ub.

4,096 bytes per context token of FlashAttention dequant scratch — only because the cache is quantized. In ggml_cuda_flash_attn_ext_get_alloc_size the TILE and MMA_F16 kernels set need_f16_K = need_f16_V = true, and fattn-common.cuh then adds ggml_nelements(K) * ggml_type_size(GGML_TYPE_F16) for K and again for V, each behind if (need_f16_K && K->type != GGML_TYPE_F16). So an f16 cache pays exactly none of it, and a quantized one pays (1024 + 1024) × 2 B per token — 0.250 GiB at 65,536. This term is a property of the reserved graph, not of your card: the reserve is 512 rows wide, which fails the compute-capability-8.9 test for the cheap vector kernel (Q->ne[1] <= 2), fails the pre-Ada test (Q->ne[1] == 1) and fails the trailing single-row fallback, so MMA_F16 is returned on every NVIDIA architecture alike. Do not expect a newer or older card to escape it.

Adding all four terms, the practical envelope on 8 GiB:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Unallocated on 8 GiB
Q4_K_M · q4_0 · 65,536 (this recipe)	2.398	3.094	0.317	0.250	6.059 GiB	1.941 GiB
Q4_K_M · q8_0 · 32,768	2.398	2.922	0.317	0.125	5.762 GiB	2.238 GiB
Q4_K_M · q4_0 · 32,768	2.398	1.547	0.317	0.125	4.387 GiB	3.613 GiB
Q5_K_M · q4_0 · 65,536	2.782	3.094	0.317	0.250	6.443 GiB	1.557 GiB
Q4_K_M · q4_0 · 98,304	2.398	4.641	0.317	0.375	7.730 GiB	0.270 GiB — no room for the CUDA context
Q4_K_M · q8_0 · 65,536	2.398	5.844	0.317	0.250	8.809 GiB	will not fit
Q4_K_M · f16 · 32,768	2.398	5.500	0.317	0.000	8.215 GiB	will not fit

The last row is the trap: a reader who budgets from num_hidden_layers: 22 predicts 2.750 GiB of f16 cache at 32,768, concludes there is room to spare, and is wrong by a factor of two.

Two consumers sit on the same 8 GiB and are deliberately not in the accounted total, because nothing here measures them: the CUDA runtime's own per-process context, and your desktop compositor if this card drives a monitor. The 1.941 GiB left over is what they share — comfortable headless, snug on a desktop, and the first lever if it is too snug is -ub 256, which costs no context at all. One further consumer lives outside VRAM entirely: the attention mask is n_kv × n_ubatch in f16, and src/llama-graph.cpp asserts ggml_backend_buffer_is_host(self_kq_mask->buffer) on it, so it is pinned system RAM. Counting it as VRAM is a common way to over-budget this model.

What the 4-bit cache actually costs

Honestly: unknown. There is no published evaluation of this model under a quantized KV cache at any tier, so q4_0 is the aggressive choice on the ecosystem's general reputation rather than on a measured quality difference for this architecture. The third row above is the conservative alternative — q8_0 at 32,768 — and it is a defensible reading of the same numbers if your prompts are short. This page leads the 65,536 configuration because that is the generation budget the model card's own scenario table names for agentic and tool-use work, and because a window that size is reachable here at all: at q8_0 it is not.

There is one corroborating datapoint that the configuration runs on an 8 GB card, though it is not on this one. On discussion #18 community user thermi6 posts a llama-server log for a Q4_K_M model with KV_TYPE_K=q4_0, KV_TYPE_V=q4_0 and MAX_CTX=65536 on an RTX 5060 8 GB, and it loads and serves. That is a different, newer card, so nothing about its speed transfers here — but 8 GB is 8 GB, and it is worth noting that the same author wrote a day earlier that "Right now this works on 8 GB cards: 4 bit quant + 32768 tokens context length. Any higher and it is too large for VRAM." without stating a cache type. His own log the next day doubles that window on the same card. The variable between the two messages is the cache, which is exactly the trade this page is about.

Results

• Speed: omitted — no measurement exists for this pair, and none exists for this model on any Ada card. The only throughput figures anywhere for Nanbeige4.2-3B are the ones thermi6 posted for an RTX 5060, a newer card with substantially faster memory; decode here is memory-bound, so those figures are a ceiling this card will not reach, not an estimate for it. Publishing a number derived from them would be inventing one. If you run this configuration, please contribute the measurement so /check/nanbeige4-2-3b/rtx-4060-ti-8gb stops being empty.
• VRAM usage: 6.059 GiB accounted at the lead configuration — 2.398 GiB of Q4_K_M weights, 3.094 GiB of q4_0 cache at 65,536 tokens, 0.317 GiB of reserved logits and 0.250 GiB of FlashAttention dequant scratch — leaving 1.941 GiB of the card's 8 GiB for the CUDA context, the activation working set and your display. Derived from measured file bytes and llama.cpp's own allocation rules, not measured on hardware.
• Quality notes: the model is ~4.17 B parameters total and ~3.15 B non-embedding — the "3B" in the name counts the non-embedding half, and the extra billion is the untied 166,144-token vocabulary at both ends of the stack. That vocabulary is also why the logits reservation above is as large as it is: the same 512-row slot would hold 64.0 MiB for a 32,768-token vocabulary and holds 324.5 MiB here.

Reaching the full 256K context

The card states that "The model supports a context length of up to 262,144 tokens (256K)." At 44 logical layers that cache is 12.375 GiB even at q4_0 — more than this entire card, before a single weight byte. It stays reachable by keeping the weights on the GPU and the cache in system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q4_K_M.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM and trades decode speed for window size. Treat it as a capability, not a default.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-4060-ti-8gb.

Troubleshooting

llama-server OOMs at 65,536 tokens

Check the cache flags first. At this context the default f16 cache is 11.000 GiB and q8_0 is 5.844 GiB — both exceed the card before any weights. Only -ctk q4_0 -ctv q4_0 fits.

If the flags are already right and you are a couple of hundred MiB short, reach for -ub before you touch -c, because the logits reservation scales with the micro-batch and not with the window:

-ub	reserved logits	vs. default
512 (default)	0.317 GiB	—
256	0.158 GiB	−0.158 GiB
128	0.079 GiB	−0.238 GiB

It costs prompt-processing throughput, not context and not quality. Note it does not touch the 0.250 GiB of dequant scratch, which is sized by -c. Past that, every 16,384 tokens you give back returns 0.836 GiB between cache and scratch, so -c 49152 accounts to 5.223 GiB.

unknown model architecture: 'nanbeige'

The build predates b10153. Rebuild from mainline master, or take a release tag at or above b10153. Distribution channels lag: the model card notes that LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into the LM Studio backend directory.

nvcc fatal: Unsupported gpu architecture 'compute_89'

Your CUDA toolkit predates 11.8. The 89-real target in ggml/src/ggml-cuda/CMakeLists.txt is gated on exactly that version, and a native build asks the toolkit to compile for the card it found. Install a 11.8-or-newer toolkit; there is no flag that works around a compiler that does not know the architecture.

Tool calls come back as plain text instead of executing

Fixed in mainline as of tag b10227 — check your build before debugging anything else. It was a llama.cpp parser bug, not a model bug and not your configuration: the model sometimes emits <tool_call> followed by a space rather than a newline, and the auto-generated parser required that newline verbatim, so the whole reply was consumed as content and the tool-call rule never ran. PR #26324 measured the loss at roughly a quarter of calls across 240 markers; the model's own discussion #17 reported roughly one call in six, though on the vendor nanbeige42 fork rather than on mainline. Both are community reports rather than a maintainer measurement, so read them as "some calls" rather than a constant. The fix arrived by a different route — PR #26252, a specialized parser merged on 2026-08-02 and released as tag b10227 — after which the author of #26324 reported tool calls working every time on a newer build and closed his own pull request unmerged on 2026-08-10. If calls still come back as text, your build predates b10227; rebuild. The failure bit hardest at the --temp 1.0 this recipe recommends — #17's small probe found every generation parsing at temperature 0.

response_format: json_schema fails before any token is generated

Grammar-constrained decoding aborts at sampler init with Failed to initialize samplers: Unexpected empty grammar stack. Two users reproduced it independently on discussion #6 — but both were running the vendor fork rather than the mainline build installed here, so treat the status on mainline as untested rather than known-broken. If you hit it, restart the server with --no-jinja; plain generation is unaffected either way.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads exist under user namespaces, but most search hits for "nanbeige" are 4.1 builds — a different model generation. Use llama-server as above.

The model loads but the output is subtly poor

Re-run step 3 and confirm the GGUF reports nanbeige.num_loops = 2. A file converted by a path that omits the key loads happily and runs at half depth. Also confirm the build post-dates 2026-07-27; older mainline builds refuse the architecture outright rather than misbehave.