Nanbeige4.2-3B on RTX 3090: the full 256K context in VRAM with llama.cpp

What You'll Build

A llama-server endpoint running Nanbeige4.2-3B at 262,144 tokens of context, entirely in the RTX 3090's VRAM, using near-lossless Q8_0 weights. This is the card where the model's whole declared context stops being a spec-sheet number: on 8 GB a user on the model's own discussion #18 reports a practical ceiling of 32,768 tokens, and the reason is the same architectural quirk that makes 24 GB the interesting card.

Hardware data: RTX 3090 (24 GB VRAM) · Q8_0 weights 4.130 GiB + q4_0 KV cache at 262,144 tokens 12.375 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 1.000 GiB = 17.822 GiB accounted · See benchmark data

⚠️ The 2.4 GiB "3B" is not the constraint here — the KV cache is. Nanbeige4.2-3B is a Looped Transformer: its 22 blocks are executed twice per forward pass over one shared set of weights. The weights are counted once, but llama.cpp allocates KV for 44 logical layers, so every context figure you would compute from num_hidden_layers: 22 is exactly half the truth. At f16 the full 262,144-token cache is 44 GiB — nearly twice the card. Everything below exists to get around that.

ℹ️ This is the 24 GB anchor for the model, and it is deliberately not the 8 GB one. The install steps are the same on any CUDA card; what changes is which -c and which -ctk/-ctv you can afford. A separate recipe covers the 8 GB tier, where the answer is "32K and be grateful". Here the answer is "all of it, if you quantize the cache".

Requirements

Component	Minimum	This recipe
GPU	24 GB VRAM, CUDA	— not measured; the budget below is derived (/contribute)
RAM	8 GB system RAM	—
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	~13 GB if you also clone and build llama.cpp
Software	llama.cpp b10153+, CUDA toolkit, CMake	—

Where the parameter counts come from

The card says "3B"; the repository is 8.34 GB in bf16, which is not what a 3B looks like. Both are correct, and the difference matters for the budget, so here is the census rather than an assertion.

HuggingFace's own tensor index for Nanbeige/Nanbeige4.2-3B reports 4,169,800,704 parameters, all BF16. That reconciles exactly with config.json:

• Embeddings, untied (tie_word_embeddings: false): 166144 × 3072 × 2 = 1,020,788,736
• Per block: attention 3072×6144 + 2×(3072×1024) + 6144×3072 = 44,040,192, plus MLP 3 × 3072 × 10752 = 99,090,432, plus two RMS norms = 143,136,768
• 22 blocks + the final norm = 3,149,011,968
• Total: 1,020,788,736 + 3,149,011,968 = 4,169,800,704 — the HF figure, to the parameter.

So "3B" is the non-embedding count, which is also how the model card's own comparison table labels it ("Total Params 4B / Non-embedding Params 3B"). The important consequence: 22 blocks appear once in that census. If the two loop passes had their own weights the model would be ~7.3B. They do not — the loop reuses one stack. The technical report puts it as reusing "the same Transformer stack to process hidden states for an additional pass".

The KV cache doubles, and that is not a bug

This is the fact the whole recipe turns on, and it is confirmed independently three ways.

1. The vendor designed it that way. The technical report has a section titled "KV Cache Sharing and Scaling Configuration" stating that a variant sharing the KV cache across loop passes was investigated, that it halves the cache, and that it was rejected because its performance gains were consistently lower than the full non-sharing loop. A Nanbeige team member repeated it on the model's own discussions when a user opened a thread called "the modal is tiny but kv cache exploding!" — "We have also investigated KV-cache sharing across loop passes, but the performance gains were notably smaller than with the full looped setup." (discussion #10, leran1995, flagged as an org member), and again three days later: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2" (discussion #18).

2. llama.cpp implements it that way. src/models/nanbeige.cpp expands the logical layer count before allocation — hparams.n_layer_all = n_layer_phys * n_loops under a comment reading "Expand logical layer count before load_tensors() allocates layers / KV" — then copies the layer structs across loops under a second comment: "Share physical weights across loops; each slot still has its own KV index." src/llama-kv-cache.cpp sizes the cache with const uint32_t n_layer = hparams.n_layer_all; and iterates every one of them. For nanbeige with num_loops: 2 that is 44, not 22.

3. The GGUF carries the flag — and you should check that yours does. conversion/nanbeige.py writes num_loops into the file (self.gguf_writer.add_num_loops(n_loops)), so any GGUF from the mainline converter has it. The loader reads it as optional with a default of 1 — uint32_t n_loops_u = 1; ml.get_key(LLM_KV_NUM_LOOPS, n_loops_u, false); — so a file converted by some other path that omits the key loads happily and runs 22 layers instead of 44. That is a silently different model, not an error.

The check costs nothing: llama-kv-cache.cpp prints the layer count in its startup line, so confirm the server logs 44 layers before you trust anything below.

llama_kv_cache: size = ... MiB (262144 cells,  44 layers,  4/1 seqs), K (q4_0): ..., V (q4_0): ...

If that says 22 layers, your cache is half-size, your context is wrong, and the model is not the one the benchmarks describe.

KV cost per token, and what 24 GB buys

config.json gives num_key_value_heads: 8 and head_dim: 128, so one layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Over 44 logical layers that is 90,112 elements per token — double what the 22 in config.json would suggest.

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224 (176.0 KiB)
q8_0	34/32 = 1.0625	95,744 (93.5 KiB)
q4_0	18/32 = 0.5625	50,688 (49.5 KiB)

Block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 elements, from ggml-common.h. Multiplying out:

Context	f16 KV	q8_0 KV	q4_0 KV
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

Two more terms, before you add weights

Weights plus cache is the bulk of the bill but not all of it, and at this context one of the two remaining terms is a full gigabyte.

A flat 0.317 GiB of reserved logits. llama.cpp sizes one worst-case graph at startup and src/llama-context.cpp sets the logit rows it must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is 512 rows regardless of -c, each a full f32 distribution over the 166,144-token vocabulary: 512 × 166,144 × 4 B = 340,262,912 B = 324.5 MiB. Large only because this vocabulary is; it scales with -ub, not with context.

4,096 bytes per token of FlashAttention dequant scratch — but only on a quantized cache. The kernel reads f16, so a quantized cache is expanded into scratch VRAM first. ggml/src/ggml-cuda/fattn.cu sets need_f16_K = need_f16_V = true for the tile and MMA kernels and fattn-common.cuh allocates ggml_nelements(K) * ggml_type_size(GGML_TYPE_F16) for each — behind if (need_f16_K && K->type != GGML_TYPE_F16), so an f16 cache pays none of it. For this model that is (1024 + 1024) × 2 B per token: 0.250 GiB at 65,536, 0.500 GiB at 131,072 and 1.000 GiB at 262,144.

That second term is why the effective price of a quantized cache is the table above plus 4,096: 54,784 B/token at q4_0 and 99,840 B/token at q8_0. It is 2.3% of the f16 rate, so it never overturns the decision to quantize — but at a quarter-million tokens it is a gigabyte, and a gigabyte is not a rounding error even on this card.

Add all four terms and the shape of the card becomes obvious. Both weight tiers below are from the bartowski GGUF repo, byte counts via the HuggingFace tree API:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Slack on 24 GiB
Q8_0 · q4_0 KV · 262,144 ctx (this recipe)	4.130	12.375	0.317	1.000	17.822 GiB	6.178 GiB
Q8_0 · q8_0 KV · 131,072 ctx	4.130	11.688	0.317	0.500	16.635 GiB	7.365 GiB
Q8_0 · f16 KV · 65,536 ctx	4.130	11.000	0.317	0.000	15.447 GiB	8.553 GiB
Q4_K_M · f16 KV · 65,536 ctx	2.500	11.000	0.317	0.000	13.817 GiB	10.183 GiB

The slack still has to absorb the CUDA context, the activation working set and the graph for a 44-layer unrolled forward pass — none of which I could source a measurement for on this card, which is why the recipe leaves 6.178 GiB rather than pushing the context further. Separately, budget 256 MiB of system RAM for the attention mask, which is host memory and deliberately not part of any figure above: src/llama-graph.cpp asserts ggml_backend_buffer_is_host(self_kq_mask->buffer), and allocates it as n_kv × n_tokens in f16 when Flash Attention is on — 262,144 × 512 × 2 B here. Counting it as VRAM is a common way to over-budget this model.

Why Q8_0 weights and not Q4_K_M

On an 8 GB card the quant tier is the whole decision. On 24 GB it barely registers, and the arithmetic says so: within the same repo, Q8_0 (4,434,787,488 B) costs 1.631 GiB more than Q4_K_M (2,684,023,968 B). Spent on cache instead, 1.631 GiB buys 31,957 more tokens at q4_0 or 17,535 at q8_0 — 12% and 7% of the 262,144 ceiling. (Both at the effective per-token cost, dequant scratch included; on the sticker KV price alone they would read 34,540 and 18,285, and that gap is the tax.) On a model whose entire pitch is agentic and reasoning accuracy at 3B, trading a near-lossless weight tier for at most 13% more context is the wrong way round. Both figures are bartowski's files; quantizers differ, and owao's Q4_K_M is a different 2,574,807,904 B, so do not mix the two when you redo this sum. owao ships a 13-rung ladder and bartowski a 23-rung one if you want to make the other trade.

Installation

1. Build llama.cpp with CUDA

Mainline support arrived in PR #25994, merged 2026-07-27, and the first tagged build containing that merge commit is b10153. Anything older will refuse to load the architecture — the symptom one user reports on the model's discussion #23, where a second user then reports a mainline release loading it.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

The model card still tells you to build the vendor's own fork (git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git). That was correct at release and is not any more — mainline carries the architecture, and mainline is what gets the bug fixes.

2. Download the weights

Nanbeige publishes no first-party GGUF. None of the org's ten repositories carries one — the quantized builds it does publish are FP8 and GPTQ-Int8 — and a team member said on discussion #1 that official quantized versions were being prepared, which as of this writing has not happened. Everything below uses a community conversion.

pip install -U huggingface_hub

hf download bartowski/Nanbeige_Nanbeige4.2-3B-GGUF \
  --include "Nanbeige_Nanbeige4.2-3B-Q8_0.gguf" \
  --local-dir ./nanbeige4.2-3b

Two independent quantizers ship a byte-for-byte comparable Q8_0 — bartowski's at 4,434,787,488 B and owao's at 4,434,787,168 B, a 320-byte metadata difference — and both report general.architecture = nanbeige with context_length = 262144 in HuggingFace's GGUF metadata view. Check the build provenance on whichever you pick, because this model has a date cliff. The canonical repo landed a commit titled "tokenzier update for rstirp" at 2026-07-27T07:41Z, about seven hours before mainline support merged at 15:04Z. bartowski's card states the quants were made with llama.cpp release b10159, and the repo was created 2026-07-28; owao deleted and re-uploaded the entire quant set on 2026-07-28. Both current sets therefore postdate both events. owao's original 2026-07-21 upload predated both, which is also why discussion #1's link to a lowercase nanbeige4.2-3b-Q5_K_M.gguf no longer resolves — the filenames changed case in the re-upload.

Running

The long-context configuration

./build/bin/llama-server \
  -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 262144 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

That is the full declared context — "The model supports a context length of up to 262,144 tokens (256K)." — resident in VRAM on one card, at 17.822 GiB accounted.

Four of those flags are load-bearing:

• -c 262144. Set it explicitly. Omitting -c is not the same as -c 0: with -c unset, llama.cpp's -fit on default (see common/fit.cpp) interpolates the context down until the model fits free memory with a 1 GiB margin, so you silently get some arbitrary reduced number. With -c 0, common/arg.cpp sets fit_params_min_ctx = UINT32_MAX — a deliberate "I want the full trained context, do not shrink it" — and you get 262,144 with the f16 default cache, i.e. 44 GiB, i.e. an out-of-memory abort.
• -fa on. Quantized V-cache requires Flash Attention; src/llama-context.cpp throws quantized V cache was requested, but this requires Flash Attention otherwise. The auto default will turn it on for you and log enabling flash_attn since it is required for quantized V cache, but being explicit makes the failure mode legible.
• -ctk q4_0 -ctv q4_0 — matched types, not mixed. A stock CUDA build only instantiates Flash Attention kernels for identical K and V types. ggml/src/ggml-cuda/fattn.cu guards this literally: #ifndef GGML_CUDA_FA_ALL_QUANTS / if (K->type != V->type) { return BEST_FATTN_KERNEL_NONE; }, and the same file's default dispatch table lists only f16/f16, q4_0/q4_0, q8_0/q8_0 and bf16/bf16. A clever-looking -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. llama-server defaults it to -1, and tools/server/server.cpp resolves the sentinel with params.n_parallel = 4; params.kv_unified = true;. Unified means one shared cache of -c cells that any single slot may consume in full — so the table above is the whole KV bill, and one conversation can still reach 262,144. Passing -np 4 explicitly skips that branch, leaves kv_unified = false, and src/llama-context.cpp then computes n_ctx_seq = n_ctx / n_seq_max — same total memory, a quarter of the context per conversation. If you want one slot, say -np 1, not -np 4.

The quality-first alternative

If you would rather not run a 4-bit K cache, halve the context and keep the near-lossless one. 16.635 GiB accounted, and 131,072 is exactly the max-new-tokens the model card recommends for reasoning and chat:

./build/bin/llama-server \
  -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 131072 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

For agentic and tool-use work the card recommends --temp 1.0 instead, with 65,536 new tokens.

Spend the headroom on preserve_thinking

The chat template takes preserve_thinking, which controls whether reasoning from earlier assistant turns stays in the context. It is a KV-versus-quality dial, and on this card you are on the right side of it — a Nanbeige team member on discussion #9: "In our evaluations, enabling preserve_thinking generally provides better performance and better kv-cache reuse, so we recommend keeping it enabled when context length and memory allow." Pass it through llama-server's OpenAI-compatible endpoint as "chat_template_kwargs": {"preserve_thinking": true}.

Results

• Speed: omitted — there is no measurement of this model on an RTX 3090. /check/nanbeige4-2-3b/rtx-3090 has zero benchmarks, none of the 27 discussion threads on the canonical repo mentions a 3090, a 4090 or any 24 GB card, and llama.cpp's issue tracker has no Nanbeige report on this GPU. Deriving a number from a differently-sized card would be inventing one. If you run this, please contribute your measurement so the next reader gets a real figure.
• VRAM usage: 17.822 GiB accounted at the lead configuration — 4.130 GiB weights, 12.375 GiB cache, 0.317 GiB reserved logits and 1.000 GiB of FlashAttention dequant scratch — leaving 6.178 GiB of the card's 24 GiB. The q8_0/131,072 alternative comes to 16.635 GiB. Both derived from file bytes and llama.cpp's own allocation rules, not measured — see /check/nanbeige4-2-3b/rtx-3090.
• Quality notes: I found no evaluation of Nanbeige4.2-3B under a quantized KV cache at any tier in the spaces I searched — the model card, the technical report, all 27 discussion threads on the canonical repo, the bartowski and owao GGUF cards, and llama.cpp's issue tracker. Treat that as a gap, not as evidence the loss is small. q8_0 cache is generally treated as near-lossless across the llama.cpp ecosystem; q4_0 is the aggressive rung, and the reason the alternative configuration above exists. The vendor publishes no throughput figures at all — the model card and the technical report carry quality benchmarks only, so there is nothing to repeat and nothing to check them against.
• Context is trained, not extrapolated: the technical report describes a three-stage SFT curriculum that extends the supervised context from 64K to 128K to 256K, with the final stage weighted 68.9% toward agentic data. The 262,144 figure is a training target, not a RoPE-scaling claim — rope_scaling is null in config.json.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-3090.

Troubleshooting

llama_model_load: error loading model architecture: unknown model architecture: 'nanbeige'

Your build predates b10153. Rebuild from mainline master, or grab a release tag at or above b10153 — that build's commit is the merge of PR #25994. On discussion #23 one user reports still being unable to load the architecture, and a different user reports that mainline release b10199 loads it. Note that some distribution channels lag: the model card itself says LM Studio's bundled llama-server does not support nanbeige, and tells you to copy your own build's binaries into the LM Studio backend directory.

Out of memory at startup with the default cache type

You almost certainly left -ctk/-ctv at f16. At 262,144 tokens that is 44.000 GiB of cache — the arithmetic is in the table above, and it is the single most common way to be surprised by this model. Either quantize the cache or drop to -c 65536, where f16 fits at 15.447 GiB accounted — and pays no dequant scratch at all, which is why it lands closer to the quantized options than the cache column alone suggests.

If you need the full context on a card that genuinely cannot hold the cache, --no-kv-offload moves the cache to system RAM and keeps the weights on the GPU. A community user running an 8 GB RTX 5060 posted their entrypoint script on discussion #18; its comments annotate both modes — KV in VRAM as the fast path with a context ceiling, KV in system RAM as the way to reach 262,144 tokens at a much lower decode rate. Those are the author's own annotations rather than a published benchmark, and they are for a different card. On a 24 GB card you should not need the flag at all.

About a quarter of tool calls come back as plain text instead of executing

Known, open, and not your configuration. Nanbeige4.2-3B sometimes emits <tool_call> followed by a space rather than a newline, and llama.cpp's chat parser matches the marker with the newline attached. The reporter of PR #26324 puts the rate at roughly 25% and the consequence plainly: "All such tool calls currently fail and are displayed verbatim to the user instead of being executed."

The fix is contested rather than pending. A llama.cpp maintainer pushed back on the proposed whitespace trimming — "trimming whitespaces for some of the Qwen models was really detrimental to their output" — and suggested a Nanbeige-specific workaround instead; another contributor has since suggested the grammar sampler should constrain the marker. The PR was still open at the time of writing. Until it lands, treat tool-call reliability as a known ceiling on this model under llama.cpp, and check the raw completion text when a call appears to vanish. The same whitespace behaviour is discussed on the model's own discussion #17.

HTTP 400 — Failed to initialize samplers: Unexpected empty grammar stack after accepting piece: assistant (13886)

Any request with a response_format of json_schema fails at sampler init on the default jinja chat path — before a token is generated. Two users reproduced it independently on discussion #6, on different builds and different quant tiers, and both found the same workaround:

# add --no-jinja to the llama-server command
./build/bin/llama-server -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf --no-jinja ...

Overriding the surface template with --chat-template chatml does not help, and neither does strict: false or disabling reasoning; the second reporter also confirmed that the raw /completion endpoint with the same schema returns conformant JSON. That isolates the fault to the chat-completions grammar-trigger construction rather than the model or the grammar. The cost of --no-jinja is that you lose the model's own chat template, so apply the prompt format yourself if you take this route.

ollama run nanbeige/nanbeige4.2:3b-Q4_K_M returns not found

The model card lists that command under "Option 1: pull a published model", but the registry entry it names does not exist — ollama.com/nanbeige/nanbeige4.2 returns 404, as do the library/ paths for every spelling of the name. The card's other Ollama route, building Ollama from the vendor's fork and copying a llama.cpp build into its runtime payload directory, is a lot of machinery to reach a llama-server you already have. Use llama-server directly.

Mismatched -ctk/-ctv fails or silently disables Flash Attention

See the flag notes under Running: a stock CUDA build has kernels only for matched K/V types. If you want an asymmetric cache, rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.

Anything else, or a real throughput measurement on this card, is welcome via the submission form.