Nanbeige4.2-3B on RTX 5090: 256K context with a near-lossless q8_0 KV cache

What You'll Build

A llama-server endpoint running Nanbeige4.2-3B at its full 262,144-token context with a q8_0 KV cache — not the 4-bit cache a 24 GB card is forced into. The weights stay at near-lossless Q8_0. Everything lives in the RTX 5090's 32 GB; nothing spills to system RAM.

Hardware data: RTX 5090 (32 GB VRAM) · Q8_0 weights 4.130 GiB + q8_0 KV cache at 262,144 tokens 23.375 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 1.000 GiB = 28.822 GiB accounted · no benchmark submitted yet · See benchmark data

⚠️ The KV cache is double what config.json implies, and on this card that is the entire budget. config.json reports num_hidden_layers: 22 — and also num_loops: 2. Nanbeige4.2 is a Looped Transformer: the same 22 blocks execute twice per forward pass, sharing one set of weights but not one cache. llama.cpp expands the count before it allocates anything: src/models/nanbeige.cpp sets hparams.n_layer_all = n_layer_phys * n_loops under the comment "Expand logical layer count before load_tensors() allocates layers / KV.", and src/llama-kv-cache.cpp then takes const uint32_t n_layer = hparams.n_layer_all;. 44 logical layers, not 22. Every context figure you would compute from the config file is exactly half the truth.

The vendor built it this way on purpose. Team member leran1995 on discussion #10: "We have also investigated KV-cache sharing across loop passes, but the performance gains were notably smaller than with the full looped setup."

Requirements

Component	Minimum	This recipe
GPU	32 GB VRAM, CUDA compute capability 12.0	— not measured; the budget below is derived from file bytes and llama.cpp's cache formula (/contribute)
RAM	8 GB system RAM	—
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	4,434,787,488 bytes, from the HF tree API
Software	llama.cpp b10153+, CUDA Toolkit ≥ 12.8, CMake	—

What the extra 8 GB actually buys

This is the only 32 GB card in the catalogue, and the honest answer is narrower than "more of everything": the extra memory buys exactly one rung of KV-cache precision at the full context, and nothing else.

A 24 GB card already reaches 262,144 tokens — the RTX 3090 recipe does it — but only by quantizing the cache to q4_0, the most aggressive rung llama.cpp's CUDA Flash-Attention path supports. Upgrading that cache from q4_0 to q8_0 at 262,144 tokens costs exactly 11.000 GiB (23.375 − 12.375). That configuration accounts for 17.822 GiB, so on a 24 GB card it has 6.178 GiB of slack; 6.178 < 11.000, and it cannot pay. On 32 GiB the same configuration has 14.178 GiB of slack, which covers the 11.000 GiB upgrade and leaves 3.178 GiB over. (Both figures carry the same two fixed terms — the 0.317 GiB logits reservation and 1.000 GiB of dequant scratch, both derived below — and both terms depend on the context and the vocabulary rather than on which quantized rung you pick, so they cancel out of the comparison.)

So the headline is not "more context" — the context ceiling was already reached one tier down. It is the same context with a cache that is no longer 4-bit.

There is a second thing the extra memory buys, and it is easy to miss because it looks unaffordable: a cache that is not quantized at all. Q8_0 weights with an f16 cache at 131,072 tokens comes to 26.447 GiB — which does not fit a 24 GB card, and sits on this one with 5.553 GiB to spare. It is cheaper than the raw cache sizes imply, because a quantized cache is the only configuration that pays the dequant scratch: f16 pays none. The honest premium for going unquantized at 131,072 is 9.813 GiB, not the 10.313 GiB you get by differencing the cache rows. Half the window, zero cache approximation, and no smaller card in the catalogue can offer it.

The tempting third upgrade genuinely does not fit, and it is worth showing why rather than leaving it as an open question. bartowski also ships a full-precision bf16 GGUF, 8,343,845,760 B = 7.771 GiB, which is 3.641 GiB more than Q8_0. Paired with the q8_0 cache at full context that is 32.463 GiB against a 32 GiB card — over the card outright, before the CUDA context and the rest of the compute buffer exist. That verdict does not hinge on any one term: strip the dequant scratch entirely and the row still stands at 31.463 GiB, past the ~31 GiB a headless box actually leaves you. If you want exact weights you must give the cache back to q4_0, and that is the wrong trade: Q8_0 is already near-lossless against bf16, while q4_0 → q8_0 is a real change to what the model remembers.

The KV arithmetic

config.json gives num_key_value_heads: 8 and head_dim: 128, so each layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Across 44 logical layers that is 90,112 elements per token. Block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 values, from ggml/src/ggml-common.h:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

Context	f16 KV	q8_0 KV	q4_0 KV
65,536	11.000 GiB	5.844 GiB	3.094 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

Adding weights — all byte counts from bartowski/Nanbeige_Nanbeige4.2-3B-GGUF via the HF tree API — gives the shape of the card:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Slack on 32 GiB
Q8_0 · q8_0 KV · 262,144 ctx (this recipe)	4.130	23.375	0.317	1.000	28.822 GiB	3.178 GiB
Q8_0 · f16 KV · 131,072 ctx	4.130	22.000	0.317	0.000	26.447 GiB	5.553 GiB
bf16 · q4_0 KV · 262,144 ctx	7.771	12.375	0.317	1.000	21.463 GiB	10.537 GiB
Q8_0 · q4_0 KV · 262,144 ctx (the 24 GB configuration)	4.130	12.375	0.317	1.000	17.822 GiB	14.178 GiB
bf16 · f16 KV · 131,072 ctx	7.771	22.000	0.317	0.000	30.088 GiB	1.912 GiB — too tight, see below
bf16 · q8_0 KV · 262,144 ctx	7.771	23.375	0.317	1.000	32.463 GiB	negative — will not fit

Every row carries the same 0.317 GiB of reserved logits and the same per-token dequant scratch; both are derived below, and neither is optional. The Slack column is measured against the card's raw 32 GiB, so read it against the ~31 GiB a real machine leaves usable, not as free memory.

The quantized cache is not free: Flash Attention's dequant scratch

The column above that no arithmetic from the model's own config predicts. A quantized KV cache has to be expanded back to f16 before the tensor-core Flash-Attention kernel can multiply it, and llama.cpp allocates that scratch out of VRAM, appended to the attention output tensor. ggml/src/ggml-cuda/fattn-common.cuh sets data.end = (uintptr_t) dst->data + ggml_nbytes(dst) and then, only when K->type != GGML_TYPE_F16, adds ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16) — and the same again for V unless V is a view of K. It is a real reservation, not a runtime pool: ggml_backend_cuda_buffer_type_get_alloc_size routes GGML_OP_FLASH_ATTN_EXT tensors through ggml_cuda_flash_attn_ext_get_alloc_size, so the graph allocator sizes the compute buffer to include it.

Three facts decide what it costs here:

• Both halves are charged. V_is_K_view requires V to share K's view_src; this model's cache stores K and V as separate per-layer tensors, so the V branch allocates too. No halving.
• The K and V views are 128 × n_kv × 8, so each expands to 1024 × n_kv × 2 B. Together: 4,096 bytes per token — linear in -c, and independent of the rung, since q4_0 and q8_0 dequantize to the same f16.
• It is charged once, not 44 times. The scratch is sized per attention node and each layer's output dies immediately, so the graph allocator reuses one block across the loop's 44 passes. This is the one place in this recipe where the doubled layer count does not double the bill.

Context	Scratch (q4_0 or q8_0 cache)	Scratch (f16 cache)
65,536	0.250 GiB	—
131,072	0.500 GiB	—
262,144	1.000 GiB	—

At the full window that is just under a third of the lead configuration's slack, and it exists because this recipe recommends a quantized cache. On the bf16 + q8_0 row it is the difference between over-budget and over the card outright.

The other fixed term: reserved logits

Weights + KV is not the whole allocation, and the largest remaining term is not the one the architecture makes you expect.

The attention mask is not in VRAM, so do not budget card memory for it however large n_kv gets. llm_graph_input_attn_kv::set_input fills the mask from the CPU, and src/llama-graph.cpp asserts exactly that before writing: GGML_ASSERT(ggml_backend_buffer_is_host(self_kq_mask->buffer));. On CUDA a host-accessible buffer is a pinned system-RAM allocation, not device memory.

What does sit in the device compute buffer is the reserved logits tensor, and this model's vocabulary makes it big enough to belong in the table above rather than in a footnote. nanbeige.cpp closes its graph with res->t_logits = cur; on the output projection, so the tensor is n_vocab × n_outputs in f32. src/llama-context.cpp reserves the prompt-processing graph at n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), with n_tokens = std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaulting to n_batch — at llama.cpp's stock n_ubatch 512 and n_batch 2048, that resolves to 512 rows. Against a 166,144-token vocabulary: 512 × 166,144 × 4 B = 324.5 MiB — the flat 0.317 GiB every row of the table carries.

That term is context-independent: it scales with the micro-batch and the vocabulary, not with n_kv. Which is the useful part — the largest compute-buffer allocation on this card costs the same at 262,144 tokens as it does at 8,192, so pushing to the full window does not enlarge it. The CUDA context and the per-micro-batch activations sit on top, and I could not source a measured peak for this pair, so they are not quantified anywhere on this page.

Budget against roughly 31 GiB usable of the card's 32, which is the shape of a headless Linux box after the driver's own reserve; a desktop compositor takes another 1–2 GiB and you should subtract it. The lead configuration lands at 28.822 GiB accounted — weights, cache, logits and dequant scratch — against that ~31 GiB, with only an unquantified CUDA context still to land on top. That is a real margin rather than a comfortable one, and it is the reason this page does not spend the remainder on bf16 weights.

Blackwell: what a current llama.cpp build actually needs here

The RTX 5090 is compute capability 12.0 (sm_120), and llama.cpp treats it as a first-class target — ggml/src/ggml-cuda/common.cuh defines GGML_CUDA_CC_BLACKWELL 1200, and because that clears turing_mma_available, the tensor-core Flash-Attention path this recipe depends on is enabled. There is no Blackwell-shaped gap in the runtime.

There is a Blackwell-shaped gap in toolkits. ggml/src/ggml-cuda/CMakeLists.txt documents the floor in a comment — "Blackwell, needs CUDA v12.8, FP4 tensor cores" — and enforces it: 120a-real is appended to CMAKE_CUDA_ARCHITECTURES only inside if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8"). Below 12.8 the list tops out at 90-virtual and no sm_120 device code is produced.

That has one consequence a reader of this page will actually hit. llama.cpp's release job builds Windows CUDA binaries from a ['12.4', '13.3'] matrix with -DGGML_NATIVE=OFF and no architecture override (.github/workflows/release.yml), so the default list above is exactly what ships: the CUDA 12.4 download contains no Blackwell device code and the CUDA 13.3 download does. Take the CUDA 13 asset.

The container route is fine as-is, which answers the question raised on discussion #27 about avoiding a custom build: .devops/cuda.Dockerfile pins CUDA_VERSION=12.8.1, and the docker workflow builds ghcr.io/ggml-org/llama.cpp:server-cuda from it (with server-cuda13 on 13.3.0). Both are at or above the floor, both carry mainline nanbeige, and neither needs a fork.

Installation

1. Build llama.cpp with CUDA

Mainline architecture support arrived in PR #25994, merged 2026-07-27 as commit b77d646; release b10153 carries that exact commit and is the first tagged build that loads the model.

nvcc --version   # must report release 12.8 or newer

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

With the default GGML_NATIVE=ON, CMake sets CMAKE_CUDA_ARCHITECTURES to native and compiles for the card in the machine — correct and fastest. If you build in a container without the GPU visible, or cross-build for this card, name it explicitly:

cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120

The model card still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release and is not any more — mainline carries the architecture, and mainline is what gets the bug fixes.

2. Download the weights

Nanbeige publishes no first-party GGUF: enumerating the org's repositories returns ten models, none tagged gguf, and the quantized builds it does publish are FP8 and GPTQ-Int8. This recipe uses bartowski's conversion, whose ladder includes the bf16 build priced in the table above.

pip install -U huggingface_hub numpy

hf download bartowski/Nanbeige_Nanbeige4.2-3B-GGUF \
  --include "Nanbeige_Nanbeige4.2-3B-Q8_0.gguf" \
  --local-dir ./nanbeige4.2-3b

owao's repo is an equally current 13-rung alternative — its Q8_0 is 4,434,787,168 B against bartowski's 4,434,787,488 B, a 320-byte metadata difference that rounds to the same 4.130 GiB. Do not mix the two repos' byte counts in one sum, though: their Q4_K_M files are genuinely different (2,684,023,968 B versus 2,574,807,904 B). Both ladders postdate the vendor's last weights-affecting change — the 2026-07-27 chat-template commit fab06df, which added a .rstrip('\n') to the template's reasoning extraction. owao's discussion #3 announces the re-upload for exactly that commit; bartowski's repo was created the following day and needed no re-upload. Either is current.

3. Verify the artifact carries the loop parameter

Worth doing by hand once. load_arch_hparams reads the key as optional with a default of 1 — uint32_t n_loops_u = 1; ml.get_key(LLM_KV_NUM_LOOPS, n_loops_u, false); — so a GGUF converted without it loads happily, runs 22 layers instead of 44, and is silently a different model with half the cache you budgeted.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

Two lines, ending = 22 and = 2:

     18: UINT32     |        1 | nanbeige.block_count = 22
     30: UINT32     |        1 | nanbeige.num_loops = 2

A missing num_loops line is the failure case. Two commands that look like substitutes are not: llama-gguf <file> r n prints key names without values, and llama-cli suppresses the loader's metadata dump at its default verbosity (it is emitted at trace level) and then waits for interactive input rather than exiting.

Running

./build/bin/llama-server \
  -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 262144 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

That is the model's whole declared window — "The model supports a context length of up to 262,144 tokens (256K)." — resident in VRAM at 28.822 GiB accounted, with an 8-bit cache. Confirm the startup line reports 44 layers before trusting any of the arithmetic above:

llama_kv_cache: size = ... MiB (262144 cells,  44 layers,  4/1 seqs), K (q8_0): ..., V (q8_0): ...

Four flags carry weight:

• -c 262144, and never -c 0. They are not the same thing. common/arg.cpp reacts to a literal zero by setting params.fit_params_min_ctx = UINT32_MAX under the comment "disable context reduction in llama_params_fit if the user explicitly requests the full context size" — you get 262,144 cells at the f16 default, i.e. a 44.000 GiB allocation, i.e. an instant abort. 32 GB feels like enough to survive that mistake and it is not.
• -fa on. A quantized V cache requires Flash Attention; src/llama-context.cpp throws quantized V cache was requested, but this requires Flash Attention otherwise. The auto default would enable it for you, but being explicit makes the failure legible.
• -ctk q8_0 -ctv q8_0 — matched types. A stock CUDA build instantiates Flash-Attention kernels only for identical K and V types: ggml/src/ggml-cuda/fattn.cu guards it as #ifndef GGML_CUDA_FA_ALL_QUANTS / if (K->type != V->type) { return BEST_FATTN_KERNEL_NONE; }, and the same file's ggml_cuda_fattn_kv_type_supported accepts only f32, f16, bf16, q4_0 and q8_0 by default. An asymmetric -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON — which adds q4_1, q5_0 and q5_1, all larger than q4_0, so q4_0 stays the most aggressive rung either way.
• Leave --parallel alone. llama-server defaults it to -1, and tools/server/server.cpp resolves the sentinel in one branch: "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true". Unified means a single shared pool of -c cells that one conversation may consume entirely — so the table above is the whole KV bill and there is no ×4 multiplier hiding in it. Passing an explicit positive --parallel 4 skips that branch, leaves kv_unified false, and quarters the per-conversation window for identical memory. If you want a single slot, say --parallel 1.

If you would rather not quantize the cache at all

Half the context, an exact f16 cache, and no dequant scratch at all — 26.447 GiB accounted, which is 2.375 GiB less than the lead despite the cache being nominally larger:

./build/bin/llama-server \
  -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 131072 -fa on \
  --temp 0.6 --top-p 0.95 --top-k 20

131,072 is also the max-new-tokens the model card recommends for reasoning and chat; for agentic and tool-use work the card recommends --temp 1.0 with 65,536.

Spend the headroom on preserve_thinking

The chat template takes preserve_thinking, which decides whether reasoning from earlier assistant turns stays in context. It is a KV-versus-quality dial and this card is on the comfortable side of it. leran1995 on discussion #9: "In our evaluations, enabling preserve_thinking generally provides better performance and better kv-cache reuse, so we recommend keeping it enabled when context length and memory allow." Pass it through the OpenAI-compatible endpoint as "chat_template_kwargs": {"preserve_thinking": true}.

Results

• Speed: omitted — nothing has been measured on this card. There is one Blackwell datapoint, and it is worth naming precisely rather than borrowing: community user thermi6 posted llama-server logs on discussion #18, prefaced "For information, this is what I get with an RTX 5060 that has 8 GB of VRAM:" — 63.07 t/s decoding at a short context, falling to about 16 t/s once the window filled, on Q4_K_M weights with a q4_0 cache at 65,536 tokens and --parallel 1. That is the entry-level part of the same architecture generation running a smaller quant at a quarter of this recipe's context, and llama.cpp decode is memory-bandwidth-bound — so treat it as a floor this card comfortably beats, not an estimate for it, and note that the configuration differs on every axis that matters. If you run the configuration above, please contribute the numbers so /check/nanbeige4-2-3b/rtx-5090 stops being empty.
• VRAM usage: 28.822 GiB accounted for the lead configuration — 4.130 GiB of Q8_0 weights, 23.375 GiB of q8_0 cache, 0.317 GiB of reserved logits and 1.000 GiB of Flash-Attention dequant scratch — leaving 3.178 GiB of the card's 32 GiB. What that remainder still has to cover: the CUDA context, the rest of the compute buffer and your desktop. The attention mask is not among them — it lives in host memory, as shown above. Derived from file bytes and llama.cpp's own allocation rules, not measured; see /check/nanbeige4-2-3b/rtx-5090.
• Quality notes: I found no evaluation of Nanbeige4.2-3B under a quantized KV cache at any tier, in the model card, the technical report, all 27 discussion threads on the canonical repo, either GGUF repo, or llama.cpp's issue tracker. q8_0 is generally treated as near-lossless across the llama.cpp ecosystem and q4_0 as the aggressive rung, which is the reasoning behind preferring cache precision over weight precision here — but treat the absence of a measurement as a gap, not as evidence the loss is small.
• Parameter counts: ~4.17 B total, ~3.15 B non-embedding. The "3B" in the name counts the non-embedding half, exactly as the model card's own comparison table labels it ("Total Params 4B / Non-embedding Params 3B"); the extra billion is the untied 166,144-token vocabulary at both ends of the stack. The 22 blocks appear once in that census, which is the direct evidence that the loop shares weights — two independent stacks would be about 7.3 B.
• The context is trained, not extrapolated: rope_scaling is null in config.json, and the technical report describes an SFT curriculum that extends the supervised context in stages to 256K. 262,144 is a training target, not a RoPE-scaling claim.

For the full benchmark data, see /check/nanbeige4-2-3b/rtx-5090.

Troubleshooting

unknown model architecture: 'nanbeige'

Your build predates b10153. Rebuild from mainline master or take a release tag at or above b10153 — that build's commit is the merge of PR #25994. This is also what an ollama run hf.co/<repo> attempt reports, because Ollama carries its own llama.cpp copy: the exact error appears on owao's discussion #1, answered at the time with "use the vendor fork" — advice that has since expired.

It builds and loads, but throughput is poor or the first run stalls

Check the toolkit, not the model. A build made with CUDA older than 12.8 emits no sm_120 code for this card, because 120a-real is gated on CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8". Run nvcc --version, and if you are on the prebuilt Windows binaries make sure you took the CUDA 13 archive rather than the CUDA 12.4 one. The same 12.8 floor shows up elsewhere in the ecosystem on this card — flash-attention's setup.py likewise emits arch=compute_120,code=sm_120 only from CUDA 12.8 up — so if you abandon llama.cpp for the model card's transformers route, the toolkit requirement follows you.

Out of memory at startup

Almost always the f16 cache. At 262,144 tokens across 44 logical layers that is 44.000 GiB — larger than the card by itself, before weights. Either pass -ctk q8_0 -ctv q8_0 as above, or drop to -c 131072, where f16 fits at 26.447 GiB accounted. If you passed -c 0 expecting "use the model's full context", see the flag note under Running: it does exactly that, at f16, and aborts.

If you OOM on the lead configuration specifically, having budgeted weights + cache and found 27.505 GiB against a 32 GB card, the missing 1.317 GiB is the two fixed terms this page derives: 1.000 GiB of dequant scratch (4,096 B per token of context, allocated because the cache is quantized) and 0.317 GiB of reserved logits. Dropping -c is the lever that moves the first; nothing about -c moves the second, and changing q8_0 to q4_0 moves neither, since both rungs dequantize to the same f16.

The other way to lose this budget silently is an explicit positive --parallel N, which leaves kv_unified false and allocates per slot. Memory does not change; your usable window becomes 262144 / N.

About a quarter of tool calls come back as plain text

Known, open, and not your configuration. The model sometimes emits <tool_call> followed by a space rather than a newline, and llama.cpp's chat parser matches the marker with the newline attached. The reporter of PR #26324 puts the rate at roughly 25% and the consequence plainly: "All such tool calls currently fail and are displayed verbatim to the user instead of being executed." The PR was still open at the time of writing — a llama.cpp maintainer pushed back on blanket whitespace trimming as harmful to other models and suggested a Nanbeige-specific fix instead — so treat tool-call reliability as a known ceiling under llama.cpp and check the raw completion text when a call appears to vanish. The same behaviour is discussed on the model's discussion #17 and on owao's discussion #2.

response_format: json_schema fails before any token is generated

Grammar-constrained decoding can fail at sampler init with Failed to initialize samplers: Unexpected empty grammar stack after accepting piece: assistant (13886). Two users reproduced it independently on discussion #6, and both were on the vendor fork rather than the mainline build this recipe installs — so read it as untested here rather than known-broken.

Only one of them says so in words, so the second is worth showing. dmhagar states his build outright: git log -1 = "support nanbeige4.2 model", which is the fork branch's own first commit. IingHu gives only a hash — d28da86 — and it reads like a mainline commit while not being one. It is the third of the five commits of PR #25994, whose head branch was nanbeige42 on Nanbeige/llama.cpp; its parent chain runs back through 03327d6 to that very commit, and the PR was squash-merged, so the hash appears in no version of master before or after the merge. The clock closes it independently: that comment was posted at 07:19 UTC on 2026-07-27, roughly eight hours before mainline gained the architecture at 15:04 UTC, and it reports the GGUF loading as general.architecture=nanbeige — which no mainline build in existence at that moment could have done.

Untested is not fixed, though: the code they implicate is the jinja grammar-trigger construction — common/chat-auto-parser-generator.cpp, which both name, plus common/chat.cpp, which only dmhagar adds — and both files are upstream code the fork inherits rather than anything the fork added. The workaround both found is --no-jinja, at the cost of the model's own chat template — apply the prompt format yourself if you take that route. Plain generation is unaffected either way.

ollama run nanbeige/nanbeige4.2:3b-Q4_K_M returns not found

The model card lists that command, but the namespace it names does not exist in the Ollama registry, and neither does a library/ entry under any spelling of the name. The card's other Ollama route — building Ollama from the vendor's fork and copying a llama.cpp build into its runtime payload directory — is a lot of machinery to reach a llama-server you already have. Use llama-server directly, or ghcr.io/ggml-org/llama.cpp:server-cuda if you want a container.

Anything else, or a real throughput measurement on this card, is welcome via the submission form.