Nanbeige4.2-3B on RTX 5070: Q8_0 weights, 65,536-token cache, CUDA 12.8 floor

What You'll Build

A local llama-server endpoint running Nanbeige4.2-3B on one RTX 5070 with near-lossless Q8_0 weights and a 65,536-token context, both resident in VRAM. 65,536 is not an arbitrary round number: it is the max-new-tokens figure the model card itself recommends for agentic and tool-use work, and 12 GB is the smallest card class that holds that whole window without quantizing the weights down.

Hardware data: RTX 5070 (12 GB VRAM) · Q8_0 weights 4.130 GiB + q8_0 KV at 65,536 tokens 5.844 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 0.250 GiB = 10.541 GiB derived · no benchmark submitted yet · See benchmark data

⚠️ The KV cache is sized for 44 layers, not the 22 in config.json. Nanbeige4.2-3B is a Looped Transformer — the model card says "Its Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters." — and config.json sets num_loops: 2. The 22 blocks execute twice per forward pass over one shared set of weights, but each pass keeps its own keys and values. src/models/nanbeige.cpp does the expansion before anything is allocated — "Expand logical layer count before load_tensors() allocates layers / KV" — and src/llama-kv-cache.cpp then sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. Every context figure you compute from num_hidden_layers: 22 is exactly half the truth.

This is deliberate, not an oversight. A Nanbeige team member on discussion #18: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2".

A new generation, new memory, and the same 12 GB wall

NVIDIA's 50-series family spec table records this card as Blackwell with 12 GB GDDR7 on a 192-bit interface — a generation newer and a memory standard newer than the Ada parts it replaces, at an identical capacity. That combination makes it the clearest case in the 12 GB class of an upgrade that changes everything except the number this recipe is built around.

Everything below is a capacity budget: weight bytes, cache bytes, and two compute-buffer terms, summed against 12 GiB. GDDR7 sets how fast those bytes move; it does not add any. So the configurations that fit here are the same ones that fit on a two-generation-old 12 GB card, and the genuinely Blackwell-specific work on this page sits in the build rather than the budget — see the toolkit note under Requirements, which is the one thing here that will actually stop you.

What the 12 GB does buy, against an 8 GB card, is unusually tidy: one doubling at every cache tier, plus the weight tier, at the same time.

KV cache type	Largest context on 8 GB	Largest context on 12 GB
f16 (unquantized)	16,384	32,768
q8_0	32,768	65,536
q4_0	65,536	131,072

Every cell is the largest doubling whose four accounted terms — weights, KV cache, the reserved logits and the FlashAttention dequant scratch, all derived below — stay inside the card. The 8 GB column is computed with Q4_K_M weights (2.398 GiB), because that is the largest tier that leaves room for those contexts. The 12 GB column is computed with Q8_0 (4.130 GiB) — the extra capacity absorbs a 1.732 GiB weight upgrade and still doubles every window.

What 12 GB does not buy is the model's full declared context. The card states "The model supports a context length of up to 262,144 tokens (256K)." At 44 logical layers that cache is 12.375 GiB even at q4_0 — more than the entire card, before a single weight byte. Here 262,144 is reachable only by moving the cache to system RAM (last section of Running); in VRAM it needs a 24 GB card.

Requirements

Component	Minimum	This recipe
GPU	12 GB VRAM, CUDA compute capability 12.0 or newer	RTX 5070 (12 GB) — not measured; the budget below is derived from file bytes and llama.cpp's own allocation rule (/contribute)
RAM	8 GB system RAM	— (16 GB+ for the --no-kv-offload long-context mode)
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	4,434,787,168 bytes, from the HF tree API
Software	llama.cpp b10153+, CMake 3.18+, CUDA 12.8+	—

CUDA 12.8 is a hard floor here, and it is the single most likely thing to go wrong. NVIDIA's CUDA GPUs table lists the GeForce RTX 5070 under compute capability 12.0, and llama.cpp's ggml/src/ggml-cuda/CMakeLists.txt only reaches that architecture inside if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8"), where it appends list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real); the file's own annotation for the architecture reads 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores. Below 12.8 there is no Blackwell target for nvcc to compile against at all. Note the CMake floor is 3.18, not the 3.14 the root project asks for: ggml/src/ggml-cuda/CMakeLists.txt opens with cmake_minimum_required(VERSION 3.18) # for CMAKE_CUDA_ARCHITECTURES, and a -DGGML_CUDA=ON build always processes it.

A distribution's packaged CUDA is the usual culprit. The toolkit shipped in an older distribution release, or the one already installed for a previous card, is frequently 12.0–12.6 — new enough to build llama.cpp, too old to target this GPU. Check nvcc --version before the first build rather than after.

Do not follow the model card's llama.cpp instructions. The card still tells you to git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git. That was correct at release; mainline merged native nanbeige support in PR #25994 on 2026-07-27, and the tracking request issue #26086 was closed the same day. Third-party write-ups still repeating "requires the authors' branch" are stale.

The usable-VRAM assumption, stated up front

The card is 12 GiB (12,288 MiB), and the budget below accounts for four terms, not two. Weights and KV cache are the two you choose. The other two both live in llama.cpp's CUDA compute buffer, and on this recipe's configuration they come to 580.5 MiB between them. They flip no configuration on this page from fits to does-not-fit, but they are far too large to leave out of a sum, and at the long end they are what decides whether a configuration is desktop-viable or headless-only.

A flat 0.317 GiB of reserved logits. llama.cpp sizes one worst-case prompt-processing graph at startup, and src/llama-context.cpp sets the logit rows that graph must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is 512 rows regardless of -c, each a full f32 distribution over this model's unusually large 166,144-token vocabulary: 512 × 166,144 × 4 B = 340,262,912 B = 324.5 MiB. It is large only because the vocabulary is, and it scales with -ub, not with context.

4,096 bytes per token of FlashAttention dequant scratch — but only because the cache is quantized. The FA kernels read f16, so a quantized cache is expanded into scratch VRAM first. ggml/src/ggml-cuda/fattn.cu sets need_f16_K = need_f16_V = true for the tile and MMA kernels, and fattn-common.cuh then adds ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16) for K and again for V — each behind if (need_f16_K && K->type != GGML_TYPE_F16), so an f16 cache pays exactly none of it. For this model that is (1024 + 1024) × 2 B per context token: 0.125 GiB at 32,768, 0.250 GiB at 65,536, 0.500 GiB at 131,072. It is per-layer and transient rather than ×44 — ggml_nelements(K) is one layer's view over the cache, and ggml's allocator reuses the space across the 44 unrolled layers.

Two properties of that second term are worth stating, because both are counter-intuitive:

• It is charged at the reserve, not just while a prompt is processing, and it does not depend on which GPU you own. The graph llama.cpp sizes at startup is the prompt-processing one, at n_tokens = min(n_ctx, n_ubatch) = 512 rows, so the query tensor has Q->ne[1] = 512. In ggml_cuda_get_best_fattn_kernel the vector kernel — the only one that reads a quantized cache directly and needs no scratch — is gated on Q->ne[1] <= 2 at compute capability 8.9 and above, on Q->ne[1] == 1 below that, and on a trailing !gqa_opt_applies && Q->ne[1] == 1. A 512-row reserve fails all three, so the function returns BEST_FATTN_KERNEL_MMA_F16, which sets both need_f16 flags. This term is a property of the reserved graph's shape, not of the silicon; the compute-capability branches in that function differ only at decode, where Q->ne[1] is 1 or 2 — and decode cannot un-allocate what the reserve already claimed.
• -ub does not shrink it. Its size comes from -c, not from the micro-batch. Halving -ub halves the logits reservation and leaves this term exactly where it was.

Two more consumers sit on the same 12 GiB and are deliberately not in the accounted total, because nothing here measures them:

• The CUDA runtime's own context. A few hundred MiB per process that llama.cpp never reports. Budget for it; do not assume it away.
• Your desktop. A compositor driving a monitor off the same card typically holds several hundred MiB more.

So: 10.541 GiB accounted, 1.459 GiB left on a 12 GiB card for those two. On a headless card the lead configuration has comfortable room. On a desktop card driving a 4K monitor it is snug — and if it is too snug, the first lever is -ub 256, which buys back 162 MiB without costing a single token of context. (Budget one more thing outside VRAM entirely: the attention mask is n_kv × n_ubatch in f16 — 64 MiB here — and src/llama-graph.cpp asserts ggml_backend_buffer_is_host(self_kq_mask->buffer), so it is pinned system RAM, not card memory. Counting it as VRAM is a common way to over-budget this model.)

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

Verify the toolkit first on this card. nvcc --version must report 12.8 or newer; llama.cpp appends the Blackwell architecture only above that version, so an older toolkit cannot produce a working binary here whatever else is configured. With a current toolkit no architecture flags are needed — a normal (non-cross-compiling) build sets CMAKE_CUDA_ARCHITECTURES to native, and llama.cpp rewrites a plain 120 into the architecture-specific 120a form itself.

Confirm the checkout actually carries the architecture — this file does not exist in builds older than 2026-07-27:

test -f src/models/nanbeige.cpp && echo "nanbeige support present"

2. Download the Q8_0 GGUF

Nanbeige publishes no first-party GGUF — the org's quantized builds of this model are FP8 and GPTQ-Int8 only — so this uses a community conversion. owao/Nanbeige4.2-3B-GGUF ships a 13-rung ladder; bartowski/Nanbeige_Nanbeige4.2-3B-GGUF is a 23-rung alternative. Both re-uploaded after the vendor's 2026-07-27 tokenizer fix. Pick one repo and stay in it: the two quantizers' files are not byte-identical, and mixing their numbers into one budget is how a sum goes quietly wrong (their Q4_K_M builds differ by 109 MB).

pip install -U huggingface_hub numpy

hf download owao/Nanbeige4.2-3B-GGUF \
  Nanbeige4.2-3B-Q8_0.gguf \
  --local-dir ./models

numpy is not a dependency of huggingface_hub, and the verification step below needs it.

3. Verify the artifact carries the loop parameter

Worth doing by hand exactly once. If a GGUF was produced without num_loops, llama.cpp defaults the key to 1, runs 22 layers instead of 44, and you get a silently different model — no error, no warning, half the depth.

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ./models/Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

Two lines come back, ending = 22 and = 2 respectively. (The leading index numbers vary from file to file; only the key names and the values matter.) A missing num_loops line is the failure case.

Two commands that look like they should do this job do not, so do not substitute them. llama-gguf <file> r n prints key names only, never their values. And llama-cli suppresses the loader's metadata dump at its default verbosity — besides which it is an interactive chat client that will sit waiting for input rather than exiting.

Running

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 65536 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 1.0 --top-p 0.95 --top-k 20

Open http://127.0.0.1:8080 for the built-in chat UI, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

--temp 1.0 with a 65,536-token budget is the model card's own recommendation for agentic and tool-use tasks. For reasoning and chat it recommends --temp 0.6; the card pairs that with 131,072 new tokens, which on this card means the q4_0 configuration below.

Four flags are load-bearing:

• -c 65536, set explicitly. Omitting -c is not neutral: llama.cpp's -fit on default interpolates the context downward until the model fits free memory, so you silently get some reduced number. -c 0 is not neutral either, and it is the more dangerous of the two — common/arg.cpp reads it as "give me the full trained context, do not shrink it" and sets fit_params_min_ctx = UINT32_MAX, which at the f16 cache default means a 44 GiB allocation on a 12 GiB card and an immediate abort.
• -fa on. A quantized V cache requires Flash Attention; without it llama.cpp refuses with quantized V cache was requested, but this requires Flash Attention. The auto default would enable it for you, but being explicit makes the failure mode legible.
• -ctk q8_0 -ctv q8_0 — matched types. A stock CUDA build instantiates Flash Attention kernels only for identical K and V types: ggml/src/ggml-cuda/fattn.cu contains if (K->type != V->type) { return BEST_FATTN_KERNEL_NONE; } outside GGML_CUDA_FA_ALL_QUANTS, and its default table covers f16, bf16, q4_0 and q8_0 in matched pairs only. A clever-looking -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON.
• Leave --parallel alone. There is no ×4 multiplier hiding in these numbers. tools/server/server.cpp resolves the default -np -1 with params.n_parallel = 4; params.kv_unified = true; in the same branch — unified means one shared pool of -c cells that a single conversation may consume in full, so the 5.844 GiB above is the whole cache bill. Passing an explicit positive --parallel N skips that branch, leaves kv_unified = false, and gives each conversation -c / N for identical memory. If you want one slot, say -np 1.

The maximum-context configuration

Trade cache precision for window. q4_0 at 131,072 tokens is 6.188 GiB of cache, and with the same Q8_0 weights the accounted total is 11.135 GiB — inside the card, but with only 0.865 GiB left for the CUDA context and your display. Treat this one as headless-friendly and desktop-marginal. Doubling the context also doubles the dequant scratch to 0.500 GiB, which is the point at which it overtakes the logits reservation and becomes the larger of the two compute-buffer terms:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 131072 -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

That covers the card's recommended 131,072-token reasoning budget.

If you would rather keep the near-lossless cache and give up context instead, -c 32768 -ctk f16 -ctv f16 runs the cache completely unquantized. It is also, counter-intuitively, the roomiest of the three Q8_0 configurations on this page: 9.947 GiB accounted, 2.053 GiB left over, against the lead configuration's 1.459 GiB and this section's 0.865 GiB. The KV column makes the two look close (5.500 GiB against 5.844 GiB), but an f16 cache pays no dequant scratch at all, so the real gap is 0.594 GiB rather than 0.344 GiB. Quantizing the cache is a way to buy context, not a way to save memory — and on this card the configuration framed as the sacrifice turns out to have the most headroom.

There is a second, unmeasured reason to prefer the f16 cache when you do not need the window. The dequant this recipe budgets in VRAM also has to happen on every pass, and a community user raised exactly that on 2026-08-10 in [discussion #28] on the canonical repository(https://huggingface.co/Nanbeige/Nanbeige4.2-3B/discussions/28): "Currently any implementation with quantized kv cache still reads quantized fp16 instead of the quantized kv cache format." — his conclusion being that this multiplies DRAM traffic and slows long-context inference. No measurement is attached to that thread and none of it changes a byte of the budget above; treat it as a reason to check f16 against your own workload rather than as a published number.

Reaching the full 262,144 tokens

Not possible in VRAM here: 12.375 GiB of q4_0 cache alone exceeds the card. Keep the weights on the GPU and put the cache in system RAM:

./build/bin/llama-server \
  -m ./models/Nanbeige4.2-3B-Q8_0.gguf \
  -ngl 99 -c 262144 -fa on \
  -ctk q4_0 -ctv q4_0 --no-kv-offload

This needs roughly 13 GB of free system RAM and trades decode speed for window size. Treat it as a capability, not a default. A community user running an 8 GB card annotates both modes in the entrypoint script he posted on discussion #18 — "KV_OFFLOAD=1 (default) keeps KV in VRAM: fastest, ctx up to ~64k." — those are his own script comments for a smaller card, not a published benchmark.

Results

• VRAM usage: 10.541 GiB accounted at the lead configuration — 4.130 GiB of Q8_0 weights, 5.844 GiB of q8_0 cache at 65,536 tokens, 0.317 GiB of reserved logits and 0.250 GiB of FlashAttention dequant scratch — leaving 1.459 GiB of the card's 12 GiB for the CUDA context, the activation working set and your desktop. Derived from measured file bytes and llama.cpp's own allocation rules, not measured on hardware; see /check/nanbeige4-2-3b/rtx-5070. Separately, budget ~64 MiB of system RAM for the pinned attention mask.
• Throughput: no measurement exists for this pair, and none exists for this model on any 12 GB NVIDIA card. I enumerated all 28 discussion threads on the canonical repository and all 3 on the distribution repository, searched llama.cpp's tracker for nanbeige (5 results, all architecture-support or tool-call parsing, none GPU-specific), and searched the open web; the only throughput figures anywhere are a community user's on an 8 GB RTX 5060, posted in discussion #18 at a different weight tier and cache type from this recipe's — its 65,536-token context is the same, and that is the only thing about it that is. Publishing a number derived from that would be inventing one — and it is the nearest miss in the batch, because that board shares this card's Blackwell generation while differing in capacity, tier and cache type, which makes its figures an optimistic ceiling rather than a comparable number. If you run this configuration, please contribute the measurement so /check/nanbeige4-2-3b/rtx-5070 stops being empty.
• Quality notes: the model is ~4.17 B parameters total, ~3.15 B non-embedding — the "3B" in the name counts the non-embedding half, and the extra billion is the untied 166,144-token vocabulary at both ends of the stack. I found no evaluation of this model under a quantized KV cache at any tier: treat q8_0 cache as the conservative choice on the ecosystem's general reputation, and q4_0 as the aggressive one, rather than as a measured quality difference on this architecture.

Where the numbers come from

llama.cpp sizes the cache as n_layer × (n_embd_k_gqa + n_embd_v_gqa) × bytes_per_element per token. config.json gives num_key_value_heads: 8 and head_dim: 128, so each layer stores 8 × 128 = 1024 elements for K and 1024 for V. Over 44 logical layers that is 90,112 elements per token:

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224
q8_0	34/32 = 1.0625	95,744
q4_0	18/32 = 0.5625	50,688

The block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 elements, from ggml/src/ggml-common.h. Multiplying out:

Context	f16 KV	q8_0 KV	q4_0 KV
16,384	2.750 GiB	1.461 GiB	0.773 GiB
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
98,304	16.500 GiB	8.766 GiB	4.641 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

A quantized cache also carries a flat 4,096 bytes per token of FlashAttention dequant scratch, derived in the usable-VRAM section — zero on an f16 cache. So the price of a quantized cache is the table above plus 4,096: 99,840 B/token effective at q8_0, 54,784 at q4_0.

Adding weights and both compute-buffer terms, the practical envelope on 12 GiB. Weight byte counts are from the owao repository via the HuggingFace tree API:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Unallocated on 12 GiB
Q8_0 · q8_0 · 65,536 (this recipe)	4.130	5.844	0.317	0.250	10.541 GiB	1.459 GiB
Q8_0 · f16 · 32,768	4.130	5.500	0.317	0.000	9.947 GiB	2.053 GiB
Q8_0 · q4_0 · 131,072	4.130	6.188	0.317	0.500	11.135 GiB	0.865 GiB — headless only
Q4_K_M · q4_0 · 131,072	2.398	6.188	0.317	0.500	9.402 GiB	2.598 GiB
Q4_K_M · q8_0 · 98,304	2.398	8.766	0.317	0.375	11.855 GiB	0.145 GiB — no room for the CUDA context
Q8_0 · q8_0 · 98,304	4.130	8.766	0.317	0.375	13.588 GiB	will not fit
Q4_K_M · f16 · 65,536	2.398	11.000	0.317	0.000	13.715 GiB	will not fit

The Q4_K_M rows are there to show what dropping the weight tier actually buys, and the answer is: not much. Q8_0 costs 1.732 GiB more than Q4_K_M, which spent on cache instead would buy 18,629 more tokens at q8_0 or 33,951 at q4_0 — under 30% more window in the first case. (Both at the effective per-token cost, dequant scratch included; on the sticker KV price alone they would read 19,426 and 36,694, and that gap is the tax.) On a model whose entire pitch is reasoning and agentic accuracy at 3B, that is the wrong trade on a card with room.

Troubleshooting

Out of memory at startup

Almost always the default f16 cache. At 65,536 tokens that is 11.000 GiB before any weights — see the table above. Add -ctk q8_0 -ctv q8_0.

If it still fails, your display is holding VRAM, and the levers go in this order: -ub 256, then -c 49152, then -ctk q4_0 -ctv q4_0. Reach for -ub first because it is the only one that costs no context — but be clear about how much it buys. It halves the logits half of the compute buffer, from 324.5 MiB to 162.2 MiB, and does not touch the 256.0 MiB of dequant scratch at all, because that term is sized by -c. So -ub 256 cuts about 28% of the lead configuration's 580.5 MiB compute buffer, not half of it, and -ub 128 about 42%. Past that the cache tier is the bigger lever: q8_0 → q4_0 saves the full 45,056 bytes per token and adds nothing back, since the scratch is a flat 4,096 bytes per token on both.

nvcc reports an unsupported GPU architecture

The toolkit predates Blackwell. llama.cpp resolves CMAKE_CUDA_ARCHITECTURES to native on a machine that has the card, which asks nvcc for compute capability 12.0 — a target CUDA only gained at 12.8. Install CUDA 12.8 or newer, then delete build/ and reconfigure; a stale CMake cache keeps the old toolkit path otherwise. Confirm the result in the configure output's Using CMAKE_CUDA_ARCHITECTURES= line, which should carry a 120a entry.

unknown model architecture: 'nanbeige'

The build predates b10153. Rebuild from mainline master, or take a release tag at or above b10153 — that build's commit is the merge of PR #25994. Distribution channels lag: the model card notes that LM Studio's bundled llama-server does not support nanbeige and tells you to copy your own build's binaries into the LM Studio backend directory.

Tool calls come back as plain text instead of executing

Fixed upstream in release tag b10227 — check what you built before you change anything else. It was a llama.cpp parser bug, not a model bug and not your configuration: the model sometimes emits <tool_call> followed by a space rather than a newline, and the auto-generated parser required that newline, so the reply was kept as content and the tool-call rule never fired. PR #26324 put the loss at roughly a quarter of calls and the model's own discussion #17 at roughly one in six — both community reports rather than a maintainer measurement, and #17's rate was taken on the vendor fork at d28da86 rather than on a mainline build, so read the pre-fix behaviour as "some calls" and not as a constant. That PR is not what repaired it: it never merged, a maintainer having objected that blanket whitespace trimming was rolled back once before for hurting other models' output. The fix came from PR #26252, a specialized parser merged on 2026-08-02 and released as b10227, and on 2026-08-10 #26324's author closed his own pull request reporting tool calls working every time on a newer build.

So tool use carries a build floor of its own, one notch above the architecture's: b10153 loads the model, b10227 parses its tool calls. If calls still come back as text, that is what your build predates — rebuild. So does the temperature workaround: the --temp 1.0 this recipe recommends for agentic work needs no hedge on a binary at or above b10227.

response_format: json_schema fails before any token is generated

Grammar-constrained decoding aborts at sampler init with Failed to initialize samplers: Unexpected empty grammar stack. Two users reproduced it independently on discussion #6 — but both were running the vendor fork rather than the mainline build installed here, so treat the status on mainline as untested rather than known-broken. If you hit it, restart the server with --no-jinja; plain generation is unaffected either way, and the cost is that you apply the prompt format yourself.

ollama pull cannot find the model

There is no entry in the official Ollama library, and the model card's own ./ollama run nanbeige/nanbeige4.2:3b-Q4_K_M names a namespace that returns 404 on ollama.com. Community re-uploads exist under user namespaces, but most search hits for "nanbeige" are 4.1 builds — a different model generation with different architecture behaviour. Use llama-server as above.

The model loads but the output is subtly poor

Re-run step 3 and confirm the GGUF reports nanbeige.num_loops = 2. A file converted by a path that omits the key loads happily and runs at half depth. Also confirm the build post-dates 2026-07-27; older mainline builds refuse to load the architecture outright rather than misbehave.

Anything else, or a real throughput measurement on this card, is welcome via the submission form.