Nanbeige4.2-3B on RX 7900 XTX: the full 256K context on a ROCm llama.cpp build

What You'll Build

A llama-server endpoint running Nanbeige4.2-3B at 262,144 tokens of context, entirely in the RX 7900 XTX's VRAM, on a HIP build of llama.cpp targeting gfx1100. The weights are near-lossless Q8_0; the cache is q4_0; and the reason the cache is the interesting part is that this model allocates twice the KV you would compute from its config.

Hardware data: RX 7900 XTX (24 GB VRAM) · Q8_0 weights 4.130 GiB + q4_0 KV at 262,144 tokens 12.375 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 1.000 GiB = 17.822 GiB accounted · See benchmark data

⚠️ A 3B’s weights are not the constraint here — the KV cache is. Nanbeige4.2-3B is a Looped Transformer: its 22 blocks are executed twice per forward pass over one shared set of weights. The weights are counted once, but llama.cpp allocates KV for 44 logical layers, so every context figure you would compute from num_hidden_layers: 22 is exactly half the truth. At f16 the full 262,144-token cache is 44.000 GiB — nearly twice the card. Everything below exists to get around that.

ℹ️ This recipe is in two halves and they fail independently. The platform half — ROCm, the gfx1100 build, which llama.cpp flags mean what on the HIP backend — is the same on every RDNA3 Radeon. The memory half below is arithmetic against 24 GB and changes card by card. If you are adapting this to a smaller Radeon, keep the first and redo the second.

Requirements

Component	Minimum	This recipe
GPU	24 GB VRAM, ROCm-supported AMD GPU	— not measured; the budget below is derived (/contribute)
RAM	8 GB system RAM	—
Storage	4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it)	~15 GB if you also clone and build llama.cpp against ROCm
Software	Linux, ROCm ≥ 6.1, llama.cpp b10153+, CMake	—

The gfx target decides the build, and it is per-board

Everything in the install hangs off one string. AMD's ROCm install-on-linux system-requirements matrix — which documents ROCm 7.14.0 at the time of writing — lists the RX 7900 XTX with LLVM target gfx1100, officially supported on Linux. llama.cpp's own build guide agrees, saying of its gfx1100 example "that corresponds to Radeon RX 7900XTX/XT/GRE" (docs/build.md).

Two consequences worth stating before you type anything:

• Read the target off AMD's per-board matrix, not off a series-level compatibility page. The matrix has one row per board and gives support status alongside the target id. The LLVM AMDGPU processor list that docs/build.md links to does map boards to targets correctly — the gfx1100 row names the RX 7900 XTX, the gfx1101 row below it the RX 7800 XT — but it is a compiler-backend reference, not a support matrix, and it is the wrong table to answer "will ROCm run on my card". Its board column is headed "Example Products" and is blank or TBA for 39 of its 68 processor rows, and its support column is captioned "see amdgpu-os and corresponding runtime release notes for current information and level of support" — the gfx1100 cell there lists only pal-amdpal and the gfx1101 cell is empty, neither of which is what AMD's own matrix says about those boards. Confirm a target id against it; do not conclude anything about ROCm support from it.
• Do not set HSA_OVERRIDE_GFX_VERSION. That variable exists to make an unsupported card masquerade as a supported one; docs/build.md introduces it with "If your GPU is not officially supported". gfx1100 is officially supported, so setting it can only mislead the runtime. Confirm your own card rather than trusting this paragraph:

rocminfo | grep gfx | head -1 | awk '{print $2}'
# expect: gfx1100

The KV cache doubles, and that is not a bug

1. The vendor designed it that way. config.json carries "num_loops": 2 alongside "num_hidden_layers": 22. A Nanbeige team member, replying on the model's own discussion #18 (leran1995, flagged as an org member): "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2". Cache sharing was built, measured and rejected — the doubling is the design.

2. llama.cpp implements it that way. src/models/nanbeige.cpp expands the logical layer count before allocation — hparams.n_layer_all = (uint32_t) ((size_t) n_layer_phys * (size_t) n_loops); under a comment reading "Expand logical layer count before load_tensors() allocates layers / KV" — then aliases the layer structs across loops under a second comment: "Share physical weights across loops; each slot still has its own KV index." src/llama-kv-cache.cpp sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. For nanbeige with num_loops: 2 that is 44, not 22.

3. Your file has to carry the flag. The loader reads the key as optional with a default of 1 — uint32_t n_loops_u = 1; ml.get_key(LLM_KV_NUM_LOOPS, n_loops_u, false); — so a GGUF converted by a path that omits it loads happily and runs 22 layers. That is a silently different model, not an error. Check the file you downloaded:

# run from your llama.cpp checkout (see Installation step 2)
pip install -U huggingface_hub numpy
python3 gguf-py/gguf/scripts/gguf_dump.py --no-tensors \
  ../nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf | grep -E "num_loops|block_count"

numpy is not a dependency of huggingface_hub, so install both. Two shorter-looking commands do not substitute here: llama-gguf <file> r n prints key names only, and piping llama-cli into grep shows nothing because tools/cli/cli.cpp sets params.verbosity = LOG_LEVEL_ERROR; before the metadata is printed.

At runtime, llama_kv_cache prints the layer count in its startup line. Confirm 44 layers before trusting anything below; if it says 22 layers, your cache is half-size and your context is wrong.

KV cost per token, and what 24 GB buys

config.json gives num_key_value_heads: 8 and head_dim: 128, so one layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Over 44 logical layers that is 90,112 elements per token — double what the 22 in config.json would suggest.

Cache type	Bytes per element	Bytes per token
f16 (default)	2	180,224 (176.0 KiB)
q8_0	34/32 = 1.0625	95,744 (93.5 KiB)
q4_0	18/32 = 0.5625	50,688 (49.5 KiB)

Block sizes are sizeof(block_q8_0) = 34 bytes and sizeof(block_q4_0) = 18 bytes per 32 elements, from ggml-common.h. Multiplying out:

Context	f16 KV	q8_0 KV	q4_0 KV
32,768	5.500 GiB	2.922 GiB	1.547 GiB
65,536	11.000 GiB	5.844 GiB	3.094 GiB
131,072	22.000 GiB	11.688 GiB	6.188 GiB
262,144	44.000 GiB	23.375 GiB	12.375 GiB

Two more terms before you add weights

Weights plus cache is the bulk of the bill but not all of it. One of the two remaining terms is flat; the other is a full gigabyte at this context, and neither is AMD-specific.

A flat 0.317 GiB of reserved logits. llama.cpp sizes one worst-case graph at startup, and src/llama-context.cpp sets the logit rows that graph must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is 512 rows regardless of -c, each a full f32 distribution over this model's 166,144-token vocabulary:

512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB

It is large only because this vocabulary is, and the dial that moves it is -ub, not -c — so it is the same 0.317 GiB in every row of the table below.

And the dequant scratch, which only a quantized cache pays. A quantized KV cache buys its saving back a little at prompt-processing time, and this is the one place a 24 GB budget for this model can quietly go wrong.

llama.cpp's FlashAttention dispatch picks a kernel per call. On RDNA3 the host-side predicate amd_wmma_available(cc) is true (ggml/src/ggml-cuda/common.cuh defines GGML_CUDA_CC_RDNA3 as GGML_CUDA_CC_OFFSET_AMD + 0x1100, commented "RX 7000, minimum for WMMA"), so a large-batch prefill routes to the WMMA MMA kernel while single-token decode falls to the vector kernel. The two differ in what they need from a quantized cache: ggml/src/ggml-cuda/fattn.cu sets need_f16_K = true; need_f16_V = true; for the tile and MMA kernels and only for GGML_TYPE_F32 inputs on the vector kernel. When those flags are set, fattn-common.cuh appends ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16) — and the same again for V — to the attention output tensor's allocation.

So on the prefill graph, at -c 262144, with K and V shaped 128 × 262144 × 8:

128 × 262,144 × 8 × 2 bytes = 536,870,912 B for K, the same for V → 1,073,741,824 B = 1.000 GiB of compute buffer that an f16 cache would not need. It scales linearly with -c, so it is 0.500 GiB at 131,072 and nothing at all when -ctk f16 -ctv f16 (the code skips the conversion when the tensor is already f16).

Now the card:

Configuration	Weights	KV	Logits	FA scratch	Accounted total	Slack on 24 GiB
Q8_0 · q4_0 KV · 262,144 ctx (this recipe)	4.130	12.375	0.317	1.000	17.822 GiB	6.178 GiB
Q8_0 · q8_0 KV · 131,072 ctx	4.130	11.688	0.317	0.500	16.635 GiB	7.365 GiB
Q8_0 · f16 KV · 65,536 ctx	4.130	11.000	0.317	0.000	15.447 GiB	8.553 GiB
bf16 · q4_0 KV · 131,072 ctx	7.771	6.188	0.317	0.500	14.775 GiB	9.225 GiB

Weight byte counts are from the bartowski GGUF repo via the HuggingFace tree API.

The verdict is the same as on a 24 GB NVIDIA card, and I checked two AMD-specific reasons it might not have been. Neither the KV arithmetic nor the scratch arithmetic contains anything backend-specific — the same fattn.cu and the same ggml-common.h compile into the HIP backend. What could have differed is the allocator, and there is a real, documented ROCm problem there: on a GPU without working virtual memory management, llama.cpp falls back to a legacy pool that retains buffers at peak size, and issue #22107 reported exactly that shape — FlashAttention temporaries growing with context until the process OOMs, on ROCm, worse with a quantized cache than with f16. It does not apply to the build this recipe pins. At tag b10153 the f16 dequant temporaries are already taken from the destination tensor's own allocation via ggml_cuda_flash_attn_ext_get_f16_extra_data, not from the pool — which is why they are counted as compute buffer in the table above rather than as an unbounded runtime leak. The residual ROCm pool issue is still live for other allocations; it has a Troubleshooting entry below.

The remaining 6.178 GiB is not spare — it absorbs the HIP context, the rest of llama.cpp's compute buffer for a 44-layer unrolled forward pass, ROCm's allocation granularity, and the framebuffer if this card also drives your displays. None of that is measured on this GPU, which is why the recipe stops at the declared context instead of arguing for more.

Separately, budget 256 MiB of system RAM for the attention mask. It is deliberately absent from every figure above, because it is not VRAM: src/llama-graph.cpp asserts ggml_backend_buffer_is_host(self_kq_mask->buffer) when it fills the mask, so it is a host allocation on the HIP backend exactly as it is on CUDA. At this context it is n_kv × n_ubatch × 2 B = 262,144 × 512 × 2 = 268,435,456 B — the largest host mask of any configuration on this page, and comfortably inside the 8 GB of system RAM the Requirements table asks for. Counting it as VRAM is a common way to over-budget this model.

Why Q8_0, and the two rungs either side

Within the bartowski repo, Q8_0 (4,434,787,488 B) costs 1.631 GiB more than Q4_K_M (2,684,023,968 B). Spent on cache instead, 1.631 GiB buys 31,957 more q4_0 tokens — 12% of a ceiling you already reach. (That is at the effective per-token cost of 54,784 B: the 50,688 B of cache plus the 4,096 B of dequant scratch that each extra context token drags in on a quantized cache. On the sticker KV price alone it would read 34,540, and the gap between the two is the tax.) On a model whose pitch is agentic and reasoning accuracy at 3B, that is the wrong trade.

Upward, bf16 (8,343,845,760 B = 7.771 GiB) is the one rung where RDNA3 has a genuine edge over a same-capacity NVIDIA card: in ggml-cuda.cu's batched hipBLAS path, prefer_f32_output = !GGML_CUDA_CC_IS_RDNA3(cc) && !GGML_CUDA_CC_IS_CDNA(cc) for a BF16 compute type — RDNA3 is one of only two architectures that keeps a BF16 output type instead of widening to F32. It still costs 3.641 GiB over Q8_0 and cannot reach 262,144 tokens on this card, so it is a row in the table, not the lead.

Quantizers differ and their bytes are not interchangeable: owao's Q8_0 is 4,434,787,168 B (320 bytes apart, same to three decimals) but its Q4_K_M is a different file at 2,574,807,904 B = 2.398 GiB. owao ships a 13-rung ladder, bartowski a 23-rung one. Pick one repo and do not mix their numbers in one sum.

Installation

1. Install ROCm

Follow AMD's ROCm quick start for Linux for your distribution. The system-requirements matrix linked above is the authority on which OS releases are supported, and support is per board — read your card's row rather than assuming every distribution listed on the page applies to it.

The version floor is not advisory. ggml/src/ggml-hip/CMakeLists.txt hard-stops the build:

if (${hip_VERSION} VERSION_LESS 6.1)
    message(FATAL_ERROR "At least ROCM/HIP V6.1 is required")
endif()

Verify before you go further:

hipconfig --version
rocminfo | grep gfx | head -1 | awk '{print $2}'

2. Build llama.cpp for gfx1100

Mainline support for this architecture arrived in PR #25994, merged 2026-07-27. The first tagged build containing that merge is b10153 — src/models/nanbeige.cpp does not exist at tag b10152 and does at b10153, which is why every source link in this recipe is pinned to that tag. The model card still tells you to clone the vendor's own nanbeige42 fork; that was correct at release and is not any more. Mainline carries the architecture and mainline gets the bug fixes.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release \
  && cmake --build build --config Release -- -j 16

That is docs/build.md's own Linux invocation with the target set to this card. GPU_TARGETS is optional — omitting it builds for every GPU present — but naming it keeps the compile to one ISA and makes the binary's target explicit.

Note what you are not doing. There is no flash-attn pip step: llama.cpp's FlashAttention is its own HIP kernel set, compiled by the command above, and GGML_CUDA_FA defaults to ON in ggml/CMakeLists.txt. There is no bitsandbytes, no ExLlamaV2 and no FP8 or FP4 weight path — RDNA3's matrix units take FP16, BF16, INT8 and INT4, so an FP8 checkpoint would upcast rather than save memory. GGUF through this HIP build is the path.

3. Download the weights

Nanbeige publishes no first-party GGUF — the org's quantized releases are FP8 and GPTQ-Int8, and a team member said on discussion #1 that official quantized versions were being prepared, which has not happened yet. Use a community conversion:

pip install -U huggingface_hub

hf download bartowski/Nanbeige_Nanbeige4.2-3B-GGUF \
  --include "Nanbeige_Nanbeige4.2-3B-Q8_0.gguf" \
  --local-dir ./nanbeige4.2-3b

Check provenance on whichever repo you pick, because this model has a date cliff: mainline support merged on 2026-07-27, hours after the canonical repo landed a tokenizer commit the same day, and a quant is a frozen copy of upstream as of the moment it was uploaded. bartowski's card states the quants were built with llama.cpp release b10159, and the HuggingFace API gives that repo createdAt 2026-07-28T14:38Z — comfortably after both events. owao's repo has an earlier createdAt (2026-07-21T20:33Z) but a lastModified of 2026-07-29T13:06Z, so check the individual file's commit before assuming a given rung there is post-merge.

Running

The long-context configuration

./build/bin/llama-server \
  -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 262144 \
  -fa on \
  -ctk q4_0 -ctv q4_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

That is the full declared context — "The model supports a context length of up to 262,144 tokens (256K)." — resident in VRAM on one card, at 17.822 GiB accounted.

Four flags are load-bearing, and it matters which of them are about the runtime rather than the backend, because only the runtime ones are the same advice you would give a CUDA reader:

• -c 262144 — set it explicitly, and never -c 0. Omitting -c is not the same as passing 0: with -c unset, llama.cpp's -fit on default (common/fit.cpp) interpolates the context down until the model fits free memory with a margin, so you silently get some arbitrary reduced number. With -c 0, common/arg.cpp sets fit_params_min_ctx = UINT32_MAX — a deliberate "give me the full trained context, do not shrink it" — and you get 262,144 cells with the f16 default cache, i.e. 44.000 GiB, i.e. an immediate out-of-memory abort. This is backend-independent argument parsing; it reads identically on ROCm.
• -fa on. A quantized V cache requires FlashAttention; src/llama-context.cpp throws quantized V cache was requested, but this requires Flash Attention otherwise. The auto default will enable it for you, but being explicit makes the failure mode legible.
• -ctk q4_0 -ctv q4_0 — matched types, and this one really is a backend question, with the same answer. A stock build instantiates FlashAttention kernels only for identical K and V types, and the restriction is not inherited from the CUDA side by accident: the guard in fattn.cu is #ifndef GGML_CUDA_FA_ALL_QUANTS / if (K->type != V->type) { return BEST_FATTN_KERNEL_NONE; } with no backend condition on it, and ggml/src/ggml-hip/CMakeLists.txt carries its own copy of the same option, whose default branch compiles exactly four vector instances: f16-f16, q4_0-q4_0, q8_0-q8_0, bf16-bf16. A clever-looking -ctk q8_0 -ctv q4_0 needs a rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON — the HIP build file reads the same variable name, so the escape hatch transfers too.
• Leave --parallel alone. llama-server defaults it to -1, and tools/server/server.cpp resolves the sentinel with params.n_parallel = 4; params.kv_unified = true; in the same branch. Unified means one shared cache of -c cells that any single slot may consume in full, so the table above is the whole KV bill and one conversation can still reach 262,144. Passing -np 4 explicitly skips that branch, leaves kv_unified = false, and src/llama-context.cpp then computes cparams.n_ctx_seq = cparams.n_ctx / cparams.n_seq_max — same total memory, a quarter of the context per conversation. If you want one slot, say -np 1, not -np 4. Also runtime, not backend.

The quality-first alternative

If you would rather not run a 4-bit K cache, halve the context and keep the near-lossless one — 16.635 GiB accounted, and 131,072 is exactly the max-new-tokens the model card recommends for reasoning and chat:

./build/bin/llama-server \
  -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 \
  -c 131072 \
  -fa on \
  -ctk q8_0 -ctv q8_0 \
  --temp 0.6 --top-p 0.95 --top-k 20

For agentic and tool-use work the card recommends --temp 1.0 instead, with 65,536 new tokens.

Environment variables: two to know, one not to set

• HIP_VISIBLE_DEVICES selects which GPU the process sees — the documented way to pin a card in a multi-GPU box.
• GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 exists but is for integrated graphics. docs/build.md is explicit: "However, this hurts performance for non-integrated GPUs". Leave it unset on a discrete Radeon.
• HSA_OVERRIDE_GFX_VERSION — do not set it, per the gfx-target section above.

Results

• Speed: omitted. There is no measurement of this model on any AMD GPU that I could find. I searched: /check/nanbeige4-2-3b/rx-7900-xtx (zero benchmarks); all 27 discussion threads on the canonical HuggingFace repo, fetched individually — the only hardware datapoint in any of them is on an 8 GB NVIDIA card, and the strings amd, rocm, radeon, gfx and 7900 appear nowhere except once as the CPU ISA "AMD64"; llama.cpp's issue and PR search for nanbeige, which returns 5 items, none hardware-related, and zero when crossed with gfx1100, ROCm, AMD or Radeon; and open-web search for the model name against this card. Inventing a figure from a different vendor's silicon would be inventing one. If you run this, please contribute your measurement so the next reader gets a real number.
• VRAM usage: 17.822 GiB accounted at the lead configuration (4.130 GiB weights + 12.375 GiB cache + 0.317 GiB reserved logits + 1.000 GiB FlashAttention dequant scratch), leaving 6.178 GiB of the card's 24 GiB; 16.635 GiB at the q8_0/131,072 alternative. Both derived, not measured — see /check/nanbeige4-2-3b/rx-7900-xtx. Separately, budget ~256 MiB of system RAM for the pinned attention mask.
• Quality notes: I found no evaluation of this model under a quantized KV cache at any tier — not in the model card, not in the 27 canonical discussions, not in either GGUF repo's card, not in llama.cpp's tracker. Treat that as a gap, not as evidence the loss is small. q8_0 cache is generally treated as near-lossless across the llama.cpp ecosystem; q4_0 is the aggressive rung, and the reason the alternative configuration above exists. The vendor publishes no throughput figures of its own at any hardware tier — the card carries quality benchmarks only, with a sampling table (temperature 0.6 / 131,072 max new tokens for reasoning and chat; 1.0 / 65,536 for agentic and tool use) and nothing about speed.
• Context is trained, not extrapolated: rope_scaling is null in config.json, so 262,144 is a training target rather than a RoPE-scaling claim.

For the full benchmark data, see /check/nanbeige4-2-3b/rx-7900-xtx.

Troubleshooting

llama_model_load: error loading model architecture: unknown model architecture: 'nanbeige'

Your build predates b10153. Rebuild from mainline master, or use a release tag at or above b10153 — that build's commit is the merge of PR #25994. On discussion #23 one user reports still being unable to load the architecture and a second reports that a later mainline release loads it. Distribution channels lag: the model card itself notes that LM Studio's bundled llama-server does not support nanbeige.

clang: error: cannot find ROCm device library

A ROCm install-layout problem, not a llama.cpp one, and docs/build.md documents the fix: find the directory under HIP_PATH containing oclc_abi_version_400.bc and prepend HIP_DEVICE_LIB_PATH=<that directory> to the cmake command.

VRAM climbs during a long session, or the process OOMs well below the budget above

This is the residual ROCm allocator problem. llama.cpp defines GGML_USE_VMM on HIP builds unless you pass -DGGML_HIP_NO_VMM=ON, and the startup banner prints VMM: yes or VMM: no per device — but a llama.cpp contributor states on issue #22107 that "virtual memory is completely broken in rocm since ROCM 7.0 with zero movement from AMD", tracking it at ROCm/rocm-systems#2516, which is still open. Without working VMM the runtime falls back to a pool that keeps buffers at their peak size, and the thread's report is of that pool growing until it faults.

Two things bound your exposure. The FlashAttention dequant temporaries — historically the worst offender, because they scale with context — are not pool allocations at b10153 and later; they are the 1.000 GiB already counted above. And -DGGML_HIP_NO_VMM=ON at build time is the supported switch if you want to take the pool out of the picture and compare. If you see growth, record it and file it: a real gfx1100 trace is worth more than this paragraph.

Out of memory at startup with the default cache type

You almost certainly left -ctk/-ctv at f16. At 262,144 tokens that is 44.000 GiB of cache — the arithmetic is in the table above, and it is the most common way to be surprised by this model. Quantize the cache, or drop to -c 65536, where f16 fits at 15.447 GiB accounted — and pays no dequant scratch at all, which is why it lands closer to the quantized options than the cache column alone suggests.

Mismatched -ctk / -ctv silently disables FlashAttention

See the flag notes under Running. A stock HIP build compiles only the four matched vector instances; an asymmetric cache needs -DGGML_CUDA_FA_ALL_QUANTS=ON. The symptom is not an error message — the dispatcher just returns BEST_FATTN_KERNEL_NONE and you lose the kernel you were counting on.

About a quarter of tool calls come back as plain text instead of executing

Known, open upstream, and not your configuration. The model sometimes emits <tool_call> followed by a space rather than a newline, and llama.cpp's chat parser matches the marker with the newline attached. The author of PR #26324 puts the rate at roughly 25% and the consequence plainly: "All such tool calls currently fail and are displayed verbatim to the user instead of being executed." The PR is open and unmerged as of this writing, so treat tool-call reliability as a known ceiling under llama.cpp and check the raw completion text when a call appears to vanish. The same two-line fix is proposed on the model's own discussion #17, whose author measured 24/29 parseable calls before and 29/29 after.

HTTP 400 — Failed to initialize samplers on a json_schema request

Any request with a response_format of json_schema fails at sampler init on the default jinja chat path, before a token is generated. Two users reproduced it independently on discussion #6, on different builds and quant tiers, and both found the same workaround: add --no-jinja to the llama-server command. Overriding the surface template with --chat-template chatml does not help, and the raw /completion endpoint with the same schema returns conformant JSON — which isolates the fault to the chat-completions grammar trigger rather than the model. The cost of --no-jinja is that you lose the model's own chat template, so apply the prompt format yourself.

ollama run nanbeige/nanbeige4.2:3b-Q4_K_M returns not found

The model card lists that command, but the registry entry it names does not exist, and neither do the library/ paths for any spelling of the name. Use llama-server directly.

A note on Vulkan and on Windows

llama.cpp also has a Vulkan backend that runs on this card, and Windows HIP builds exist. Everything above — the kernel tables, the matched-type restriction, the scratch arithmetic — was verified against the HIP backend on Linux and nothing here should be assumed to carry to either. docs/build.md notes separately that HSA_OVERRIDE_GFX_VERSION is not supported on Windows at all.

Anything else, or a real throughput measurement on this card, is welcome via the submission form.