What You'll Build
A llama-server endpoint running Nanbeige4.2-3B at 262,144 tokens of context, entirely in the RTX 4090's VRAM, on near-lossless Q8_0 weights with a q4_0 KV cache. The 4090 reaches this model's whole declared context for the same reason every 24 GB card does — capacity — but it is the one card in this recipe family that needs a different build target, and that is where most of this page's card-specific content lives.
Hardware data: RTX 4090 (24 GB VRAM) · Q8_0 weights 4.130 GiB + q4_0 KV cache at 262,144 tokens 12.375 GiB + reserved logits 0.317 GiB + FlashAttention dequant scratch 1.000 GiB = 17.822 GiB accounted · no benchmark submitted yet · See benchmark data
⚠️ The 3B's weights are not the constraint here — the KV cache is. Nanbeige4.2-3B is a Looped Transformer: the model card says its "Looped Transformer architecture reuses the transformer layers to increase model capacity without adding parameters."
config.jsonsetsnum_loops: 2, so its 22 blocks execute twice per forward pass over one shared set of weights — and each pass keeps its own keys and values. llama.cpp therefore allocates KV for 44 logical layers, and every context figure you would compute fromnum_hidden_layers: 22is exactly half the truth. Atf16the full 262,144-token cache is 44.000 GiB, nearly twice this card. Everything below exists to get around that.
Requirements
| Component | Minimum | This recipe |
|---|---|---|
| GPU | 24 GB VRAM, CUDA | — not measured; the budget below is derived from file bytes and llama.cpp's allocation rules (/contribute) |
| RAM | 8 GB system RAM | — |
| Storage | 4.43 GB for the Q8_0 GGUF (decimal, as HuggingFace lists it) | 4,434,787,488 bytes from the HF tree API; ~13 GB more if you also clone and build llama.cpp |
| Software | CUDA Toolkit 11.8+, llama.cpp b10153+, CMake | — |
This card is 89, not 86 — and that is the one thing a 24 GB sibling recipe cannot hand you
Every other number on this page is identical on any 24 GB CUDA card, because it is made of file bytes and allocation rules. The build target is not. NVIDIA's own CUDA GPUs table puts GeForce RTX 4090 in the compute-capability 8.9 row — alongside the L40S and the RTX 6000 Ada — while the RTX 3090 and RTX 3090 Ti sit in the 8.6 row. So a recipe written for a 24 GB Ampere card names the wrong architecture for this one.
Two consequences, both from NVIDIA's Ada Compatibility Guide:
- Your toolkit has to be new enough. With version 11.8 of the CUDA Toolkit,
nvcc"can generate cubin native to the NVIDIA Ada GPU architecture (compute capability 8.9)". The guide's preceding section says of 11.0 through 11.7 that theirnvcc"can generate cubins native to the Ampere architecture (compute capability 8.0 and 8.6)", and its worked-gencodeexamples for those versions stop atcompute_86— so on an older toolkitsm_89is not an option you can select. - An
sm_86build still runs — it is just not Ada-native. The guide states the rule for cubin portability plainly: "a cubin generated for compute capability 8.6 is supported to run on a GPU with compute capability 8.9". Minor-version compatibility runs forward, not backward. So if you already built llama.cpp for an Ampere card and moved the binary, it will load and run here; you have simply left the Ada-native code path on the table. That is worth knowing precisely because it means a wrong build target on this card produces no error message — nothing crashes, nothing warns, and the only signal is a build you did not intend.
NVIDIA's RTX 4090 spec page gives the card a Standard Memory Config of 24 GB GDDR6X on a 384-bit Memory Interface Width. It publishes no memory-bandwidth figure, and decode from a quantized KV cache is bandwidth-bound rather than core-clock-bound — so do not translate this card's generational lead in shader throughput into an expected token rate. Nobody has measured this model on this card; see Results.
Where the parameter counts come from
The card says "3B"; the repository is 8.34 GB in bf16, which is not what a 3B looks like. Both are correct, and the difference decides the budget, so here is the census rather than an assertion.
HuggingFace's tensor index for Nanbeige/Nanbeige4.2-3B reports 4,169,800,704 parameters, all BF16. That reconciles exactly with config.json:
- Untied embeddings (
tie_word_embeddings: false):166144 × 3072 × 2= 1,020,788,736 - Per block: attention
3072×6144 + 2×(3072×1024) + 6144×3072= 44,040,192, plus MLP3 × 3072 × 10752= 99,090,432, plus two RMS norms = 143,136,768 - 22 blocks plus the final norm = 3,149,011,968
- Total: 1,020,788,736 + 3,149,011,968 = 4,169,800,704 — the HF figure, to the parameter
So "3B" is the non-embedding count, which is how the model card's own comparison table labels it. The consequence that matters: the 22 blocks appear once in that census. If the two loop passes had their own weights the model would be roughly 7.3B. They do not — the loop reuses one stack, so you pay for the weights once and for the cache twice.
The KV cache doubles, and that is not a bug
1. The vendor designed it that way. A Nanbeige team member, replying on a thread titled "the modal is tiny but kv cache exploding!" — "We have also investigated KV-cache sharing across loop passes, but the performance gains were notably smaller than with the full looped setup." (discussion #10, leran1995, flagged as an org member), and again on discussion #18: "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2". Sharing was built, measured and rejected.
2. llama.cpp implements it that way. src/models/nanbeige.cpp expands the logical layer count before anything is allocated — hparams.n_layer_all = (uint32_t) ((size_t) n_layer_phys * (size_t) n_loops);, under a comment reading "Expand logical layer count before load_tensors() allocates layers / KV." — and aliases the layer structs across loops under a second comment, "Share physical weights across loops; each slot still has its own KV index." src/llama-kv-cache.cpp then sizes the cache with const uint32_t n_layer = hparams.n_layer_all;. For nanbeige with num_loops: 2 that is 44, not 22.
3. Your file has to carry the flag. The loader reads the key as optional with a default of 1 — uint32_t n_loops_u = 1; ml.get_key(LLM_KV_NUM_LOOPS, n_loops_u, false); — so a GGUF converted by a path that omits it loads happily and runs 22 layers. That is a silently different model, not an error. llama_kv_cache prints the layer count in its startup line; confirm 44 layers before trusting anything below.
llama_kv_cache: size = ... MiB (262144 cells, 44 layers, 4/1 seqs), K (q4_0): ..., V (q4_0): ...
If it says 22 layers, your cache is half-size and your context is wrong.
KV cost per token, and what 24 GB buys
config.json gives num_key_value_heads: 8 and head_dim: 128, so each logical layer stores 8 × 128 = 1024 elements for K and 1024 for V per token. Over 44 logical layers that is 90,112 elements per token. Per-element cost comes from the block layouts in ggml/src/ggml-common.h, where QK4_0 and QK8_0 are both 32 values per block:
| Cache type | Bytes per element | Bytes per token |
|---|---|---|
f16 (default) | 2 | 180,224 (176.0 KiB) |
q8_0 | 34/32 = 1.0625 | 95,744 (93.5 KiB) |
q4_0 | 18/32 = 0.5625 | 50,688 (49.5 KiB) |
Multiplying out:
| Context | f16 KV | q8_0 KV | q4_0 KV |
|---|---|---|---|
| 32,768 | 5.500 GiB | 2.922 GiB | 1.547 GiB |
| 65,536 | 11.000 GiB | 5.844 GiB | 3.094 GiB |
| 131,072 | 22.000 GiB | 11.688 GiB | 6.188 GiB |
| 262,144 | 44.000 GiB | 23.375 GiB | 12.375 GiB |
Only the q4_0 column reaches the declared context on a 24 GiB card.
Two more terms, before you add weights
A flat 0.317 GiB of reserved logits. llama.cpp sizes one worst-case graph at startup. src/llama-context.cpp sets the logit rows that graph must hold to n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max), where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is 512 rows regardless of -c, each a full f32 distribution over this model's 166,144-token vocabulary:
512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB
It is large only because the vocabulary is, and the dial that moves it is -ub, not -c — so it is the same 0.317 GiB in every row of the budget table.
4,096 bytes per context token of FlashAttention dequant scratch — but only on a quantized cache. The kernels read f16, so a quantized cache is expanded into scratch VRAM first. ggml/src/ggml-cuda/fattn.cu sets need_f16_K = true; need_f16_V = true; for the TILE and MMA_F16 kernels, and fattn-common.cuh then appends an f16 copy of each to the attention output tensor's own allocation:
if (need_f16_K && K->type != GGML_TYPE_F16) {
data.end = GGML_PAD(data.end, 128);
data.K = data.end;
data.end += ggml_nelements(K)*ggml_type_size(GGML_TYPE_F16);
}
with a matching block for V. Two properties decide the size. The guard's second conjunct means an f16 cache pays exactly zero — this is the one place where the default cache is the cheaper one. And the term is charged at the reserve, not only during prompt processing: llama.cpp sizes its worst-case graph at n_tokens = min(n_ctx, n_ubatch) = 512, and no vector-kernel branch in fattn.cu accepts a batch that wide. On Ada the quantized-cache path takes the direct-read vector kernel when Q->ne[1] <= 2 — a genuinely wider window than the == 1 an Ampere card gets, and the only place on this page where the generation actually changes a dispatch decision — but 512 fails both tests, so the dequantizing path is what gets budgeted regardless.
For this model the cost is (1024 + 1024) × 2 B = 4,096 bytes per token of -c: 0.250 GiB at 65,536, 0.500 GiB at 131,072 and 1.000 GiB at 262,144.
Add all four terms. Both weight tiers are from the bartowski GGUF repo, byte counts via the HuggingFace tree API:
| Configuration | Weights | KV | Logits | FA scratch | Accounted total | Slack on 24 GiB |
|---|---|---|---|---|---|---|
Q8_0 · q4_0 KV · 262,144 ctx (this recipe) | 4.130 | 12.375 | 0.317 | 1.000 | 17.822 GiB | 6.178 GiB |
Q8_0 · q8_0 KV · 131,072 ctx | 4.130 | 11.688 | 0.317 | 0.500 | 16.635 GiB | 7.365 GiB |
Q8_0 · f16 KV · 65,536 ctx | 4.130 | 11.000 | 0.317 | 0.000 | 15.447 GiB | 8.553 GiB |
Q4_K_M · f16 KV · 65,536 ctx | 2.500 | 11.000 | 0.317 | 0.000 | 13.817 GiB | 10.183 GiB |
The 6.178 GiB of slack is not spare. It absorbs the CUDA context, the rest of the compute buffer for a 44-layer unrolled forward pass, and the framebuffer if this card also drives your displays — none of which is measured on this GPU, which is why the recipe stops at the declared context rather than arguing for more.
Separately, budget 256 MiB of system RAM for the attention mask, which is deliberately absent from every figure above because it is not VRAM: src/llama-graph.cpp carries GGML_ASSERT(ggml_backend_buffer_is_host(self_kq_mask->buffer));, so it is a pinned host allocation. At this context it is n_kv × n_ubatch × 2 B = 262,144 × 512 × 2 = 268,435,456 B. Counting it as VRAM is a common way to over-budget this model.
Why Q8_0 weights and not Q4_K_M
On an 8 GB card the weight tier is the whole decision. On 24 GB it barely registers, and the arithmetic says so. Within the bartowski repo, Q8_0 (4,434,787,488 B) costs 1.631 GiB more than Q4_K_M (2,684,023,968 B). Spent on cache instead, that buys 31,957 more tokens at q4_0 or 17,535 at q8_0 — 12% and 7% of the 262,144 ceiling.
Both figures use the effective per-token price of context, not the sticker KV price: raising -c by one token on a quantized cache costs the cache bytes plus 4,096 bytes of dequant scratch, so the real rate is 54,784 B/token at q4_0 and 99,840 B/token at q8_0. On the sticker rates alone the same 1.631 GiB would read 34,540 and 18,285, and the gap between the two pairs is the dequant tax. The question the comparison asks is what the freed memory buys if spent on context, and raising the context drags the scratch along with it — so the sticker rate answers a question nobody is asking.
Trading a near-lossless weight tier for at most 13% more context, on a model whose entire pitch is agentic and reasoning accuracy at 3B, is the wrong way round. Quantizers differ and their bytes are not interchangeable: owao's Q8_0 is 4,434,787,168 B — 320 bytes apart, same to three decimals — but its Q4_K_M is a different file at 2,574,807,904 B. owao ships a 13-rung ladder and bartowski a 23-rung one; pick one repo and do not mix their numbers in one sum.
What about the vendor's FP8 build?
Ada has FP8 tensor cores and Nanbeige publishes an FP8 checkpoint, so the question comes up on this card and not on the Ampere ones. It is out of scope here for a mundane reason: this recipe's runtime is llama.cpp, whose weight format is GGUF, and the org's FP8 and GPTQ-Int8 releases target vLLM-class servers instead. Nothing below changes if you have Ada FP8 hardware — the GGUF path does not use it. If you want that route, it is a different recipe with a different runtime, not a flag on this one.
Installation
1. Build llama.cpp with CUDA for sm_89
Mainline support arrived in PR #25994, merged 2026-07-27 as commit b77d646751d01c0962bc203b6809e9d94f7d50b7, and release b10153 carries exactly that commit — so b10153 is the first tagged build that will load this architecture. Anything older refuses it.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
That relies on llama.cpp's native-architecture detection, which is correct when you are building on the machine that has the card in it. If nvcc cannot see the GPU — a container build, a build host without the card, a cross-compile — docs/build.md documents naming the capability yourself, and for this board that is 89:
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="89" -DCMAKE_BUILD_TYPE=Release
Use "86;89" if one binary has to serve both this card and an Ampere one.
The model card still tells you to build the vendor's own fork (git clone -b nanbeige42 https://github.com/Nanbeige/llama.cpp.git). That was correct at release and is not any more — mainline carries the architecture, and mainline is where the fixes land.
2. Download the weights
Nanbeige publishes no first-party GGUF. Enumerating the org returns ten repositories and not one of them carries the gguf tag; the quantized builds it does publish are FP8 and GPTQ-Int8. A team member said on discussion #1 that official quantized versions were being prepared, which as of this writing has not happened. Everything below uses a community conversion.
pip install -U huggingface_hub
hf download bartowski/Nanbeige_Nanbeige4.2-3B-GGUF \
--include "Nanbeige_Nanbeige4.2-3B-Q8_0.gguf" \
--local-dir ./nanbeige4.2-3b
Check provenance on whichever repo you pick, because this model has a date cliff. The canonical repo landed a tokenizer commit on 2026-07-27, hours before mainline support merged the same day, and a quant is a frozen copy of upstream as of the moment it was uploaded. bartowski's card states the quants were built with llama.cpp release b10159, and the HuggingFace API gives that repo createdAt 2026-07-28T14:38Z — comfortably after both events. owao's repo has an earlier createdAt (2026-07-21T20:33Z) but a lastModified of 2026-07-29T13:06Z, so check the individual file's commit before assuming a given rung there is post-merge.
Running
The long-context configuration
./build/bin/llama-server \
-m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
--host 127.0.0.1 --port 8080 \
-ngl 99 \
-c 262144 \
-fa on \
-ctk q4_0 -ctv q4_0 \
--temp 0.6 --top-p 0.95 --top-k 20
That is the full declared context — "The model supports a context length of up to 262,144 tokens (256K)." — resident in VRAM on one card, at 17.822 GiB accounted.
Four of those flags are load-bearing, and none of them is architecture-specific — this is the half of the recipe that reads identically on any CUDA card:
-c 262144— set it explicitly, and never-c 0. Omitting-cis not the same as passing0: with-cunset, llama.cpp's-fit ondefault (common/fit.cpp) interpolates the context down until the model fits free memory with a margin, so you silently get some arbitrary reduced number. With-c 0,common/arg.cppsetsparams.fit_params_min_ctx = UINT32_MAX;— a deliberate "give me the full trained context, do not shrink it" — and you get 262,144 cells with thef16default cache, i.e. 44.000 GiB, i.e. an immediate out-of-memory abort.-fa on. A quantized V cache requires FlashAttention;src/llama-context.cppthrowsquantized V cache was requested, but this requires Flash Attentionotherwise. Theautodefault will enable it for you, but being explicit makes the failure mode legible.-ctk q4_0 -ctv q4_0— matched types, not mixed. A stock CUDA build instantiates FlashAttention kernels only for identical K and V types.fattn.cuguards it literally —#ifndef GGML_CUDA_FA_ALL_QUANTS/if (K->type != V->type) {/return BEST_FATTN_KERNEL_NONE;— and the default instance list compiles exactly four vector pairs:f16-f16,q4_0-q4_0,q8_0-q8_0,bf16-bf16. A clever-looking-ctk q8_0 -ctv q4_0needs a rebuild with-DGGML_CUDA_FA_ALL_QUANTS=ON.- Leave
--parallelalone.llama-serverdefaults it to-1, andtools/server/server.cppresolves the sentinel withparams.n_parallel = 4;andparams.kv_unified = true;in the same branch. Unified means one shared cache of-ccells that any single slot may consume in full — so the table above is the whole KV bill, and one conversation can still reach 262,144. Passing-np 4explicitly skips that branch, leaveskv_unifiedfalse, andsrc/llama-context.cppthen computescparams.n_ctx_seq = cparams.n_ctx / cparams.n_seq_max;— same total memory, a quarter of the context per conversation. If you want one slot, say-np 1, not-np 4.
The quality-first alternative
If you would rather not run a 4-bit K cache, halve the context and keep the near-lossless one. 16.635 GiB accounted, and 131,072 is exactly the max-new-tokens the model card recommends for reasoning and chat:
./build/bin/llama-server \
-m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf \
--host 127.0.0.1 --port 8080 \
-ngl 99 \
-c 131072 \
-fa on \
-ctk q8_0 -ctv q8_0 \
--temp 0.6 --top-p 0.95 --top-k 20
For agentic and tool-use work the card's sampling table recommends --temp 1.0 with 65,536 new tokens instead.
Spend the headroom on preserve_thinking
The chat template takes preserve_thinking, which controls whether reasoning from earlier assistant turns stays in the context. It is a KV-versus-quality dial, and on this card you are on the right side of it — a Nanbeige team member on discussion #9: "In our evaluations, enabling preserve_thinking generally provides better performance and better kv-cache reuse, so we recommend keeping it enabled when context length and memory allow." Pass it through llama-server's OpenAI-compatible endpoint as "chat_template_kwargs": {"preserve_thinking": true}.
Results
- Speed: omitted — there is no measurement of this model on an RTX 4090.
/check/nanbeige4-2-3b/rtx-4090has zero benchmarks; all 28 discussion threads on the canonical repo were fetched individually and none mentions a 4090 or any 24 GB card; and llama.cpp's issue and PR search returns no Nanbeige report on this GPU. The only throughput datapoint anywhere for this model is a community report on an 8 GB RTX 5060, posted bythermi6on discussion #18 — a different capacity on a newer generation, which constrains this card in neither direction, so its figures are not repeated here and nothing is interpolated from them. If you run this, please contribute your measurement so the next reader gets a real figure. - VRAM usage: 17.822 GiB accounted at the lead configuration — 4.130 GiB weights, 12.375 GiB cache, 0.317 GiB reserved logits and 1.000 GiB of FlashAttention dequant scratch — leaving 6.178 GiB of the card's 24 GiB. The
q8_0/131,072 alternative comes to 16.635 GiB. Both derived, not measured: see /check/nanbeige4-2-3b/rtx-4090. Separately, budget ~256 MiB of system RAM for the pinned attention mask. - Quality notes: I found no evaluation of this model under a quantized KV cache at any tier in the spaces searched — the model card, all 28 canonical discussion threads, both GGUF repos' cards, and llama.cpp's issue and PR search. Treat that as a gap, not as evidence the loss is small.
q8_0cache is generally treated as near-lossless across the llama.cpp ecosystem;q4_0is the aggressive rung, and the reason the alternative configuration above exists. The vendor publishes no throughput figures at all — the card carries quality benchmarks and a sampling table and nothing about speed. - Context is trained, not extrapolated:
rope_scalingisnullinconfig.json, so 262,144 is a training target rather than a RoPE-scaling claim.
For the full benchmark data, see /check/nanbeige4-2-3b/rtx-4090.
Troubleshooting
llama_model_load: error loading model architecture: unknown model architecture: 'nanbeige'
Your build predates b10153. Rebuild from mainline master, or use a release tag at or above b10153 — that build's commit is the merge of PR #25994. On discussion #23 one user reports still being unable to load the architecture, and a different user then reports that mainline release b10199 loads it. Distribution channels lag: the model card itself notes that LM Studio's bundled llama-server does not support nanbeige, and tells you to copy your own build's binaries into the LM Studio backend directory.
nvcc warning : Cannot find valid GPU for '-arch=native', default arch is used
llama.cpp's default build asks nvcc to compile for whatever card it can see. In a container, on a build host without the GPU, or under a driver that is not exposing it, that detection fails and you get a binary compiled for some default architecture rather than for Ada. It will still run — see the cubin-portability note above — but it is not the build you asked for. Pass -DCMAKE_CUDA_ARCHITECTURES="89" explicitly, which docs/build.md documents for exactly this case.
Out of memory at startup with the default cache type
You almost certainly left -ctk/-ctv at f16. At 262,144 tokens that is 44.000 GiB of cache — the arithmetic is in the table above, and it is the single most common way to be surprised by this model. Quantize the cache, or drop to -c 65536, where f16 fits at 15.447 GiB accounted and pays no dequant scratch at all, which is why it lands closer to the quantized options than the cache column alone suggests.
If you need the full context on a card that genuinely cannot hold the cache, --no-kv-offload moves the cache to system RAM and keeps the weights on the GPU. On a 24 GB card you should not need the flag.
Tool calls come back as plain text instead of executing
Fixed upstream, in release tag b10227 — check your build before you debug anything else. The defect was in llama.cpp rather than in your configuration: the model sometimes emits <tool_call> followed by a space rather than a newline, and the chat parser matched the marker with the newline attached, so the call was returned verbatim as content. The author of PR #26324 put the pre-fix rate at roughly 25%, and the same defect was reported independently on the model's own discussion #17, whose author measured 24 of 29 calls parsing before his patch and 29 of 29 after.
The route to the fix is worth a sentence, because #26324 is still the first thing a search turns up and it never merged. A llama.cpp maintainer pushed back on the proposed whitespace trimming — "trimming whitespaces for some of the Qwen models was really detrimental to their output" — and another contributor pointed instead at the grammar trigger, and at PR #26252. That one, a specialized parser, merged on 2026-08-02 and went out as b10227; on 2026-08-10 #26324's author closed his own pull request unmerged, reporting that tool calls now worked every time. Tool calling therefore has a build floor of its own — b10227, above the b10153 this page requires to load the model at all. If calls still come back as text your build sits between the two; rebuild.
HTTP 400 — Failed to initialize samplers on a json_schema request
Any request with a response_format of json_schema fails at sampler init on the default jinja chat path, before a token is generated. Two users reproduced it independently on discussion #6, on different builds and different quant tiers, and both found the same workaround: add --no-jinja to the llama-server command.
./build/bin/llama-server -m ./nanbeige4.2-3b/Nanbeige_Nanbeige4.2-3B-Q8_0.gguf --no-jinja ...
Overriding the surface template with --chat-template chatml does not help, and the raw /completion endpoint with the same schema returns conformant JSON — which isolates the fault to the chat-completions grammar trigger rather than the model or the grammar. The cost of --no-jinja is that you lose the model's own chat template, so apply the prompt format yourself if you take this route.
ollama run nanbeige/nanbeige4.2:3b-Q4_K_M returns not found
The model card lists that command under its Ollama section, but the registry entry it names does not exist, and neither do the library/ paths for either spelling of the name. The card's other Ollama route — building Ollama from the vendor's fork and copying a llama.cpp build into its runtime payload directory — is a lot of machinery to reach a llama-server you already have. Use llama-server directly.
Mismatched -ctk / -ctv silently disables FlashAttention
See the flag notes under Running. A stock CUDA build compiles only the four matched vector instances; an asymmetric cache needs -DGGML_CUDA_FA_ALL_QUANTS=ON. The symptom is not an error message — the dispatcher returns BEST_FATTN_KERNEL_NONE and you lose the kernel you were counting on.
Anything else, or a real throughput measurement on this card, is welcome via the submission form.