Apodex 1.1 mini on RTX 5060 Ti: a 36B agent at 128K, and what GDDR7 buys on a 128-bit bus

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one RTX 5060 Ti 16GB, at a 131,072-token working context, from a Q4_K_M GGUF that is larger than the card.

The weights load 19.908 GiB. The card holds 16. It runs because the model is a sparse Mixture of Experts and llama.cpp will leave the routed experts of the first N blocks in system RAM: 123 tensors carry 18.586 GiB of the file, and across the forty blocks a normal run loads that is 19,503,513,600 bytes — 18.164 GiB of the 21,376,133,632 (19.908 GiB) resident, or 91.24%.

Hardware data: RTX 5060 Ti 16GB (16 GB GDDR7, 128-bit, Blackwell / sm_120) · 13.546 GiB derived working set at -ncmoe 17, 131,072-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5060 Ti running Apodex. /check/apodex-1-1-mini/rtx-5060-ti returns unknown with zero benchmarks. Every figure below is arithmetic over this GGUF's own tensor table and over llama.cpp's allocation code at a pinned release, written out so you can check it. There is no throughput measurement of this model on this page — Results names the space searched. What does exist, and is closer than anything else in circulation, is a llama-bench run of a different model of this same GGUF architecture on this exact card; it is dissected under Does the arithmetic survive contact with a real card?. If you run Apodex here, please send the numbers.

🧭 The conclusion that is specific to this card, before the arithmetic that gets there. This board pairs GDDR7 with a 128-bit bus: newer memory, half the width of its 256-bit tier-mates. At -ncmoe 17 a decode step reads 0.2601 GB from system RAM and 2.00–3.43 GB from VRAM, and the two halves cost the same wall-clock only when the card's bandwidth is 7.70× to 13.18× your DIMMs'. That crossover is a property of this recipe, not of this card — any 16 GB board running these settings faces the same three numbers. What the card decides is where it lands against them. At the 448 GB/s this catalogue publishes for this board, the answer splits: host-bound on single-channel memory at every depth, VRAM-bound on dual-channel DDR4-3200 or faster, and a dead heat on dual-channel DDR4-2133 at a full window. So this page cannot tell you "buy faster RAM" or "buy a faster card" — it tells you which one you are currently limited by, and the answer depends on your DIMMs. Where the decode time actually goes prices it out. Everything about capacity on this page is what any 16 GB card gets; everything about speed is not.

ℹ️ This recipe is text-only, deliberately. The checkpoint has a vision tower and the lead GGUF repo ships two projectors for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-31). The smaller projector is also 0.838 GiB, which on this budget is just under the 0.844 GiB that the two cheapest steps of -ncmoe would free spent on a capability the recipe does not use, so the run command refuses it explicitly rather than merely omitting the flag.

⚠️ Three version floors, and only the first is about whether the model loads.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at b7989 and present at b7990 (line 42 at the pin below), so it arrived in the interval (b7989, b7990]. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that keeps blk.40 off the card arrived in (b10211, b10212]. Both are intervals bracketed by adjacent published tags, which is as fine as a tag-granularity probe resolves.
• The CUDA toolkit, and this one is specific to Blackwell. 12.8 or newer. Below that, llama.cpp's non-native build emits no sm_120 code at all — see Build llama.cpp with CUDA, where the gate is quoted line by line.
• A throughput floor. llama.cpp issue #25162 — "Performance regression on Turing GPUs after 9e58d4d69 — Qwen35 SSM kernels affected", closed as completed 2026-08-03 — records a 24–42% loss in ssm-scan.cu / ssm-conv.cu, the kernels behind thirty of this model's forty layers, with Gemma4 unaffected because it has no SSM layers. Use b10630 or newer — the release the lead quant was produced with — and understand that an older build can cost you silently rather than loudly. ⚠️ That regression was measured on Turing (SM75), and nothing about Blackwell follows from it — including from the guard people reach for. The branch fact, stated and then left alone: ggml/src/ggml-cuda/common.cuh:1650-1654 defines GGML_CUDA_RESTRICT as empty when defined(GGML_CUDA_USE_PDL) && __CUDA_ARCH__ >= GGML_CUDA_CC_HOPPER and as __restrict__ otherwise, with GGML_CUDA_CC_HOPPER 900 at :56 and GGML_CUDA_CC_BLACKWELL 1200 at :59. So this card compiles the empty branch, the opposite side from both Ada (890) and Ampere (860). Draw nothing from that. The guard cannot be #25162's mechanism: fetched at 9e58d4d69 (the blamed commit) and at 0b14b87d7 (the commit the reporter retested clean on), that block is byte-identical — 279 bytes, same digest at both — so it did not change across the regression. The real mechanism is __restrict__ moving from parameters to local variables, and no fix for the Turing loss was ever identified. Use b10630+ because the issue closed on a clean retest there, not because of a story about compute capabilities.

Every source line quoted on this page was read at release b10666, whose tag dereferences to 4e97ac86ebe2c4cb8212d98d2641ad6768810896, and line numbers are given for that tree only. The tag moves several times a day — b10701 was published 2026-08-30T18:56:31Z, three tags inside one hour — so this page pins rather than claiming currency. Note also that GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, because every bNNNN tag is marked prerelease; use /releases?per_page=N if you script a version check.

Requirements

Check which card you have before anything else. NVIDIA's own RTX 5060 family page gives the RTX 5060 Ti's Standard Memory Config as "16 GB / 8 GB GDDR7" — one name, two capacities, and this recipe needs the larger one.

nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.max,pcie.link.width.max --format=csv

What the vendor publishes about this board, and what it does not. Read off that page on 2026-08-31: 4608 CUDA cores, 2.57 GHz boost / 2.41 GHz base, 16 GB / 8 GB GDDR7, Memory Interface Width 128-bit, NVIDIA Architecture Blackwell, and a Technology Support row reading "PCI Express Gen 5: Yes". What it does not publish, counted on that page: no memory-bandwidth figure (the string Memory Bandwidth occurs zero times) and no PCIe lane count (lane, lanes, x8 and x16 all occur zero times). Two numbers this page needs are therefore not vendor facts, and they are treated differently because only one of them can be settled:

• 448 GB/s of local bandwidth. Not from NVIDIA. It is what this catalogue already publishes for this board — our Qwen3.8-27B page for this card sources it to Hardware Corner's RTX 5060 Ti 16GB page, which states "Bandwidth 448 GB/s" against "Memory bus 128 bit" (re-read 2026-08-31). ⚠️ Do not check it with the clock × 2 × bus-bytes shortcut that works on GDDR6 boards. GDDR7 signals in PAM3, which carries 1.5 bits per cycle rather than PAM4's 2, so the doubling rule does not describe this memory. The arithmetic that reproduces the published figure is per-pin data rate × bus bytes: 28 Gbps × 16 B = 448 GB/s. And note that the same Hardware Corner page prints "Memory speed 20 Gbps" three paragraphs from its own 448 — 20 × 16 = 320, not 448, so that page is internally inconsistent and only one of the two numbers can be right. This page uses 448 because it is the figure the rest of this catalogue is built on and the one the vendor's 128-bit width supports at the 28 Gbps GDDR7 grade; if you want a number you own, measure it rather than deriving it.
• A PCIe lane count. This page asserts none. NVIDIA publishes the generation and not the width, and the tech-press databases that carry the width are not readable by automation: techpowerup.com's page for this card answered HTTP 200 on 2026-08-31 with a body containing "Automated bot check in progress" and a <title> naming a completely different product (AMD Radeon RX 6750 GRE 12 GB), which is a 200 that carries neither the content asked for nor an honest error. So the width goes unstated here. What this page does instead is show you where a narrow link would bite, give you the one command that settles it on your machine, and price both widths — and the useful part of that arithmetic is that the generation is a vendor fact and it is the half that changed from the previous 16 GB generation. See Prefill is the phase the link pays for.

For your DIMMs, do the arithmetic from their rated transfer rate: dual-channel DDR4-3200 is 3200 MT/s × 8 B × 2 = 51.2 GB/s, dual-channel DDR4-2133 is 34.1 GB/s, dual-channel DDR5-6000 is 96 GB/s, and a single-channel configuration halves whichever you have. Both sides of that comparison matter on this page, and on this card neither one dominates the other outright.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered as LLM_ARCH_QWEN35MOE at src/llama-arch.cpp:42. That part needs no flags. The toolkit version does.

nvcc --version          # must report 12.8 or newer
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Why 12.8 is a floor and not a preference, quoted from the build file itself. ggml/src/ggml-cuda/CMakeLists.txt annotates its own architecture list at :19 — # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores — and then acts on it. On the non-native path the list is built in rungs: :34 appends 75-virtual 80-virtual 86-real unconditionally, :36-37 add 89-real 90-virtual at toolkit ≥ 11.8, and 120a-real is appended only inside if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") at :40 and :51. There is no Blackwell virtual architecture anywhere in that list — :41-42 explain why, that 120f-virtual "would in principle work" but "the newly added 'f' suffix conflicted with a preexising regex for validating CUDA architectures in CMake". So on a toolkit below 12.8 the highest entry your card can use is 90-virtual, and it runs JIT-compiled PTX from a Hopper virtual architecture rather than native sm_120 code, with the FP4 tensor cores that :77 calls "not forwards compatible" unreachable. Nothing warns you.

Which builds escape that gate, and which do not. GGML_NATIVE defaults ON — ggml/CMakeLists.txt:105-110 sets GGML_NATIVE_DEFAULT to OFF only if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}) and to ON otherwise, and :123 takes that as the option's default. With it on, :27-28 of the CUDA file shortcut the whole ladder to CMAKE_CUDA_ARCHITECTURES "native". So an ordinary local build like the one above skips the rungs — but native cannot emit sm_120 from a toolkit that has never heard of sm_120, so 12.8 still binds. Cross-compiles, and container recipes that set SOURCE_DATE_EPOCH for reproducible builds, get GGML_NATIVE=OFF by default and take the ladder, where the gate is explicit. -DGGML_NATIVE=OFF does the same deliberately.

And the native path only works at all because of a workaround, which is worth knowing exists. :75-92 of the same file rewrite every plain 12X architecture to 12Xa, with :94 stating the reason outright: "If we try to compile a 'native' build it will use the 12X architectures and fail." :76-77 give the mechanism — "12X is forwards-compatible, 12Xa is not. Notably the Blackwell FP4 tensor core instructions are not forwards compatible and therefore need 12Xa." If your build prints Replacing 120 in CMAKE_CUDA_ARCHITECTURES_NATIVE with 120a (:85), that is the workaround firing correctly, not a warning.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes (20.363 GiB), imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z; the file list, the byte count and the download URL were re-fetched and HEAD-checked on 2026-08-31 (a 302 to a CDN redirect answering 200 with content-length: 21864082336) rather than copied from a sibling page.

Disk size and VRAM residency are different numbers here, and the gap has two parts. The file is 21,864,082,336 bytes (20.363 GiB) on disk; a normal run loads 21,376,133,632 (19.908 GiB). The difference is 487,948,704 bytes, and only most of it is the model. 476,956,672 of it is blk.40, the multi-token-prediction head, which is created with TENSOR_SKIP unless you ask for it (src/models/qwen35moe.cpp). The remaining 10,992,032 bytes (10.5 MiB) is the file's non-tensor content — the GGUF header, the 52 key-value metadata entries, the tensor-info table for 753 tensors and the alignment padding — and it is exactly 21,864,082,336 − 21,853,090,304, the file size minus the sum of its tensor payloads, which is by construction everything that is not payload. So the MTP block alone does not account for the gap. Budget 22 GB of disk and 19.908 GiB of weights, and do not substitute one for the other.

Do not start from the publisher's quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card itself says "llama.cpp downloads the mmproj automatically when using" -hf. It does: find_best_mmproj at :632 matches any sibling filename containing mmproj, this repo ships two (899,283,488 and 902,822,432 bytes), and no_mmproj defaults to false (common/common.h:596) while mmproj_use_gpu defaults to true (:594). So the quickstart puts 0.838 GiB of vision projector on a card this recipe has already budgeted to within about a third of a gigabyte. Download the file explicitly, as above, and load it with -m.

Q4_K_M on a 16 GB card is a choice, and the alternative is narrower than it looks. Twelve of bartowski's rungs are under 16 GiB on disk, so "small enough to load" is not the constraint — what has to sit beside the weights is. Hold this page's own ceiling (derived below: 13.917 GiB for weights plus KV plus recurrent state) and a 32,768-token q8_0 cache, and a fully resident configuration needs weights at or below 13.524 GiB. The whole 3-bit tier misses that: the smallest, IQ3_XXS, is 14.287 GiB and Q3_K_S is 14.886. The escape from offload is not "drop to 3-bit", it is "drop to 2-bit" — Q2_K_L at 12.650 GiB, Q2_K at 12.187, or IQ2_M at 11.682. That is exactly where a second publisher draws the line: abenzerps's hardware table gives the 16 GB row as "IQ2_M full offload; IQ3_M may require hybrid offload" and then closes the section with "Prefer Q4_K_M or higher when system memory permits." This recipe follows that closing sentence, because the model is an agent that runs tool calls and multi-step plans, and a malformed argument is a failed task rather than a clumsy sentence. Capacity-driven — this paragraph is the same on any 16 GB card.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 17 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

Nine of those flags are decisions rather than defaults. Each default below was read at b10666 for this page rather than carried across from another card, and each decision is labelled capacity-driven — meaning it is the same on any 16 GB card — or bandwidth-driven, meaning it is not.

• --no-mmproj refuses the vision projector rather than merely not asking for it: 0.838 GiB, for the reason under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point; -ncmoe then walks part of it back off. Capacity-driven — it is the baseline the offload is measured back from, and it is also what keeps the KV cache and the recurrent state on the card.
• -ncmoe 17 keeps the routed experts of blocks 0–16 in system RAM. The option is {"-ncmoe", "--n-cpu-moe"}, "N", at common/arg.cpp:2789 and its help string is the next line, :2790, reading "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" — the implementation matches the wording exactly. 17 is derived in Choosing your own -ncmoe as the lowest rung that clears this page's ceiling at 131,072 tokens — it is not a measured optimum, and it is not inherited. Capacity-driven.
• -np 1. llama-server sets params.n_parallel = -1; // auto by default (common/arg.cpp:1400) and resolves that to several slots sharing one KV pool. The Gated-DeltaNet recurrent state is allocated per row of that pool, so more slots multiply it and split your context. One slot is what a single-user agent loop wants, and pinning it removes a resolution step this page has not traced at this pin. Capacity-driven.
• -c 131072 is half the model's declared context_length of 262144. The ladder below prices the other half; it costs three more steps of -ncmoe. Both, and it is this page's largest speed lever — capacity-driven for the 1.328 GiB the cache occupies, and bandwidth-driven because that cache is re-read from VRAM on every generated token. Context depth moves the crossover from 7.70× on an empty window to 13.18× on a full one, which on this card is what decides which side you are limited by.
• -ctk q8_0 -ctv q8_0 quantises the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 131,072-token cache from 2,684,354,560 bytes (2.500 GiB) at f16 to 1,426,063,360 (1.328 GiB), which on this card is a saving of 1.172 GiB — worth between two and three steps of -ncmoe you do not have to spend, since a step costs 0.4219 or 0.4863 GiB so two come to 0.844–0.973 and three to 1.266–1.459. Capacity-driven — and, on this card, it buys speed too: the KV cache is read out of VRAM on every decode step, so removing those 1,258,291,200 bytes takes about 2.81 ms per token off the decode budget at a full window. Note it is not quite a halving — 8.5 bits against 16 — and the absolute cost of the cache that remains is 3.18 ms, which is a different figure used elsewhere on this page.
• -fa on makes Flash Attention explicit. The default at this pin is auto — flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO (common/common.h:499) — and auto would turn it on anyway here, because a quantised V cache requires it (src/llama-context.cpp:3675-3678). Pinning it means a future change to what auto decides cannot silently change your memory profile. ⚠️ There are two open reports in this area and this page prices the alternative rather than merely preferring this branch — one against exactly this flag combination on this architecture, The crash this recipe's flags are shaped like, and one specific to Blackwell that leaves the flag alone and changes an environment variable instead, The GPU stops responding after minutes of agent work. Capacity-driven.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp:1194). Taking that advice makes the 7.752 GiB of host-side experts ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure and not a cache you can squeeze. Bandwidth-driven, on the host side: the point of the pinned host buffer is the rate the CPU reads those experts at during decode.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632 — and it costs more on this architecture than on a plain transformer. See System RAM under Troubleshooting. Capacity-driven, but on host RAM rather than on VRAM — it does not appear anywhere in the card's budget.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05), and the GGUF header carries two of them itself as general.sampling.temp = 1.0 and general.sampling.top_p = 0.95. The card's text notes that Apodex "follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF (tokenizer.ggml.pre = qwen35). Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It appends one buffer-type override per block index 0 … N-1, each built from a regex at common/common.h:1130:

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";   // :1130
inline void llm_add_n_cpu_ffn_overrides(int n, const char * ffn_regex, ...)          // :1142

The loader matches that against tensor names and, on a hit, gives the tensor a CPU buffer type at tensor-creation time, before any buffer exists — so those bytes are allocated in host memory and never enter VRAM at all. Nothing in the path consults the architecture, which is why it works identically on this hybrid recurrent model and on a plain transformer MoE. 123 tensors in this file match, three per block across all 41 blocks, totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads.

Three things it does not move, all of which stay on the card and all of which show up in the decode budget later:

• The attention and Gated-DeltaNet weights. A block's attn_* tensors and its ssm_* tensors do not match _exps.
• The shared expert and the router. ffn_*_shexp and ffn_gate_inp do not match either, so the dense path every token takes stays resident, and routing is decided on the GPU.
• The KV cache and the recurrent state. Both take their buffer type from the layer's device, which -ngl alone decides; -ncmoe rewrites weight tensors and never touches the layer assignment. A block whose experts are in RAM still keeps its recurrent state in VRAM — which is what makes the budget below decomposable at all.

Add it up and offloading everything the regex can reach, -ncmoe 40, still leaves 1.744 GiB resident: 0.655 GiB of token_embd and output, 8,192 bytes of output_norm, and 1.089 GiB of everything else across forty blocks.

Does that actually happen on this architecture? The source says so, and there is direct evidence that a qwen35moe model runs on this exact card with more weights than the card holds — the llama-bench rows in llama.cpp issue #26674, dissected below. ⚠️ What has not been established is that llama.cpp honours the flag on this architecture in every backend. llama.cpp issue #27698 (open, 0 comments, opened 2026-08-25) has two numbered branches and they behave differently: its gemma4moe branch records a run with --fit off --n-gpu-layers 46 --n-cpu-moe 6 where "the weights load (expert offload is honored)" and the failure comes later, while its qwen35moe branch reports the loader ignoring --n-gpu-layers / --n-cpu-moe and segfaulting on a full-model allocation. That is an RX 6700 XT (gfx1031, RDNA2) on ROCm, so nothing about it transfers to a CUDA card — it is named here because it is the only report anywhere that touches this flag on this architecture, and because a reader who finds it should know which branch is which.

The VRAM budget

Weights, KV cache and recurrent state at the recipe's settings. Every figure is derived from the GGUF's own tensor table and from llama.cpp's allocation code at b10666.

The three rows come from:

• Weights. The file's 753 tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 17 moves 8,323,596,288 of that (7.752 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks load; full_attention_interval is 4 — the key is qwen35moe.full_attention_interval, with no .attention. segment — which makes ten of them full-attention, blocks 3, 7, 11, 15, 19, 23, 27, 31, 35 and 39. Each carries head_count_kv 2 at key_length and value_length 256, so 1,024 cache elements per token per layer and 10,240 across the ten. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows with context.
• Recurrent state. Those thirty blocks each allocate n_embd_r() = (ssm_d_conv − 1) × (ssm_d_inner + 2 × ssm_n_group × ssm_d_state) = 3 × (4096 + 2×16×128) = 24,576 elements and n_embd_s() = ssm_d_state × ssm_d_inner = 128 × 4096 = 524,288 elements, both F32 (src/llama-hparams.cpp:183,211). That is 2,195,456 bytes per layer per row and 65,863,680 across thirty. llama-server prints it at startup as RS buffer size.

A note on head_count, because getting it wrong doubles a whole table further down. It is 16, and there are two independent routes to it. The GGUF header carries qwen35moe.attention.head_count = 16 directly — it is one of 52 keys in this file's metadata, and it is easy to miss because it is a strict prefix of qwen35moe.attention.head_count_kv, so a substring probe finds the longer key and reports the shorter one absent. The check that separates them is the u64 length prefix GGUF writes before every key: 30 bytes for the exact key, 33 for the _kv one. The independent route is blk.3.attn_output.weight, whose dims are [4096, 2048], and 4096 ÷ key_length 256 = 16. ❌ Do not use attn_q.weight: it is [2048, 8192] and 8192 ÷ 256 = 32, because the q projection carries a concatenated gate. That route doubles every cell of the non-FA graph table below.

The two terms that table cannot produce, and what this page reserves for them

13.546 GiB is not a peak. Two things sit outside it, and on a card this tight they decide whether a configuration loads. Rather than leave them as a warning, this page reserves for them explicitly.

1. A 16 GB card does not give you 16 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. The only expert-offload run in our catalogue with the numbers written down is gemma4-26b on an RTX 3060 at -ncmoe 12, whose submitter recorded a peak of 11,179 MiB of that card's 12,288 on a headless box — so at most 1,109 MiB was unavailable to the model there. ⚠️ That is a different board in a different VRAM tier, and the term is not proportional to capacity (it is a context plus a driver reservation, not a percentage), so borrowing the absolute 1,109 MiB onto a 16 GB card is a transfer of an absolute quantity and not a measurement of this one. This page reserves 1,109 MiB. Two independent statements about this card point the same way and neither is a measurement to substitute for it: on the #26674 thread — an RTX 5060 Ti 16GB on Windows — a commenter writes that "the OS will use about 0.5-1 GB VRAM for itself (especially Windows)" and that anything else using hardware acceleration makes it larger; and the publisher vcruz305, smoke-testing their own Q4_K_M of this model on a Windows box, ran it at -ngl 80 rather than -ngl 99 and recorded the reason as "too tight with WDDM/display headroom". If your card is driving a monitor, expect to want more than 1,109 MiB, and start a step deeper than the table says.

2. The compute buffers. llama.cpp allocates graph working memory on top of everything above, and under expert offload it also stages host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 0.4219 or 0.4863 GiB, so that staging term alone is of that order. This page reserves 1024 MiB, which is llama.cpp's own per-device auto-fit margin: fit_params_target is initialised to 1024 * 1024*1024 at common/common.h:481. It is the runtime's opinion of how much room it wants, not a figure this page invented. For scale on leaving nothing: #27698's gemma4moe branch records a clean weight load followed by context creation failing on a ~504 MiB compute buffer with cudaMalloc failed: out of memory — a different card, a different backend and a different MoE architecture, quoted only for the magnitude of a term every table in this family excludes and for the shape of the failure it produces.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 16,384 − 1,109 − 1,024 MiB = 14,251 MiB = 13.917 GiB.

At -ncmoe 17 and 131,072 tokens the sub-total is 13.546 GiB, which clears that by 0.371 GiB. Read your own numbers off the server's startup log rather than trusting mine: llama.cpp prints model buffer size once per buffer, so under -ncmoe you get a CUDA line and a host line and the split above is directly checkable, followed by KV buffer size, RS buffer size and compute buffer size. If you do, those figures are the most useful thing you could send us — more useful than a speed number, because they are the two terms nobody in this family of recipes has been able to derive.

Choosing your own -ncmoe

Every row is weights + KV + recurrent state, q8_0 KV, one slot. Free of 16 GiB is what the naive arithmetic says; Slack is what is left once the driver share and the compute reserve are taken out, and the two columns disagree by more than 2 GiB, which is the whole point of the previous section.

Two rows are worth reading twice. -ncmoe 16 at 131,072 tokens has 1.968 GiB "free" and does not load — that is what the ceiling is for. And -ncmoe 17 is the recipe's setting because it is the lowest rung that clears the ceiling at this context, which is a derivation and not a preference: 16 misses by 0.115 GiB. Going deeper is always safe and always slower; the bottom row is the model's entire declared 262,144-token window, which this card can hold at -ncmoe 20.

Capacity-driven, and it transfers: this table is a property of the file and the ceiling, so any 16 GB card at these settings gets the same rows.

Then stop guessing and measure it. llama.cpp ships a tool that answers this on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 131072

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 131072 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device split of total / free / model / context / compute / unaccounted. That compute column is what this page reserved 1024 MiB for, and unaccounted is your driver's share.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys X GB" is wrong here. bartowski's recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28, 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates. Read the cumulative column above rather than multiplying. Nine of the seventeen blocks this recipe offloads are the expensive kind, which is why -ncmoe 17 moves 7.752 GiB rather than seventeen equal shares of anything.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 17.955 GiB on disk and its ladder is linear. If you would rather spend quality than offload depth, that is the trade — but derive its table from its own tensor list; do not scale this one.

Where the decode time actually goes

This is the section that is not the same on another 16 GB card, and it is the reason the capacity numbers above are interchangeable across the tier while the performance advice is not.

The mechanism first. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights" (ggml/src/ggml-backend.cpp:942) — unless the scheduler finds "a backend with higher prio wants to offload the op" (:960). CUDA answers that with a single comparison:

// ggml/src/ggml-cuda/ggml-cuda.cu:5341-5344 at b10666
static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
    ...
    return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size;
}
// :5515 — const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;

Generating a token is a batch of one, and one is less than 32, so at decode the offloaded expert matmuls stay on the CPU and those weights never cross PCIe at all. Prompt processing is a batch of hundreds, so at prefill those same tensors are copied host→device. The two phases are bound by different things and this page treats them separately.

The decode roofline, in bytes

expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token. Everything else on the card is read whole. At -ncmoe 17:

Three things about that table are load-bearing.

The KV cache is decode traffic, not just decode capacity. Every generated token reads the whole occupied cache back out of VRAM. At a full 131,072-token window that is 1,426,063,360 bytes — the single largest term in the table, larger than every weight the card holds put together. It is easy to price the KV cache as a capacity line and then leave it out of the bandwidth arithmetic — this page's own budget table above treats it as capacity — and leaving it out changes the answer rather than merely rounding it.

The output.weight head is separated from the embedding table. They are usually lumped as "0.655 GiB of embeddings", but only one of them is read per token: output.weight is 417,177,600 bytes of Q6_K read in full to produce logits over 248,320 tokens, while token_embd.weight is read one row at a time — 2,048 Q4_K elements, 1,152 bytes. Splitting them takes the non-expert VRAM read from a loose upper bound of 1.744 GiB down to a near-exact 1.4776 GiB.

The recurrent state is counted once, and that is the one term here that is a floor rather than an estimate. Thirty Gated-DeltaNet layers read a state and write it back; whether the traffic is 65,863,680 bytes or twice that depends on whether the kernel updates in place, which this page has not established. Counting it once is the conservative direction for the card's side of the comparison below: doubling it moves the crossovers from 7.70×/9.07×/13.18× to 7.95×/9.61×/13.43×, i.e. it makes this card slightly more likely to be host-bound, not less. Every crossover on this page counts it once.

The crossover, and where this card sits

The two halves of a decode step take equal wall-clock exactly when your card's bandwidth is VRAM bytes ÷ host bytes times your DIMMs'. That ratio needs no bandwidth figure at all, which is why it is the form this page states — and it is worth being explicit that this table is a property of the recipe, not of the card: any 16 GB board at -ncmoe 17 faces these same three numbers.

Now put your two bandwidths against it, at the 448 GB/s this catalogue publishes for this board:

That table is why this page will not give you a one-line answer, and the honest reading is that this card sits on the fence. On single-channel memory your DIMMs are the limit at every depth and a memory upgrade is the whole fix. On dual-channel DDR5 the card is the limit at every depth and faster DIMMs buy you comparatively little. On dual-channel DDR4 it depends on how full your context is: the card takes over between an empty window and 32,768 tokens on DDR4-3200, and on DDR4-2133 the two halves are within half a percent of each other at this recipe's window, which is a tie and should be read as one rather than as a verdict.

Priced out as a roofline — the two halves treated as serial, which they broadly are, since within a layer the CPU's expert matmul consumes what the GPU's router produced:

⚠️ Those are ceilings from two bandwidth terms, not predictions. They exclude the CPU time to actually multiply the experts, kernel launch overhead, attention arithmetic, sampling and everything else, and they credit both memories with perfect efficiency. Read them as a ceiling you cannot beat, never as a rate you will reach — and see the next section for how far below a ceiling of this shape a real run of this architecture on this card has actually landed. On an empty context the same arithmetic gives 68 / 83 / 105 / 139 tok/s down the same four rows, with the card's share falling to 31–62%.

What GDDR7 on a 128-bit bus actually buys here. Run the identical byte table against 288 GB/s — the figure this catalogue publishes for the previous generation's 16 GB, 128-bit board — and the same four rows give 45 / 51 / 59 / 68 tok/s at a full window. So this card's 1.556× bandwidth advantage turns into 1.24× on single-channel DDR4-3200 and 1.42× on dual-channel DDR5-6000. The advantage is real, it is smaller than the spec sheets imply, and it is largest where you would least expect it — on the fastest DIMMs, because that is where the card's half is the biggest share of a smaller total. The corollary is the practical one: on this recipe, a faster card and faster memory are complements, not alternatives. Upgrading only one of them runs into the other.

The offload depth is still a lever, and it is priced per step. Moving one block from card to host frees its whole expert triple from VRAM — 0.4219 or 0.4863 GiB — and adds one thirty-second of it to every token's host read: 13.5 MiB for a cheap block, 15.6 MiB for an expensive one. On dual-channel DDR4-3200 that is 0.28–0.32 ms per token added to the host side, against about 0.03 ms saved on the card side, so each step of -ncmoe costs roughly a quarter of a millisecond per token. Going from this recipe's 17 to 20 for the full 262,144-token window costs about 0.8 ms per token in offload — and adds 1.328 GiB of KV traffic, about 3.18 ms on this card, which is four times more. At this recipe's depth, context is a far more expensive throughput decision than offload depth is.

Prefill is the phase the link pays for

At batch sizes of 32 or more the scheduler hands the expert matmuls to the GPU, which means copying host-resident expert tensors across PCIe. llama.cpp issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0", "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26" — and it finds "prefill is bottlenecked by PCIe transfers, not GPU compute", with the GPU idle around 42% of each pass on expert H2D copies that run serially with the matmuls consuming them. ⚠️ Carry its caveats: a different model, one box, gen-4-specific, and the author dates the measurement to "around the b6xxx era", so the idle fraction may be stale. It is the strongest evidence in circulation that the shape of this path is real, and it is not a measurement of Apodex.

If the whole offloaded set crosses per physical batch — the shape that report describes — then at -ncmoe 17 that is 8.32 GB per -ub batch, and -ub defaults to 512 (common/common.h:452; -b defaults to 2048 at :451).

Here the vendor fact that this card does carry is the useful one. NVIDIA publishes "PCI Express Gen 5: Yes" for this board and Gen 4 for the previous generation's 16 GB card, and Gen 5 runs 32 GT/s per lane against Gen 4's 16, i.e. about 3.94 GB/s per lane after 128b/130b encoding rather than 1.97. So:

And that is the answer to the lane question without asserting a lane count: on a Gen-5 board the ×8 worst case equals the ×16 best case of the Gen-4 generation. Whichever width this card turns out to have, its transfer ceiling is at least as good as the best a Gen-4 16 GB card could offer, and possibly twice that. Read your own:

nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.width.max,pcie.link.gen.current,pcie.link.width.current --format=csv

If width.max reads 8 where a tier-mate reads 16, use the second row; if gen.current reads 4 or 3 where gen.max reads 5, you are in a slot that halves it again, silently, and that is by far the most likely way to lose this. Either way it is a time-to-first-token problem and not a tokens-per-second one, because at decode batch 1 those weights do not move. And if the link turns out to be your constraint, --no-op-offload disables the offload rule globally, keeping the expert matmuls on the CPU in both phases and removing the transfers entirely — a real trade, untested here, and one flag to try. Bandwidth-driven, on the host link.

Does the arithmetic survive contact with a real card?

Nobody has run Apodex on an RTX 5060 Ti, so this page is derived. But it is not derived in a vacuum, and the nearest measurement is much nearer than usual.

There is a llama-bench run of this GGUF architecture on this exact card. llama.cpp issue #26674 (open, bug-unconfirmed, 15 comments, last activity 2026-08-29) is filed by an owner of an "NVIDIA GeForce RTX 5060 Ti 16GB, Compute Capability: 12.0 (Blackwell), RAM: 64 GB" on Windows, running the "Official ggml-org Windows release version: 10289" with CUDA 13.3. The issue is about Gemma 4 being slow; the rows below are the reporter's control, the thing they say "performs normally":

What it establishes, and it is worth being precise because this is the closest thing that exists. The architecture string in that row is qwen35moe — the same string this file carries — running on a 16 GB Blackwell card from a 27.9 GB file, i.e. under expert offload, on a stock release binary, producing tokens. So the path works on this card, and the 8.9× gap between the offloaded MoE and the offloaded dense model of similar size is the mechanism this whole page rests on, measured on this board rather than argued: only one thirty-second of the expert bytes moves per token.

What it does not establish, and none of these is a small difference. It is Qwen3.6-35B-A3B, not Apodex; it is Q6_K at 27.9 GB, not Q4_K_M at 20.363 GiB, so far more of the model is on the host; the offload depth is whatever llama.cpp's auto-fit chose and is not stated anywhere in the thread, so it is not this recipe's -ncmoe 17; it is Windows with a display attached; and it is build b10289, older than the b10630 this page recommends. Do not read 5.49 tok/s as a prediction for this recipe. Read it as the one datapoint that says an arithmetic ceiling of 56–97 tok/s should be treated as a ceiling with a great deal of room underneath it, on a bigger file at a deeper offload than this recipe uses. Nobody has closed the gap between the two, and one run of this configuration would.

The offload ladder has a measured analogue one tier down. gemma4-26b on an RTX 3060 is the only expert-offload run in our catalogue with a written-down ladder: -ncmoe 12 gave 37.23 tok/s at an 11,179 MiB peak and -ncmoe 30 gave 21.20 tok/s on the same unchanged card — eighteen more blocks on the host path costing 43% of the generation rate — with -ncmoe 8 and below failing to load outright. Different model, different card, DDR4-2133 dual channel, headless, llama.cpp b10088. The direction and the magnitude of the effect are the point; none of those numbers is yours.

The prefill claim has the profiling report above, with the caveats stated there.

And the 131,072-token window has a control run on this model's base architecture. llama.cpp issue #27756 is a long-context failure report against Qwen3.8-27B — a different 64-layer member of this family, 48 of whose layers are Gated-DeltaNet — which emits EOS as its first generated token beyond roughly 98–130k positions on CUDA, on CPU, and on a fork. Its reporter ran a cross-model control on the same build, same server and same machine: "The same needle test at 243k tokens PASSES on Qwen3.5-35B-A3B (30 GDN layers)" — which is exactly this model's base and exactly this model's thirty Gated-DeltaNet layers. ⚠️ It is one box, one control run reported in passing, a needle-recall test rather than a quality evaluation, and the base model rather than this fine-tune. It is not a guarantee. It is the only evidence found in the space searched (the llama.cpp tracker and the 22 model cards enumerated under Results) that this architecture's recurrent state holds up past the window this page documents, and it points the right way.

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this GGUF — Apodex 1.1 mini — under llama.cpp, on any NVIDIA card, at any offload depth exists in the space searched on 2026-08-31. That space was: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces, re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100 and fetched one by one, scanned both for unit-bearing patterns and for bare numeric table cells, because a card can print its unit in a column header where a tok/s regex cannot see it (one of the 22, aykutx21/Apodex-1.1-mini-GGUF, has no card at all — the file is zero bytes); the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 273 for qwen35moe and 729 for n-cpu-moe on the same query shape; and our own /check, which returns unknown with zero benchmarks. Two things the search did find and one nearly-relevant negative:

  • kingjones777 publishes four of his own repacks at 64.87 / 63.30 / 45.05 / 32.54 tok/s, in ROCmFP4/ROCmFPX tensor formats only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (gfx1151) unified-memory APU at full offload — a machine with no expert offload and no PCIe hop, which is the entire subject of this page. Read the top of that range as unsettled: that build's own "Speed — full offload" table holds a dash beside "Full-offload speed being measured on an idle box, card will be updated", while the shared variants table thirty lines below prints 64.87 for it under a caption reading "A dash means I haven't measured that one yet — I won't put a number in a card I didn't measure." ⚠️ Those same four cards also carry three much lower numbers — 1.183, 0.292 and 0.163 t/s — and an offload-page reader who finds them will draw the wrong conclusion. Each is retracted by its own author in the sentence after it ("that was my harness's fault, not the model's"): they were measured on a box already serving eight models, and with -ngl 17 / -ngl 9, which is a layer count, not -ncmoe 17. Offloading whole layers to the CPU moves attention, norms and the shared expert off the card as well, so it measures CPU weight streaming rather than the technique this page documents — the author says as much, that "a partial-offload t/s measures CPU weight streaming, not the ROCm path". -ngl N and -ncmoe N look alike and do opposite things; this recipe pairs -ngl 99 with -ncmoe 17 for exactly that reason.
  • ghazni101 reports "a 400-token generation measured at 226 tok/s end-to-end for MQ4R". Three disqualifications, all verified at source: it is a .mq4r file under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF; and it is an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim. The same sentence continues "The Ornith reference builds of the same recipes measured ~235 tok/s (MQ4R) vs ~212 tok/s (MQ4RP)": those two belong to a different model and must not be read as a range around the 226.
  • The one repository whose card names an NVIDIA card, vcruz305, runs a "Smoke verification (RTX 6000 24GB)" of its own Q4_K_M and Q2_K builds of this model on llama.cpp b9835 — "loads; same prompts coherent" — and publishes no throughput number at all. It is quoted above for its display-headroom note, which is the useful part of it.

  One measurement on this pair would be worth more than all of the above. If you run it, contribute it.

• VRAM usage: 13.546 GiB derived sub-total at -ncmoe 17 and 131,072 tokens, plus the ~1,109 MiB driver share and 1024 MiB compute reserve this page sizes against but cannot derive. 7.752 GiB of expert weights sit in system RAM alongside. See /check/apodex-1-1-mini/rtx-5060-ti for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: decode reads 0.26 GB from your DIMMs and 2.0–3.4 GB from the card per token. On single-channel memory your DIMMs set the pace; on dual-channel DDR5 the card does; on dual-channel DDR4 it changes hands as your context fills. Prefill is where the host link is on the critical path, and it is the only phase where a narrow slot costs you anything — and on a Gen-5 board even a ×8 link matches the previous generation's ×16.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512), which is part of why the sub-4-bit rungs give up more than their file sizes suggest. The lead quant is imatrix-calibrated — the header records quantize.imatrix.chunks_count = 590 and quantize.imatrix.entries_count = 510 — and the publisher ships both the calibration corpus and the imatrix beside the ladder, which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-5060-ti.

Troubleshooting

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE, which arrived in (b7989, b7990] — so this means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

The build succeeds and the card is slower than a worse card

Check your CUDA toolkit before anything else: nvcc --version. Below 12.8 llama.cpp emits no sm_120 device code, for the reasons quoted under step 1, and your card falls back to JIT-compiling PTX built for an older virtual architecture. There is no warning. On a correctly built binary the startup banner names the architectures it was compiled for — a line of the shape CUDA : ARCHS = ... listing 1200, plus BLACKWELL_NATIVE_FP4 = 1, is what you want to see; a run posted from a working dual-Blackwell/Ampere box on issue #26257 shows exactly that banner as ARCHS = 860,1200 | ... | BLACKWELL_NATIVE_FP4 = 1.

The crash this recipe's flags are shaped like

llama.cpp issue #26609 (open, 4 comments, updated 2026-08-29) is "CUDA illegal memory access in cudaStreamSynchronize (flash-attn path) with Qwen3.6-35B MoE + partial expert offload — deterministic, cross-build", and its frozen argument set is this recipe's shape almost line for line: qwen35moe, partial expert offload to CPU, --parallel 1, --n-gpu-layers 99, --flash-attn on, --cache-type-k q8_0 --cache-type-v q8_0, --ubatch-size 512. The crash appears on the second request of a sequence and disappears with --flash-attn off.

Read its scope carefully, because it is easy to over- and under-claim. It is architecture-scoped, not card-scoped: the original reporter is on an RTX 5070 12 GB (Blackwell, Windows) and a second reporter reproduces the same signature on an RTX 3070 Laptop GPU 8 GB (Ampere, Linux), so it is a qwen35moe-under-expert-offload defect rather than a Blackwell one or a 12 GB one. Nobody has reported it on an RTX 5060 Ti. But nobody has reported the absence either, and this recipe runs -fa on.

So here is what the fallback costs on a 16 GB card, because "don't turn it off" is not a budget. Two things happen when you disable Flash Attention, and only one of them is the one people price:

1. The graph changes, not just the cache. On the non-FA path src/llama-graph.cpp:2607 builds ggml_tensor * kq = ggml_mul_mat(ctx0, k, q); and :2612 immediately calls ggml_mul_mat_set_prec(kq, GGML_PREC_F32);. That tensor is [n_kv, n_tokens, n_head] at F32, with head_count = 16, and the graph is reserved at n_tokens = min(n_ctx, n_ubatch). At -c 131072 -ub 512 that single tensor is 4.000 GiB — against 0.371 GiB of slack. It does not allocate.
2. -fa off forces only the V half of the cache back to f16, not both. The guard at src/llama-context.cpp:463-466 is if (!cparams.flash_attn) { if (ggml_is_quantized(params.type_v)) throw ... }, and the parameter-level version at :3675 tests params.type_v only. The K check at :3686 is gated behind params.flash_attn_type != LLAMA_FLASH_ATTN_TYPE_DISABLED, so with FA off it never runs. -ctk q8_0 -ctv f16 is legal, at 15,680 bytes per token rather than 20,480.

Put both together and the fallback is affordable on 16 GB, which is not true one tier down:

So if you hit that crash, the supported move on this card is -fa off -ctk q8_0 -ctv f16 -ub 128, plus either dropping to -c 32768 or paying four more steps of -ncmoe to keep the full window. ⚠️ The 131,072 rows are computed, not tested, and #26609's own matrix does not test them either: every row in it carrying 131072 ran with flash attention on — which is the bug being reported — and the only two passing off rows are both at -c 32768. So the issue's data is consistent with the table above and does not confirm it. Note also that the reporter's off probes are labelled "(KV f16)", taking both halves to f16 when only V requires it; that over-corrects by 0.293 GiB at 65,536 tokens, which matters more on a smaller card than on this one. And note that their GGML_CUDA_DISABLE_GRAPHS=1 probe did not help with this crash — which matters, because on a Blackwell card that same variable is the complete workaround for a different failure described in the next section. Two distinct problems, one environment variable, and it fixes only one of them.

The GPU stops responding after minutes of agent work — the Blackwell one

This is the hazard on this page most likely to bite a reader who actually uses the model for what it is for, and it has a one-line workaround.

llama.cpp issue #27330 — "CUDA graphs hang the GPU channel (RC watchdog + Xid 8) on RTX 5090 Laptop / sm_120; GGML_CUDA_DISABLE_GRAPHS=1 is a complete workaround" — is open, 4 comments, last activity 2026-08-29. Under sustained load the GPU stops servicing a command channel, the NVIDIA kernel driver's Robust Channel watchdog declares the card locked after a seven-second notify timeout and raises Xid 8, and llama-server aborts inside ggml_abort.

The trigger is this recipe's use case, not a synthetic one. The reporter's step 2 is "Drive it with a real coding agent workload (I used OpenCode) so the slot stays busy with long prompts and long generations", and "The hang is stochastic, not deterministic. It never happens on the first request." Across 11 observed hangs, time from server start to hang was min 4 minutes, mean 8.6, max 15.

The workaround is unambiguous and the evidence for it is A/B. "Setting GGML_CUDA_DISABLE_GRAPHS=1 eliminates the failure completely. Nothing else needs to change." The same workload then ran 50 minutes and 32+ minutes clean on the same boot, same GPU, same model, same client; the reporter puts "the probability of surviving 32 minutes by chance" at "under 3 percent". So:

GGML_CUDA_DISABLE_GRAPHS=1 ./build/bin/llama-server -m ./apodex-1.1-mini/... # rest as above

⚠️ If you set that variable, do not later write =0 meaning "off". ggml/src/ggml-cuda/common.cuh:1258 reads static const bool disable_cuda_graphs_due_to_env = (getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr); — presence, not value — so =0 disables CUDA graphs exactly as thoroughly as =1 does. The only way back to graphs is to unset it.

Scope, carefully, because the temptation is to claim this as this card's bug and it is not quite that. Four independent reproductions in the thread are all Blackwell — the original RTX 5090 Laptop (GB203M, compute capability 12.0), an RTX PRO 4000 Blackwell workstation card, a desktop RTX 5090, and a two-card 4090 + RTX 5080 split — and the last of those is the sharpest, because its owner reports the hangs "started after I replaced a RTX 4060 Ti 16G with a RTX 5080 (both as the second GPU next to the 4090); with the previous card this never occurred." One machine, one component swapped, and the previous-generation 16 GB card in the same slot never did it. Nobody has reported it on an RTX 5060 Ti, and nobody has reported its absence either.

Three differences from this recipe are worth stating rather than glossing, because they are the reasons this is a hazard to watch for and not a certainty:

• The architecture is qwen35, not qwen35moe. The model is Qwen3.8-27B — a dense 64-layer member of the same hybrid attention + SSM generation, not this MoE. Same family, different architecture string.
• Every repro is full offload (-ngl 999), not expert offload, so nothing in the thread exercises the -ncmoe path.
• MTP is ruled out as a factor, not implicated: the RTX PRO 4000 reporter had --spec-type draft-mtp --spec-draft-n-max 2 on in "crashing and stable alike — only delta is GGML_CUDA_DISABLE_GRAPHS=1".

What the repros do share with this recipe is the shape of the run: Blackwell / sm_120, -fa on, --cache-type-k q8_0 --cache-type-v q8_0, -ub 512, a long-lived server, and long agentic sessions. If your server dies after several minutes of coding-agent traffic with a CUDA error and dmesg shows NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked!, that is this, and the environment variable above is the whole fix. If it happens on your RTX 5060 Ti, the thread is short and its author has offered to run diagnostics — and please tell us too, because it would be the first report on this board.

It loads, then runs out of memory when a long prompt lands

Two causes with opposite fixes.

The KV cache grows at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the ladder and either raise -ncmoe or lower -c. On this file, raising -ncmoe by two frees between 0.844 and 0.973 GiB depending on which blocks the steps land on.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means staging those tensors there. Each of this file's expert triples is 0.4219–0.4863 GiB, so prefill peaks materially above the generation footprint the budget table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those ops on the CPU where their weights already are; expect slower prefill in exchange. If the card is also driving a display, start one or two steps deeper than the table and read nvidia-smi after load.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it.

The first is the offload: at -ncmoe 17 this recipe puts 7.752 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache, on by default at 8192 MiB — cache_ram_mib = 8192 at common/common.h:632, exposed as -cram / --cache-ram at common/arg.cpp:1713 with "-1 - no limit, 0 - disable" and a pointer to PR #16391, whose body says the cache "is stored in regular RAM". So a default server may hold up to 8 GiB beyond the offload — close to 16 GiB of host memory before the operating system gets any, which is why the command above pins it to 2.

There is one direct measurement of that growth on this architecture. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. Closed as completed, 2026-08-28. Two cautions if you cite it: the 126 MiB is the only measured per-entry figure anywhere and it is roughly twice one of this model's recurrent states (65,863,680 bytes = 62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support even though the 126 stands as a measurement; and it was measured on a different model of the same architecture, and rebuilt with -DGGML_VULKAN=ON gave +126.0 MiB per request against ROCm's +126.3 — so this is a model-and-request-layer behaviour and not a vendor story.

So 32 GB is the practical floor for this recipe as written and 16 GB is not enough. On a 32 GB box keep -cram 2048, or pass -cram 0 to disable caching entirely and trade prompt-reuse speed for headroom.

Generation is slower than you expected

Check three things, and on this card which one matters most depends on your memory.

Your DIMMs, first, and how much they buy you depends on what you have. Single-channel memory halves the host half of every decode step — 5.08 ms becomes 10.16 on DDR4-3200 — and on this card that is a 1.40× loss at a full window, because the host half is the larger of the two there. Confirm dual channel before anything else; it is the cheapest fix. Going further, from DDR4-3200 to DDR5-6000, is worth about 1.23× at that depth, which is real but much less than the first step.

Your context depth, second. At a full 131,072-token window the KV cache is 1.328 GiB of VRAM traffic per generated token, larger than every weight on the card put together, and about 3.18 ms of the 7.65 ms VRAM half. Generation slowing down as a conversation grows is not degradation, it is the roofline. If your agent loop never fills a six-figure window, -c 65536 at -ncmoe 16 is a row in the ladder above and halves that term.

Your offload depth, third and smallest. Each step is 13.5–15.6 MiB more read from RAM per token, about a quarter of a millisecond on dual-channel DDR4-3200. Do not go deeper than the ladder requires, but do not expect much back from going shallower either.

If it is specifically the wait before the first token that is bad while generation is fine, that is the bus and not the model — see Prefill is the phase the link pays for. And if nvcc --version reports anything below 12.8, fix that before you tune anything.

A measurement warning that is specific to this shape of run. Read llama-server's own predicted_per_second from the response's timings object rather than dividing generated tokens by total request time. On an expert-offload recipe prefill is bus-bound and slow while decode is not, so completion_tokens / wall_time makes decode look as though it collapsed when only prefill did. That is not a hypothetical: llama.cpp issue #27623 carries the headline "decode throughput collapses ~25x at context" on a hybrid Gated-DeltaNet model — and its own reporter retracted it in comment 6 of 6: "My measurement metric was flawed. The numbers in my matrix were completion_tokens / total request time, which includes prompt processing… That's why every configuration 'collapsed' to ~1.4 t/s: it was prefill…" The issue is still open with the retraction at the bottom, so a reader searching their own symptom meets the alarming headline first. Cite the retraction, not the headline.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true at common/common.h:476 — and moves MoE tensors to system memory until it can leave its 1024 MiB per-device margin. That is a reasonable path with two caveats: it records its choice nowhere except the log, so you cannot reproduce a run from the command line alone — which is exactly why the nearest measurement on this card, #26674 above, cannot be tied to an offload depth — and there is an open regression report against the margin flag on this architecture family: llama.cpp issue #27171, filed 2026-08-16 against Qwen3.6-35B-A3B Q4_K_M with --fit-target, still open, 3 comments, labelled bug-unconfirmed as of 2026-08-31, with a proposed fix (PR #27207) still open and unmerged on the same date.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp:484 throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line means your offload is being honoured, not that something broke.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp drives it with --spec-type draft-mtp — the exact spelling matters. On this card, price it first, because every cost lands on the 0.371 GiB of slack this page reserved:

• blk.40 stops being skipped: +0.444 GiB, and -ncmoe 17 does not touch it — the flag installs overrides for blk.0 through blk.16 only.
• The recurrent state is multiplied by the draft depth plus one, not by the draft depth. common/common.h:394 has need_n_rs_seq() return draft.n_max for the MTP/EAGLE3/DFlash/DSpark draft types, and src/llama-memory-recurrent.cpp:101 then allocates const uint32_t n_rows = mem_size * (1 + n_rs_seq);. So --spec-draft-n-max 3 takes the state from 0.061 GiB to 0.245, a delta of 0.184 — not to 0.184 itself.
• A second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k — so at 131,072 tokens on one layer that is 2,048 bytes per token, 0.250 GiB.

That is 0.878 GiB against 0.371 GiB of slack, so on this configuration it does not fit at all; you would be raising -ncmoe to pay for it, before any throughput has arrived.

And the return is unknown. Unsloth's published guidance for this class of model is that "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs) — 1.15–1.25× at best, before offload is counted. Under offload even that is contested: an owner describing their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" running a different model of this architecture at --n-cpu-moe 25 with the MTP flags on reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" (discussion #18 on that model's GGUF repo), and a second community member there offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model.

⚠️ And there is a crash report in this area worth knowing about. llama.cpp #26558 records llama-server aborting with a cublasSgemm CUDA_ERROR_INVALID_VALUE under --spec-type draft-mtp with parallel load and a saturated KV cache, on an RTX 4090 with CMAKE_CUDA_ARCHITECTURES=89. Read its close carefully: it was closed as completed on 2026-08-28 with the comment "Closing as it no longer repros apparently", after the issue's own author reported the crash gone following a CUDA toolkit upgrade — that is a lapsed reproduction, not an identified fix. The thread does name a candidate: a collaborator points at PR #26574 ('ggml-cuda: provide static workspace for cuBLAS handles', merged 2026-08-20), which "should fix a cudaGraph-associated memory-leak" for CUDA toolkits below 12.4 — offered as a possible cause, not established as this crash's. That toolkit range is one this card cannot be in anyway, since 12.8 is its floor, and the reported condition is parallel load, which -np 1 is not. A hazard to recognise rather than one this recipe is exposed to.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps's hardware table names "IQ2_M or IQ3_M" for a 16 GB card, with "IQ2_M full offload; IQ3_M may require hybrid offload". Weigh it against the rest of that card, which is candid: "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." and "Prefer Q4_K_M or higher when system memory permits."

Check the arithmetic before taking it. Against this page's 13.917 GiB ceiling, with the recurrent state included:

So IQ2_M genuinely escapes offload at the full window, Q2_K clears it with almost nothing to spare, and the whole 3-bit tier does not clear it at any context. That is a 2-bit quantisation of a model whose routed experts are 512 wide, used for tool calling where a malformed argument is a failed task. Below Q4 the trade on this card is quality for no -ncmoe, which is a cleaner trade than the 12 GB tier gets, and it is still a large quality bet to avoid a memory purchase. Capacity-driven.

Choosing a different quant publisher

Conversions of this model are still appearing: an enumeration on 2026-08-31 returned 22 repositories matching Apodex-1.1-mini across 11 publishers, one of which appeared two days earlier. Re-enumerate before you decide, and check the byte count and the file list rather than the tier name — across publishers a file named Q4_K_M spans more than a gigabyte for the same nominal tier, and one publisher notes on their own card that their Q3_K_S, Q3_K_M and Q3_K_L "land at the same byte size" under their override set. Within bartowski's own ladder the rungs adjacent to the lead are Q4_K_S at 19.619 GiB and IQ4_XS at 17.955 GiB; each buys back host RAM rather than VRAM, which on this card is the resource you have most of.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, and on a Blackwell card an NVFP4 build looks like the obvious answer. It is not, for three separate reasons. They are vLLM/SGLang safetensors rather than GGUF and llama.cpp cannot load them at all. They are far too large for this card before anything else is counted — 22.548 GiB and 22.958 GiB of weights for the first two, against 16 GiB. And the -NVFP4 repo is not a 4-bit checkpoint anyway: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4. Note also that their Quick Starts launch on a single GPU: --tp-size 1 on both -NVFP4 and -GPTQ-Int4 and --tp 1 on -FP8; the --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone. For one RTX 5060 Ti the GGUF route above is the path.

The two llama.cpp reports that name this exact card

Both are worth knowing about and neither is a reason not to run this recipe. Note also a search trap if you go looking yourself: a tracker query for 5060 matches on the issue number — #5060 is "Add Q3_K_XS", from an entirely different era — so a card-name search returns unrelated hits unless you read each one.

• #26674 is the Gemma-4-is-slow report whose qwen35moe control rows are quoted above. Open, bug-unconfirmed. The reporter's own dense/MoE comparison on this board is the most directly relevant thing in it for this recipe.
• #26257 is "Qwen3.6-27B garbled output on dual-GPU CUDA (RTX 5060 Ti + RTX 3060), single GPU works" — open, 5 comments, reproduced on Vulkan as well as CUDA by a second reporter. ⚠️ It is about dense Qwen3.6-27B and Qwen3.8-27B, not about this architecture, and its own title states the shape of the problem: it is a multi-GPU split issue. This recipe is single-GPU by construction, and the report's own working configuration — "running on the RTX 5060 Ti + system RAM does not result in garbled output" — is this recipe's shape. If you later add a second card and see garbled text, that thread is where to start.

A third result, #26205, is titled for an RTX 5060 and its hardware line reads "NVIDIA GeForce RTX 5060 Laptop GPU", with 8 GB of VRAM, — a different part with half the memory, on a third-party Blackwell build. A card name is not a card; that one does not apply here.

Other pages on this exact card worth reading

Qwen3.8-27B on RTX 5060 Ti is a dense model of similar size that fits entirely on the card, and it is the useful contrast — as that page stood on 2026-08-31, its own arithmetic puts its per-token VRAM read at roughly 13.4 GB against this recipe's 2.0–3.4, because a dense model has no experts to skip. That is why an offloaded 36B can plausibly out-decode a resident 27B on the same board, and it is the same ratio the #26674 rows show measured. MiniMax H3 on RTX 5060 Ti is the page that works the host link on this card in detail, and Nanbeige4.2-3B on RTX 5060 Ti is what the same 16 GB looks like when the whole model fits. And if you have 24 GB and landed here by accident, Apodex 1.1 mini on RTX 3090, published 2026-08-28, fits the whole model on the card at the same 131,072-token context with none of this page's arithmetic; Apodex 1.1 mini on RTX 5090 does the same at Q5_K_M and the full 262,144-token window.

The model emits garbage, or stops immediately on a very long prompt

Two different things, and only the second is architecture-specific.

For uniform-probability gibberish on a single card, check which backend the report you found is about. The nearest to this recipe is issue #25857, filed 2026-07-18 on a Windows 12 GB NVIDIA box: every token at an identical log-probability at any offload setting. Read its scope first — it is a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins, and it remains open with the stale label and zero comments as of 2026-08-31. Rebuild at a current release before inheriting the symptom. For garbled output across two cards, see #26257 above.

For a completion that comes back empty with a clean HTTP 200 after a very long prompt, see issue #27756, which documents exactly that failure mode on a 64-layer sibling architecture beyond roughly 98–130k positions, reproduced on CUDA, on CPU and on a fork. As noted above, the same reporter's control run passed at 243k tokens on this model's 30-layer base architecture — so this window is not known to be affected, and that is the strongest statement the evidence supports rather than a guarantee. If you hit it, the signature is tokens_predicted: 1 with stop_type: "eos" and empty content, and it is worth telling us as well as the tracker.

For completeness on the issue this catalogue's older Apodex pages point at: llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label, state re-read 2026-08-31. Anything you read about it that was written before that date will call it open, including this site's own earlier pages for this model — that dates the text rather than changing the advice. It never applied to a text-only run, which is what --no-mmproj makes this one.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well — checked 2026-08-31, with ollama.com/library/qwen3.5 and the registry.ollama.ai manifest for qwen3 both returning 200 as controls. Use llama.cpp directly. If a tag appears later, please tell us.