Apodex 1.1 mini on RTX 5080: a 36B agent at 128K on Blackwell, where your DIMMs decide the pace

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one RTX 5080, at a 131,072-token working context, from a Q4_K_M GGUF that is larger than the card.

The weights load 19.908 GiB. The card holds 16. It runs because the model is a sparse Mixture of Experts and llama.cpp will leave the routed experts of the first N blocks in system RAM: 123 tensors carry 18.586 GiB of the file, and across the forty blocks a normal run loads that is 18.164 GiB of the 19.908 GiB resident — 91.2%.

Hardware data: RTX 5080 (16 GB GDDR7, 256-bit, 960 GB/s) · 13.546 GiB derived working set at -ncmoe 17, 131,072-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5080. /check/apodex-1-1-mini/rtx-5080 returns unknown with zero benchmarks (checked 2026-08-31). Every figure below is arithmetic over this GGUF's own tensor table, over llama.cpp's allocation code at a pinned release, and over specifications NVIDIA publishes — written out so you can check it. There is no throughput measurement on this page; Results names the space searched and what does exist. If you run it, please send the numbers.

🧭 The conclusion that is specific to this card, before the arithmetic that gets there. Every 16 GB card in this family runs the same offload — capacity is capacity, and the ladder below is byte-for-byte what an RTX 4080 or an RTX 4060 Ti gets. What is not the same is which half of a decode step you are waiting on, and this is the one 16 GB board where that question has no single answer. At -ncmoe 17 a decode step reads 0.2601 GB from system RAM and 2.00–3.43 GB from VRAM, so the two halves cost the same wall-clock only when the card's bandwidth is 7.70× to 13.18× your DIMMs'. This card publishes 960 GB/s. Dual-channel DDR5-6000 gives a ratio of exactly 10.0× — which sits inside that range, so on the most common configuration for this card decode is host-bound on a fresh conversation and VRAM-bound once the context passes about 55,000 tokens. The bound is about 72.85 GB/s: only memory slower than that — dual-channel DDR4-3200 and below — stays host-bound across the whole window, while every dual-channel DDR5 configuration flips somewhere inside it (DDR5-4800 at ~115,000 tokens, DDR5-5600 at ~72,000), and DDR5-8000 is flipped from the first token. Where the decode time actually goes derives the crossing point for six memory configurations. Everything about capacity on this page transfers across the 16 GB tier; everything about speed stops at this board's 256-bit GDDR7 bus.

ℹ️ This recipe is text-only, deliberately. The checkpoint has a vision tower and the lead GGUF repo ships two projectors for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-31). The smaller projector is 0.838 GiB, which on this budget is just under the 0.844 GiB that the two cheapest steps of -ncmoe would free spent on a capability the recipe does not use, so the run command refuses it explicitly rather than merely omitting the flag. Capacity-driven.

⚠️ Three version floors, and only the first is about the architecture loading.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at b7989 and present at b7990, so it arrived in the interval (b7989, b7990]. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that keeps blk.40 off the card arrived in (b10211, b10212]. Both are intervals bracketed by adjacent published tags, which is as fine as a tag-granularity probe resolves.
• The CUDA toolkit, and on this card it is a hard number rather than a suggestion. llama.cpp appends Blackwell device code only when CUDAToolkit_VERSION is at least 12.8 — see the build gate, which is the one install step on this page that is not the same as on an Ada 16 GB card.
• A throughput floor. Use b10630 or newer — the release the lead quant was produced with. llama.cpp issue #25162, closed as completed 2026-08-03, records a 24–42% regression on Turing (SM75) in ssm-scan.cu / ssm-conv.cu, the kernels behind thirty of this model's forty layers, measured directly on Qwen3.5-35B-A3B — this model's base — with "Gemma4 (no SSM layers) is unaffected." An older build can cost you silently rather than loudly. ⚠️ But do not read this floor as "the regression is fixed" — nobody ever identified a fix. The issue closed when a third party retested and reported "0b14b87d7 give：27.47 tok/s， close issue", back at the 27.2–27.3 tok/s the pre-regression commits in that thread record. 0b14b87d7 is itself "server: add notice for upcoming default port change 8080 --> 9931" — an unrelated commit, so it dates a tree state in which the regression was gone, not a repair. The one candidate the thread raises, PR #25185 ("CUDA: consistent use of __restrict__ + PDL for FA", merged), is scoped to flash attention rather than to the ssm-* kernels the regression was measured in, and no one in the thread confirmed it as the cause. So the supported statement is that performance had recovered by that tree, cause unidentified — which is a reason to stay current rather than a guarantee. That regression was measured on Turing and nothing about it is established for sm_120; what the source does say about this card is a different and narrower statement.

Every source line quoted on this page was read at release b10666 (4e97ac86ebe2c4cb8212d98d2641ad6768810896, dereferenced from the tag rather than inferred), and line numbers are given for that tree only. The tag moves several times a day — three shipped inside one hour on 2026-08-30 — so this page pins rather than claiming currency. Note also that GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, because every bNNNN tag is marked prerelease; use /releases?per_page=N if you script a version check.

Requirements

What NVIDIA publishes about this board, and what it does not. From the RTX 5080 product page, read 2026-08-31: 16 GB GDDR7, Memory Interface Width 256-bit, NVIDIA Architecture Blackwell, boost 2.62 GHz / base 2.30 GHz, Total Graphics Power 360 W, Required System Power 850 W, and a Technology Support row pairing "PCI Express Gen 5" with "Yes". What that page does not carry, counted today: no memory-bandwidth figure, no memory-speed figure, and no PCIe lane count — the strings Bandwidth, GB/s, Memory Speed and lanes occur zero times on it. The same omission as on the Ada product pages, and the same discipline follows from it.

The bandwidth figure is nevertheless a vendor fact here, because NVIDIA states it elsewhere. The RTX Blackwell GPU architecture whitepaper gives it twice — in prose, "the GeForce RTX 5080 ships with 30 Gbps GDDR7 memory, delivering" 960 GB/sec "of peak memory bandwidth", and in Table 4 (GeForce RTX 5080 vs GeForce RTX 4080 vs GeForce RTX 3080 Specs), which also gives GB203, 84 SMs, 10,752 CUDA cores, a 2617 MHz boost clock, a 256-bit interface and a 30 Gbps data rate. The same table gives the RTX 4080 716.8 GB/sec at the same 16 GB and the same 256-bit width, so the card this recipe was cloned from differs from it by a factor of 1.339 in the only spec that matters below.

⚠️ Do not derive this card's bandwidth the way you would derive an Ada card's. The familiar memory clock × 2 × bus bytes shortcut describes double-data-rate signalling; the same whitepaper states that GDDR7 moved to "PAM3 signaling (3 levels delivering 1.5 bits per cycle)" from GDDR6X's "PAM4 signaling (4 levels delivering 2 bits per cycle)", so "clock × 2" is not what is happening on this board. The arithmetic that does hold is the one on the data rate: 30 Gbps across a 256-bit (32-byte) bus is 30 × 32 = 960 GB/s, exactly the published figure. If you want to read it off your own machine, nvidia-smi reports a memory clock, not a data rate — treat the product of the two as a cross-check on 960, not as an independent derivation.

nvidia-smi --query-gpu=name,memory.total,clocks.max.memory,pcie.link.gen.max,pcie.link.width.max --format=csv

This page asserts no PCIe lane count. NVIDIA publishes the generation (Gen 5) and not the width; read pcie.link.width.max from the line above and see Prefill is the phase the link pays for, which prices both ×16 and ×8 rather than picking one.

For your DIMMs, do the arithmetic from their rated transfer rate: dual-channel DDR5-6000 is 6000 MT/s × 8 B × 2 = 96 GB/s, dual-channel DDR4-3200 is 51.2 GB/s, and a single-channel configuration halves whichever you have. On this page that number matters more than it does on any other card in the tier, because it is the one that decides the answer.

Installation

1. Build llama.cpp with CUDA, and check your toolkit version

This is the step that is genuinely different on Blackwell, and it fails quietly rather than loudly. ggml/src/ggml-cuda/CMakeLists.txt annotates its own architecture list, and the entry for this card reads "120 == Blackwell, needs CUDA v12.8, FP4 tensor cores" (:19). The logic underneath matches the comment:

# ggml/src/ggml-cuda/CMakeLists.txt at b10666
if (GGML_NATIVE AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24")
    set(CMAKE_CUDA_ARCHITECTURES "native")                                    # :27
else()
    ...
    list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real)       # :34
    if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8")
        list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real 90-virtual)              # :37
    endif()
    if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8")
        list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real)                       # :51
    endif()

Three things follow, and the third is the one that bites.

1. 120a-real is appended only at CUDA 12.8 or newer. Below that, a non-native build contains no Blackwell device code at all, and the newest virtual (PTX) architecture in the list is 90-virtual — there is no 120-virtual. Your card would run Hopper-targeted PTX, JIT-compiled by the driver on first use.
2. GGML_NATIVE normally saves you. It defaults to ON — ggml/CMakeLists.txt:123 takes its value from GGML_NATIVE_DEFAULT, and :105-110 sets that to ON in the ordinary case — so a build on the machine that owns the card compiles for the card.
3. Name the paths that escape it, because they are not exotic. :105 reads if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}) → GGML_NATIVE_DEFAULT OFF. SOURCE_DATE_EPOCH is the reproducible-builds variable that distribution packaging, container image builds and several CI images set as a matter of course — so a package or image built on a pre-12.8 toolkit can silently contain nothing for your GPU, with no error anywhere. -DGGML_NATIVE=OFF, cross-compiling, a CUDA toolkit older than 11.6 and a CMake older than 3.24 do the same by failing the :27 condition.

So: check nvcc --version before you build, and prefer building on the box that owns the card.

nvcc --version | tail -2          # must report 12.8 or newer
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

And on Windows, there is a measured reason to build rather than download — on this exact card. llama.cpp issue #26918, "Performance regression between b10356 and b10359 (RTX 5080 / Blackwell, ~40% slower)", closed as completed 2026-08-12, is a four-build comparison on an RTX 5080 (16GB, Blackwell, compute capability 12.0), Windows 11, driver 610.62, CUDA 13.3, using the official prebuilt win-cuda-13.3-x64 binaries. Prompt processing fell from 1029–1271 tok/s at b10356 to 413–534 at b10369. The reporter then rebuilt the same, newer source with MSVC instead of the prebuilt toolchain and reported "all performance regressions disappear, and performance actually exceeds the b10356 baseline" — 1427 pp512 against 1029 — concluding that "The regression is compiler-dependent, not code-dependent." and attributing it to "suboptimal kernel codegen in the Clang toolchain for the Blackwell sm_120 target." ⚠️ Carry the limits: a different model (Muse-Glimmer-30B at IQ2_M) with speculative decoding on, one box, Windows only, no maintainer verdict on the diagnosis, and a build window some 280 tags below this page's floor — nothing there says the current release binaries are affected. It is a reason to benchmark a source build against the prebuilt one on this card before concluding your machine is slow, not a claim about b10630.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes (20.363 GiB), imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z; the file list, the byte count and the download URL were re-fetched and HEAD-checked on 2026-08-31 (the resolve URL answers 302 with x-linked-size: 21864082336, then 200) rather than copied from a sibling page.

Disk size and VRAM residency are different numbers here, and the gap has two parts. The file is 21,864,082,336 bytes (20.363 GiB) on disk; a normal run loads 21,376,133,632 (19.908 GiB). The difference is 487,948,704 bytes, and only most of it is the model. 476,956,672 of it is blk.40, the multi-token-prediction head, created with TENSOR_SKIP unless you ask for it (src/models/qwen35moe.cpp). The remaining 10,992,032 bytes (10.5 MiB) is the file's non-tensor content — the GGUF header, the key-value metadata, the tensor-info table and the alignment padding — and it is exactly 21,864,082,336 − 21,853,090,304, the file size minus the sum of its tensor payloads. So the MTP block alone does not account for the gap. Budget 22 GB of disk and 19.908 GiB of weights, and do not substitute one for the other.

Do not start from the publisher's quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card itself says "llama.cpp downloads the mmproj automatically when using" -hf. It does: find_best_mmproj matches any sibling filename containing mmproj, this repo ships two (899,283,488 and 902,822,432 bytes, re-checked today), and no_mmproj defaults to false while mmproj_use_gpu defaults to true (common/common.h:596,594). So the quickstart puts 0.838 GiB of vision projector on a card this recipe has already budgeted to within about a third of a gigabyte. Download the file explicitly, as above, and load it with -m. Capacity-driven.

Q4_K_M on a 16 GB card is a choice, and the alternative is narrower than it looks. Ten of bartowski's rungs are under 16 GiB on disk (re-measured 2026-08-31; IQ3_M at 16.178, Q3_K_L at 16.165 and Q3_K_XL at 16.579 are the three that just miss), so "small enough to load" is not the constraint — what has to sit beside the weights is. Hold this page's own ceiling (derived below: 13.917 GiB for weights plus KV plus recurrent state) and a 32,768-token q8_0 cache, and a fully resident configuration needs weights at or below 13.524 GiB. The whole 3-bit tier misses that: IQ3_XXS is 14.287 GiB and Q3_K_S is 14.886. The escape from offload is not "drop to 3-bit", it is "drop to 2-bit" — Q2_K_L at 12.650 GiB, or IQ2_M at 11.682. That is exactly where a second publisher draws the line: abenzerps's hardware table gives the 16 GB row as "IQ2_M full offload; IQ3_M may require hybrid offload" and then closes the section with "Prefer Q4_K_M or higher when system memory permits." This recipe follows that closing sentence, because the model is an agent that runs tool calls and multi-step plans, and a malformed argument is a failed task rather than a clumsy sentence. Capacity-driven — the same choice on any 16 GB board.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 17 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

Nine of those flags are decisions rather than defaults. Each default was read at b10666 for this page rather than carried across from another card, and every one is labelled capacity-driven — the same on any 16 GB card — or bandwidth-driven, meaning it is not.

• --no-mmproj refuses the vision projector rather than merely not asking for it: 0.838 GiB, for the reason under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point; -ncmoe then walks part of it back off. Capacity-driven — it is the baseline the offload is measured back from, and it is also what keeps the KV cache and the recurrent state on the card.
• -ncmoe 17 keeps the routed experts of blocks 0–16 in system RAM. The flag's help text at this pin is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp) and the implementation matches the wording exactly. 17 is derived in Choosing your own -ncmoe as the shallowest rung that clears this page's ceiling at 131,072 tokens — not a measured optimum, and not inherited. Capacity-driven for the choice — but note that on this card shallow is also fast, for a bandwidth reason given below, so the two arguments agree here rather than trading off.
• -np 1. llama-server sets params.n_parallel = -1; // auto by default (common/arg.cpp) and resolves that to four slots sharing one KV pool. The Gated-DeltaNet recurrent state is allocated per sequence, so four slots take it from 0.061 GiB to 0.245 and split your context four ways. One slot is what a single-user agent loop wants. Capacity-driven.
• -c 131072 is half the model's declared context_length of 262144. The ladder below prices the other half; it costs three more steps of -ncmoe. Both, and on this card it is the single largest speed lever — capacity-driven for the 1.328 GiB the cache occupies, and bandwidth-driven because that cache is re-read from VRAM on every generated token. Context depth moves the crossover from 7.70× on an empty window to 13.18× on a full one, and on this board that range straddles the ratio a DDR5-6000 machine actually has.
• -ctk q8_0 -ctv q8_0 quantises the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 131,072-token cache from 2.500 GiB to 1.328, which on this card is a saving of 1.172 GiB — worth between two and three steps of -ncmoe you do not have to spend, since a step costs 0.4219 or 0.4863 GiB so two come to 0.844–0.973 and three to 1.266–1.459. Both — capacity-driven for the 1.328 GiB, and bandwidth-driven because halving the cache halves the largest single term in the decode budget.
• -fa on makes Flash Attention explicit. The default at this pin is auto — flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO (common/common.h:499) — and auto would turn it on anyway here, because a quantised V cache requires it (src/llama-context.cpp:3675-3683: with -fa off and a quantised -ctv the context is not created, it errors with "quantized V cache requires flash_attn to be enabled" and returns). Pinning it means a future change to what auto decides cannot silently change your memory profile. ⚠️ This is also the one flag on this page with an open, deterministic crash report against it on the same GPU architecture — see If it crashes with an illegal memory access, which prices the workaround in full — it moves two terms, not one, and the larger of them is a 4.000 GiB graph tensor that forces -ub down alongside it. Capacity-driven.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp). Taking that advice makes the 7.752 GiB of host-side experts ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure and not a cache you can squeeze. Bandwidth-driven, on the host side: the point of the pinned host buffer is the rate the CPU reads those experts at during decode — and on this card that rate is the larger half of a decode step for most of a conversation.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632 — and it costs more on this architecture than on a plain transformer. See System RAM under Troubleshooting. Capacity-driven, but on host RAM rather than on VRAM — it does not appear anywhere in the card's budget.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05), whose text notes that Apodex "follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It appends one buffer-type override per block index 0 … N-1, each built from a regex in common/common.h:1130:

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, LLM_FFN_EXPS_REGEX, params.tensor_buft_overrides)

The loader matches that against tensor names and, on a hit, gives the tensor a CPU buffer type at tensor-creation time, before any buffer exists — so those bytes are allocated in host memory and never enter VRAM at all. Nothing in the path consults the architecture, which is why it works identically on this hybrid recurrent model and on a plain transformer MoE. 123 tensors in this file match, three per block across all 41 blocks, totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads.

Three things it does not move, all of which stay on the card and all of which show up in the decode budget later:

• The attention and Gated-DeltaNet weights. A block's attn_* tensors and its ssm_conv1d / ssm_a / ssm_alpha / ssm_beta / ssm_norm / ssm_out do not match _exps.
• The shared expert and the router. ffn_*_shexp and ffn_gate_inp do not match either, so the dense path every token takes stays resident, and routing is decided on the GPU.
• The KV cache and the recurrent state. Both take their buffer type from the layer's device, which -ngl alone decides; -ncmoe rewrites weight tensors and never touches the layer assignment. A block whose experts are in RAM still keeps its recurrent state in VRAM — which is what makes the budget below decomposable at all.

Add it up and offloading everything the regex can reach, -ncmoe 40, still leaves 1.744 GiB resident: 0.655 GiB of token_embd and output, and 1.089 GiB of everything else across forty blocks.

Does that actually happen on this architecture? The source says so, and there is an independent report that the offload takes effect on a hybrid MoE: llama.cpp issue #27698 records a run with --fit off --n-gpu-layers 46 --n-cpu-moe 6 where "the weights load (expert offload is honored)" and the failure comes later. ⚠️ That report is an RX 6700 XT (gfx1031, RDNA2) on ROCm and its offload-honoured branch is a gemma4moe model rather than this one, so nothing about the failure transfers to this card — it is cited for the one clause quoted, plus a magnitude in the next section. There is also a second, closer confirmation on this architecture and this GPU architecture: llama.cpp #26609 is an RTX 5070 (Blackwell, sm_120) running a qwen35moe model with the experts of blocks 16–39 pinned to CPU by an equivalent --override-tensor regex, and its whole isolation matrix is about what happens after that offload loads successfully — including a probe with "ALL experts to CPU". The offload working is the premise of that report, not its subject.

The VRAM budget

Weights, KV cache and recurrent state at the recipe's settings. Every figure is derived from the GGUF's own tensor table and from llama.cpp's allocation code at b10666. All three rows are capacity-driven and identical on any 16 GB card.

The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 17 moves 8,323,596,288 of that (7.752 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks load; full_attention_interval 4 makes ten of them full-attention — blocks 3, 7, 11, 15, 19, 23, 27, 31, 35, 39. Each carries head_count_kv 2 at key_length and value_length 256, so 1,024 cache elements per token per layer and 10,240 across the ten. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows with context.
• Recurrent state. Those thirty blocks each allocate n_embd_r() = (ssm_d_conv − 1) × (ssm_d_inner + 2 × ssm_n_group × ssm_d_state) = 3 × (4096 + 2×16×128) = 24,576 elements and n_embd_s() = ssm_d_state × ssm_d_inner = 128 × 4096 = 524,288 elements, both F32 (src/llama-hparams.cpp). That is 2,195,456 bytes per layer per sequence and 65,863,680 across thirty. llama-server prints it at startup as RS buffer size.

The two terms that table cannot produce, and what this page reserves for them

13.546 GiB is not a peak. Two things sit outside it, and on a card this tight they decide whether a configuration loads. Rather than leave them as a warning, this page reserves for them explicitly.

1. A 16 GB card does not give you 16 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. The only expert-offload run in our catalogue with the numbers written down is gemma4-26b on an RTX 3060 at -ncmoe 12, whose submitter recorded a peak of 11,179 MiB of that card's 12,288 on a headless box (benchmark id 280, re-read 2026-08-31) — so at most 1,109 MiB was unavailable to the model there. ⚠️ That is a different board in a different VRAM tier, and the term is not proportional to capacity (it is a context plus a driver reservation, not a percentage), so borrowing the absolute 1,109 MiB onto a 16 GB card transfers an absolute quantity rather than measuring this one. It is the best number in the catalogue and it is the direction this page errs in: this page reserves 1,109 MiB, and a Windows box driving a monitor should expect to want more. There is one same-card sighting that points the other way and is worth naming because it measures a different quantity: llama.cpp #26901 logs an RTX 5080's free VRAM as "~15.8 GiB (actual VRAM)" on a Linux box — but that is free memory read before a CUDA context exists, not a peak-against-capacity gap, so it bounds only the driver half of the term.

2. The compute buffers. llama.cpp allocates graph working memory on top of everything above, and under expert offload it also stages host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 0.4219 or 0.4863 GiB, so that staging term alone is of that order. This page reserves 1024 MiB, which is llama.cpp's own per-device auto-fit margin: fit_params_target is initialised to 1024 * 1024*1024 at common/common.h:481. It is the runtime's opinion of how much room it wants, not a figure this page invented. For scale on leaving nothing: #27698 records a clean weight load followed by context creation failing on a ~504 MiB compute buffer with cudaMalloc failed: out of memory — a different card, a different backend and a gemma4moe model rather than this one, quoted only for the magnitude of a term every table in this family excludes and for the shape of the failure it produces.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 16,384 − 1,109 − 1,024 MiB = 14,251 MiB = 14,943,256,576 B = 13.917 GiB.

At -ncmoe 17 and 131,072 tokens the sub-total is 14,544,464,384 B, which clears that by 398,792,192 B (380.3 MiB, 0.371 GiB). Read your own numbers off the server's startup log rather than trusting mine: llama.cpp prints model buffer size once per buffer, so under -ncmoe you get a CUDA line and a host line and the split above is directly checkable, followed by KV buffer size, RS buffer size and compute buffer size. If you do, those figures are the most useful thing you could send us — more useful than a speed number, because they are the two terms nobody in this family of recipes has been able to derive.

Choosing your own -ncmoe

Every row is weights + KV + recurrent state, q8_0 KV, one slot. Free of 16 GiB is what the naive arithmetic says; Slack is what is left once the driver share and the compute reserve are taken out, and the two columns disagree by more than 2 GiB, which is the whole point of the previous section. Capacity-driven throughout: this table is the same on every 16 GB card in the catalogue.

Two rows are worth reading twice. -ncmoe 16 at 131,072 tokens has 1.968 GiB "free" and does not load — that is what the ceiling is for. And -ncmoe 17 is the recipe's setting because it is the shallowest rung that clears the ceiling at this context, which is a derivation and not a preference: 16 misses by 0.115 GiB. The bottom rows are the model's entire declared 262,144-token window, which this card can hold at -ncmoe 20.

On this card, shallow is also fast, which is not true everywhere. Going deeper is always safe on capacity and always worse on speed, because every step moves bytes from the fast side of the budget to the slow one — see the decode roofline. On a 128-bit 16 GB board the same step is nearly free because the card is the bottleneck anyway; here it is not. So take the shallowest rung the ceiling allows, and buy context reductions rather than offload reductions when you need room.

Then stop guessing and measure it. llama.cpp ships a tool that answers this on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 131072

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 131072 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device split of total / free / model / context / compute / unaccounted. That compute column is what this page reserved 1024 MiB for, and unaccounted is your driver's share. ⚠️ On this card, read the free figure in that output with suspicion before you trust it — see the auto-fit warning.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys X GB" is wrong here. bartowski's recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28, 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates. Read the cumulative column above rather than multiplying. Nine of the seventeen blocks this recipe offloads are the expensive kind, which is why -ncmoe 17 moves 7.752 GiB rather than seventeen equal shares of anything.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 17.955 GiB on disk (re-measured 2026-08-31) and its ladder is linear. If you would rather spend quality than offload depth, that is the trade — but derive its table from its own tensor list; do not scale this one. Capacity-driven.

Where the decode time actually goes

This is the section that is not the same on another 16 GB card, and it is the reason the capacity numbers above are interchangeable across the tier while the performance advice is not.

The mechanism first. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights" (ggml/src/ggml-backend.cpp:942) — unless the scheduler finds "a backend with higher prio wants to offload the op" (:960). CUDA answers that with a single comparison:

// ggml/src/ggml-cuda/ggml-cuda.cu:5341-5344 at b10666
static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
    ...
    return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size;
}
// :5515 — const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv(...)) : 32;

Generating a token is a batch of one, and one is less than 32, so at decode the offloaded expert matmuls stay on the CPU and those weights never cross PCIe at all. Prompt processing is a batch of hundreds, so at prefill those same tensors are copied host→device. The two phases are bound by different things and this page treats them separately.

The decode roofline, in bytes

expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token. Everything else on the card is read whole. At -ncmoe 17:

Two things there deserve a note, because they are what make the answer below sharp enough to depend on your DIMMs.

The KV cache is decode traffic, not just decode capacity. Every generated token reads the whole occupied cache back out of VRAM. At a full 131,072-token window that is 1,426,063,360 bytes — the single largest term in the table, larger than every weight the card holds put together. Pricing it as a capacity line and leaving it out of the bandwidth arithmetic is easy to do — the budget table above treats it as capacity, and so does every offload budget in this family — and on this card leaving it out moves the answer across the crossover rather than merely rounding it.

The output.weight head is separated from the embedding table. They are usually lumped as "0.655 GiB of embeddings", but only one of them is read per token: output.weight is 417,177,600 bytes of Q6_K read in full to produce logits over 248,320 tokens, while token_embd.weight is read one row at a time — 1,152 bytes at Q4_K. Splitting them takes the non-expert VRAM read from a loose upper bound of 1.744 GiB down to a near-exact 1.4776 GiB. Both refinements make the VRAM side smaller, which pushes the conclusion below toward "host-bound"; the fact that it still lands where it does is therefore conservative in the direction it is claimed.

The crossover, and why this card has no single answer

The two halves of a decode step take equal wall-clock exactly when your card's bandwidth is VRAM bytes ÷ host bytes times your DIMMs'. That ratio needs no bandwidth figure at all, which is why it is the form this page states first. Every ratio below contains the recurrent state and the KV cache at the stated occupancy, and uses the tightened non-expert bound rather than the loose 1.744 GiB one — four different values are in circulation for one such comparison, differing only in which terms they hold, so naming them is not a formality.

Now the part that is this card and no other. The right-hand column is the system-memory bandwidth at which the two halves cost the same. It runs from 72.85 to 124.74 GB/s — and that interval contains most of the DDR5 configurations an RTX 5080 is actually installed alongside. So the answer is not "host-bound" or "VRAM-bound"; it is a context depth:

On the most ordinary configuration for this card — dual-channel DDR5-6000 — the binding half changes hands about a third of the way into the context window this recipe documents. Early in a conversation your DIMMs set the pace; past roughly 55,000 tokens the card's own memory does. That is the finding, and it is neither of the two answers this family of recipes has produced elsewhere: a 384-bit 12 GB board is host-bound at every depth, and a 128-bit 16 GB board is VRAM-bound at every depth. This one crosses over inside the page's own window. Bandwidth-driven — none of it transfers to another 16 GB card.

Two independent causes put it there and they push the same way: this recipe offloads seventeen blocks where a 12 GB card offloads twenty-six, so the host share is smaller; and this board reads its own memory over 256 bits of 30 Gbps GDDR7, which is 1.339× an RTX 4080 at the same width — both figures from the same whitepaper table — and about 3.33× an RTX 4060 Ti at half of it, that last comparison resting on a 288 GB/s figure NVIDIA does not publish and this catalogue takes from the tech press.

Priced out as a roofline — the two halves treated as serial, which they broadly are, since within a layer the CPU's expert matmul consumes what the GPU's router produced. Card side fixed at 3.57 ms per token (3,427,856,000 ÷ 960 GB/s), of which the KV cache alone is 1.49 ms:

⚠️ Those are ceilings from two bandwidth terms, not predictions. They exclude the CPU time to actually multiply the experts, kernel launch overhead, attention arithmetic, sampling and everything else, and they credit both memories with perfect efficiency. Read them as a bound you cannot beat, never as a rate you will reach. Every byte count in them is derived; none is measured on this pair, and if you measure one, tell us.

Two practical consequences, and the second is the one that surprises people.

Faster DIMMs are worth more here than on a slow 16 GB card and less than on a slow one with a narrow bus — moving DDR4-3200 to DDR5-6000 improves this card's 131K ceiling by 1.38×, against about 1.16× on a 128-bit 16 GB board, because here the host half is a larger fraction of a smaller total. That is the ordinary advice and it holds.

But context depth is still the bigger lever, and it gets bigger as your memory gets faster. Halving the window to 65,536 removes 713 MB of VRAM traffic per token — 0.74 ms of the 3.57 — and lets you keep -ncmoe 17 with a full gigabyte of extra slack. On a DDR5-6000 box that is worth more than the entire difference between DDR4-3200 and DDR5-8000 on the host side once the window is full. The offload depth is the smallest lever of the three: one step of -ncmoe adds 13.5 MiB (a cheap block) or 15.6 MiB (an expensive one) to the host read per token, which is 0.147–0.170 ms on DDR5-6000 against the 0.015–0.017 ms it saves on the card — so each step costs about 0.13–0.15 ms per token, net, and going deeper than the ladder requires is the one adjustment on this page with no upside at all.

Prefill is the phase the link pays for

At batch sizes of 32 or more the scheduler hands the expert matmuls to the GPU, which means copying host-resident expert tensors across PCIe. llama.cpp issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0", "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26" — and it finds "prefill is bottlenecked by PCIe transfers, not GPU compute", with the GPU idle around 42% of each pass on expert H2D copies that run serially with the matmuls consuming them. ⚠️ Carry its caveats: a different model, one box, gen-4-specific, and the author dates the measurement to "around the b6xxx era", so the idle fraction may be stale. It is the strongest evidence in circulation that the shape of this path is real, and it is not a measurement of Apodex. (Open with 8 comments, re-checked 2026-08-31.)

And this is the one place where this card's newer spec sheet buys something concrete. NVIDIA's spec table pairs "PCI Express Gen 5" with "Yes" for this board — that is a vendor fact, unlike the lane count. Gen 5 runs 32 GT/s per lane against Gen 4's 16, and both use 128b/130b encoding, so a lane carries about 3.938 GB/s against 1.969. If the whole offloaded set crosses per physical batch — the shape that report describes — then at -ncmoe 17 that is 8,323,596,288 bytes per -ub batch, and -ub defaults to 512 (common/common.h:452; -b defaults to 2048 at :451):

Theoretical link rates, with real achieved throughput lower. The useful reading is the middle two rows being identical: a Gen 5 ×8 slot gives you what a Gen 4 ×16 slot gives an Ada card, so on this board a halved link costs you a generation rather than a catastrophe. Bandwidth-driven, and specific to this card only in the sense that the generation is — the byte count is the tier's.

This page asserts no lane count of its own. Read yours:

nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.width.max,pcie.link.gen.current,pcie.link.width.current --format=csv

If gen.current reads 4 where gen.max reads 5, you are in a slot that halves the table above, silently — worth checking on this card specifically, because Gen 5 slots are newer than Gen 5 cards. Either way it is a time-to-first-token problem and not a tokens-per-second one, because at decode batch 1 those weights do not move. And if the link turns out to be your constraint, --no-op-offload disables the offload rule globally, keeping the expert matmuls on the CPU in both phases and removing the transfers entirely — a real trade, untested here, and one flag to try.

Three Blackwell-specific things in llama.cpp that are not about capacity

The VRAM budget on this page is the tier's. These three are not: each one is either gated on compute capability 12.0 or reported only on Blackwell silicon, so none of them can arise on the Ada and Ampere 16 GB boards this recipe otherwise shares its arithmetic with. All three were read at source or in the tracker on 2026-08-31.

1. Auto-fit can read your system RAM as if it were VRAM, which is why this recipe passes an explicit -ncmoe. llama.cpp issue #26901 — open, zero comments, filed 2026-08-11 — reports that ggml_backend_cuda_device_get_type() classifies a device as an integrated GPU when prop.integrated > 0, and that "Blackwell GPUs report prop.integrated > 0 because they support hardware-level unified memory access (a new Blackwell feature). This is not the same as being a true integrated GPU." Its headline consequence is multi-GPU, but the second one is not: the report states that "the UMA memory path reads system RAM (~77 GB on a 96 GB system) instead of actual VRAM (15.8 GB), causing common_params_fit_impl to believe a single GPU has enough memory for the entire model." Auto-fit is on by default — fit_params = true at common/common.h:476 — so on an affected build a reader who omits -ncmoe may get no offload chosen at all and a cudaMalloc failure at 19.908 GiB, on a card where the fitter would do the right thing if it could see the card. ⚠️ One open report, zero comments, no maintainer response, and it is Linux with two GPUs; whether a single-card box on a current build is affected is not established here. It is a cheap thing to check — if llama_memory_breakdown_print or the startup log reports tens of gigabytes free on a 16 GB card, that is the signature — and it is a reason to state your offload rather than delegate it that the Ada pages do not have.

2. The MoE routing kernel has an unfixed Blackwell crash path, and its mitigation is a driver rather than a build. llama.cpp issue #23385 — "Fatal MMQ crashes on Blackwell (RTX 5090/5080) due to unhandled sharedMemPerBlockOptin driver bug" — records early Blackwell drivers returning 0x100000001 or 0 for cudaDeviceProp.sharedMemPerBlockOptin, which llama.cpp copies straight into smpbo and uses to pick MMQ configurations. Its "Hardware" field is "RTX 5090 and RTX 5080 (Blackwell SM 12.0)" and its "Models" field is "Any model". The follow-up #27215 reproduces it on an RTX 5080 at driver 595.84 / CUDA 13.3 and pins down where it lands: the bad value is rejected by cudaFuncSetAttribute, and then "the next kernel launch that relies on the raised limit then fails the same way (observed here with ggml_cuda_launch_mm_ids_helper, the MoE expert id-routing kernel used by mul_mat_id — i.e. any MoE model)", with the fix verified across "three different quantized MoE GGUFs", the first of them "a MXFP4_MOE full model requiring --n-cpu-moe partial CPU offload". That is this card, this class of model and this flag. ⚠️ Read the status carefully, because it is the load-bearing part: #23385 is closed as not_planned with the stale label, and #27215 is closed and merged: false — the clamp is not in mainline (checked 2026-08-31). #27215's own author also notes that an isolated cudaGetDeviceProperties() call returned a sane value on the same machine and driver, so "this being intermittent/context-dependent rather than a fixed constant to special-case" — which means "upgrade past driver X" is not a claim this page can make either. Named drivers showing it are 590.48.01, 591.86 and 595.84; the two 2026-08 reports on this architecture at 610.47 and 610.62 do not mention it. If you meet an abort in mmq or mul_mat_id on a fresh install, a driver update is the first thing to try and the only thing anyone has to offer.

3. This card is on the other side of a codegen branch from every other consumer NVIDIA card in the catalogue — and what that is worth is unknown. At b10666, ggml/src/ggml-cuda/common.cuh:1649-1654 reads, above the comment "PDL and restrict need to be mutually exclusive":

# if (defined(GGML_CUDA_USE_PDL) && defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= GGML_CUDA_CC_HOPPER)
# define GGML_CUDA_RESTRICT
# else
# define GGML_CUDA_RESTRICT __restrict__

GGML_CUDA_CC_HOPPER is 900 and GGML_CUDA_CC_BLACKWELL is 1200 (:56, :59); GGML_CUDA_USE_PDL is defined for any non-HIP CUDA build from CUDART 12.3 (:118-121), which a 12.8-or-newer toolkit satisfies. This card is compute capability 12.0, so its device pass compiles these kernels without __restrict__, while every Turing (750), Ampere (860) and Ada (890) card in the catalogue keeps it.

⚠️ What that does not license. It is tempting to connect this to #25162, the 24–42% Turing regression attributed to "Avoid PDL race conditions by disabling __restrict__ when PDL is used" — and the connection does not survive being checked, twice over. First, reading common.cuh at the blamed commit 9e58d4d69 and at 0b14b87d7, the commit the reporter retested at, the guard is byte-identical to the one above at both: the >= GGML_CUDA_CC_HOPPER condition was there the whole time, so Turing's device pass never took the empty-__restrict__ branch at all. Second, and decisively, that regression is not about __restrict__ being absent. The reporter's own root-cause section says the commit "changed 14 CUDA kernel files, moving __restrict__ from function parameters to local variables", and that on a non-Hopper card — where the macro still expands to __restrict__ — "nvcc generates different (worse) code when restrict is on local variables vs function parameters." The regression is about where the qualifier sits, on a card that keeps it. On this card the question does not even arise in that form: at compute capability 12.0 the macro expands to nothing, so there is no qualifier in either position. So: the branch difference on this card is a verified fact about the source; no measurement anywhere connects it to throughput on sm_120, and this page makes no such claim. It is recorded because it is the kind of difference that gets assumed rather than checked, and because it means a Blackwell owner comparing their tokens-per-second against an Ada owner's is not comparing the same compiled kernels.

Does the arithmetic survive contact with a real card?

Nobody has run Apodex on an RTX 5080, so the honest answer is that this page is derived and the derivation has four partial checks, none of them on this pair.

The offload ladder has a measured analogue one tier down. gemma4-26b on an RTX 3060 is the only expert-offload run in our catalogue with a written-down ladder: -ncmoe 12 gave 37.23 tok/s at an 11,179 MiB peak and -ncmoe 30 gave 21.20 tok/s on the same unchanged card — eighteen more blocks on the host path costing 43% of the generation rate — with -ncmoe 8 and below failing to load outright. Different model, different card, DDR4-2133 dual channel (~34 GB/s), headless, llama.cpp b10088. Note that its host memory is slow enough to sit deep in the host-bound regime, which is exactly why its ladder is so steep and why this card's would not be as steep at the same depths. The direction is the point; none of those numbers is yours.

The prefill claim has the profiling report above, with the caveats stated there.

The offload path itself has a same-architecture confirmation. #26609 runs a qwen35moe model with experts pinned to CPU on an RTX 5070 — Blackwell, sm_120 — and its matrix includes a probe with all experts on the host. Everything in that report happens after a successful load.

And the 131,072-token window has a control run on this model's base architecture. llama.cpp issue #27756 is a long-context failure report against Qwen3.8-27B — a different 64-layer member of this family, 48 of whose layers are Gated-DeltaNet — which emits EOS as its first generated token beyond roughly 98–130k positions on CUDA, on CPU, and on a fork. Its reporter ran a cross-model control on the same build, same server and same machine: "The same needle test at 243k tokens PASSES on Qwen3.5-35B-A3B (30 GDN layers)" — which is exactly this model's base and exactly this model's thirty Gated-DeltaNet layers. ⚠️ It is one box, one control run reported in passing, a needle-recall test rather than a quality evaluation, and the base model rather than this fine-tune. It is not a guarantee. It is the only evidence found in the space searched (the llama.cpp tracker and the 22 model cards enumerated under Results) that this architecture's recurrent state holds up past the window this page documents, and it points the right way. (Open, 4 comments, re-checked 2026-08-31.)

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this GGUF, under llama.cpp, on any NVIDIA card, at any offload depth exists in the space searched on 2026-08-31. That space was: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces, re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100 (unchanged from yesterday's count, and one of the 22 — aykutx21/Apodex-1.1-mini-GGUF — holds nothing but a .gitattributes and has no card at all; a control query for Qwen3.5-35B-A3B on the same endpoint returns results, so the enumeration is not a broken call); the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 273 for qwen35moe and 729 for n-cpu-moe on the same query shape; and our own /check, which returns unknown with zero benchmarks. Figures for the model do exist, and every one is a different artifact on a different engine:

  • kingjones777 publishes four of his own repacks at 64.87 / 63.30 / 45.05 / 32.54 tok/s, in ROCmFP4/ROCmFPX tensor formats only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (gfx1151) unified-memory APU at full offload — a machine with no expert offload and no PCIe hop, which is the entire subject of this page. Read the top of that range as unsettled: that build's own card states the run is still pending on an idle box while the shared variants table prints 64.87 for it, under a caption reading "A dash means I haven't measured that one yet — I won't put a number in a card I didn't measure."
  • ghazni101 reports generation "measured at 226 tok/s end-to-end for MQ4R". Three disqualifications, all verified: it is a .mq4r file under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF; and it is an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim.

  ⚠️ And one trap in that first set, which is specific enough to this page to be worth naming. Three of kingjones777's cards also carry a partial-offload figure — 1.183 t/s on the COHERENT build, 0.292 on the plain ROCmFPX Q8_0 and 0.163 on the AGENT build. A reader who finds those and reads them as "expert offload gives you one token per second" would be wrong three times over. They are -ngl, not -ncmoe — 17/40 and 9/40 whole layers on the GPU with the rest CPU-mmapped, which is a different mechanism from this page's routed-expert offload and streams every layer's full weights rather than one thirty-second of them. They were taken on a box "serving 8 live llama-server seats holding ~107 GiB of unified memory", leaving about 16 GiB. Their own cards say so: "These t/s numbers are limited by streaming the CPU-resident layers, not by the ROCm path". And the author has retracted one of them outright — "My first published number (0.292 tok/s) was measured with PARTIAL offload on a box already serving 8 models — that was my harness's fault, not the model's" — re-measuring the same build at 45.05 tok/s on an idle machine at full offload. Note the coincidence that makes this worth spelling out here rather than elsewhere: that -ngl 17 is one character away from this recipe's -ncmoe 17 and means something entirely different.

  One measurement on this pair would be worth more than all of the above. If you run it, contribute it.

• VRAM usage: 13.546 GiB derived sub-total at -ncmoe 17 and 131,072 tokens, plus the ~1,109 MiB driver share and 1024 MiB compute reserve this page sizes against but cannot derive. 7.752 GiB of expert weights sit in system RAM alongside. See /check/apodex-1-1-mini/rtx-5080 for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: decode reads 0.26 GB from your DIMMs and 2.0–3.4 GB from the card per token. On dual-channel DDR5-6000 those two are equal at about 55,000 tokens of context, so the same machine is memory-bound early in a conversation and card-bound late in one — and the single largest thing you control is how deep the context runs, not how deep the offload goes. Prefill is where the host link is on the critical path, and PCIe Gen 5 is the one specification on this board's data sheet that measurably helps this recipe.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512), which is part of why the sub-4-bit rungs give up more than their file sizes suggest. The lead quant is imatrix-calibrated and the publisher ships both the calibration corpus and the imatrix beside the ladder, which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-5080.

Troubleshooting

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE, which arrived in (b7989, b7990] — so this means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

It builds and runs, but the first CUDA kernel fails — or the first run stalls for a minute

Suspect the toolkit before anything else, because on this card that is the failure the build does not warn you about. 120a-real is appended only at CUDA 12.8 and there is no 120-virtual in the list, so a non-native build on an older toolkit either has no code for your GPU or falls back to JIT-compiling 90-virtual PTX on first use. Check what you have and what you built:

nvcc --version | tail -2
strings ./build/bin/llama-server | grep -o 'sm_[0-9]*' | sort -u   # crude, but it should mention 120

Then re-read the build gate — in particular the SOURCE_DATE_EPOCH case, which is how a packaged or containerised build ends up without GGML_NATIVE and without you asking for it.

If it crashes with an illegal memory access on the second request

This is the hazard with the best claim on your attention, because the reported hardware is the closest to yours of anything in this page. llama.cpp issue #26609 — open, 4 comments, last updated 2026-08-29 — reports CUDA error: an illegal memory access was encountered in ggml_backend_cuda_synchronize, on an RTX 5070 12 GB (Blackwell, the same sm_120 target as this card), running a qwen35moe model with the routed experts of blocks 16–39 pinned to the CPU, -fa on and q8_0 KV. The report is unusually well made: a one-variable-at-a-time matrix that refutes slot reuse, prompt caching, KV quantisation, the offload pattern ("ALL experts to CPU" also crashes) and whole-layer offload, leaving one discriminant:

"--flash-attn off (which also forces KV f16 — declared double axis) passes 3/3 on both builds."

⚠️ Read four limits before you act on it. It is not a Blackwell-only report — a second commenter hits the same signature on an RTX 3070 Laptop under --cpu-moe, on Linux and Windows, which puts it in the qwen35moe + expert-offload + CUDA-FA family rather than in this card's. It is "Strongly input-dependent": the reporter's own scrambled copy of the triggering prompt passes on the configuration where the real one crashes deterministically, so a synthetic reproduction is not available. The last confirmed reproduction is b10488, which is 142 builds below this page's floor and 178 below its pin — nobody has tested it at b10630 or later, so this page can say neither "fixed" nor "still broken". And GGML_CUDA_DISABLE_GRAPHS=1 did not prevent it in that matrix.

If you do hit it, the workaround is not free on this card — and it costs in two places, only one of which is the cache. Both terms were read at this page's pin.

Term one, and it is smaller than it looks: -fa off sends only the V half back to f16, not the whole cache. The guard at src/llama-context.cpp:3675 tests ggml_is_quantized(params.type_v) alone, and refuses to create the context — "quantized V cache requires flash_attn to be enabled" — rather than silently degrading. The two K-side checks below it, at :3686 and :3697, are each gated flash_attn_type != LLAMA_FLASH_ATTN_TYPE_DISABLED, so with Flash Attention off they never run. -ctk q8_0 -ctv f16 is therefore legal, and it costs 10 × (512 × 34/32 + 512 × 2) = 15,680 bytes per token, not 20,480 — 1.914 GiB at a full window instead of 2.500.

Term two is the one that actually breaks the fallback, and it is not a cache at all. The non-Flash-Attention path builds the attention scores as an explicit tensor: src/llama-graph.cpp:2607 is ggml_tensor * kq = ggml_mul_mat(ctx0, k, q); and :2612 immediately calls ggml_mul_mat_set_prec(kq, GGML_PREC_F32), with the comment that the op "tends to require high floating point range". That tensor is n_kv × n_tokens × head_count at F32, this model's qwen35moe.attention.head_count is 16 (read from the GGUF header, where it is easy to miss — the round's own inspection script hides it behind an allowlist), and the graph is reserved at the worst case, n_tokens = min(n_ctx, n_ubatch). So it is a pure function of -c and -ub:

At this recipe's window and the default -ub 512, that single tensor is 4.000 GiB — four times the entire 1024 MiB this page reserves for compute. No -ncmoe setting rescues that, because it is not a weights problem. Anyone who turns Flash Attention off here must lower -ub in the same breath, and at 131,072 tokens -ub 128 consumes the whole reserve on its own, so -ub 64 is the setting that leaves room — still ≥ 32, so prefill expert staging continues as described above.

Put both terms together and the capacity side reads:

So there are two priced escapes and they are not equivalent. -ncmoe 19 with -ub 64 keeps your window, at 1.330 GiB more system RAM, about 0.4 ms per token of extra host read, and a physical batch eight times smaller — which slows prefill, the phase this card is otherwise good at. -c 65536 with -ub 128 keeps your offload depth, has the most capacity slack, leaves 512 MiB of the compute reserve unspent, and is the faster of the two at decode because it also removes 0.74 ms of VRAM traffic per token — but halves the context. -ncmoe 18 at the full window clears by 212 MiB, which is inside the range of the terms this budget cannot derive; prefer 19.

⚠️ The general lesson, because this page had it wrong in both directions before a reviewer caught it: "turn the flag off" is not a budget. The first version of this section priced the cache and not the graph, and priced the cache as if the whole thing went to f16. It overstated the KV by 0.586 GiB and omitted a 4.000 GiB tensor — errors of opposite sign, either of which alone would have sent a reader to a configuration that does not allocate. A flag's alternative needs every term it moves priced, not just the one the flag is named after. Capacity-driven arithmetic throughout; the choice between the two escapes is bandwidth-driven and belongs to this card.

If the whole GPU hangs during a long agent session (Xid 8)

This is the failure most likely to find you, because its trigger is the way this model is meant to be used. llama.cpp issue #27330 — open, 4 comments, filed 2026-08-18, last updated 2026-08-29 — reports that under sustained load the GPU stops servicing a command channel: "The NVIDIA kernel driver's Robust Channel watchdog declares the GPU locked after a 7 second notify timeout, raises Xid 8, and llama-server aborts inside ggml_abort with a CUDA error." The reproduction step is the part that matters here:

"Drive it with a real coding agent workload (I used OpenCode) so the slot stays busy with long prompts and long generations."

Apodex 1.1 mini is an agentic model. Long prompts and long generations on a busy slot are not an edge case for this page's reader; they are the whole reason to run it.

The workaround is one environment variable and the issue calls it complete: "Setting GGML_CUDA_DISABLE_GRAPHS=1 eliminates the failure completely. Nothing else needs to change."

GGML_CUDA_DISABLE_GRAPHS=1 ./build/bin/llama-server -m ./apodex-1.1-mini/... # ...the rest of the command above

⚠️ Set it to 1. Never write =0 meaning "leave graphs on". ggml/src/ggml-cuda/common.cuh:1258 reads getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr — presence, not value — so =0 switches CUDA graphs off exactly as thoroughly as =1 does. On most pages that is a curiosity. Here it is load-bearing in both directions: a reader now has a real reason to set the variable, which means they also have a real way to get it wrong, and a reader who wants graphs back must unset it rather than zero it.

Why this page treats it as a Blackwell matter rather than a general one. The thread spans five Blackwell boards and no others: the reporter's RTX 5090 Laptop (GB203M, compute capability 12.0, built with CMAKE_CUDA_ARCHITECTURES = 120), an RTX PRO 4000 Blackwell workstation card, two discrete RTX 5090s — and, on 2026-08-29, this exact card. That last report is the strongest single datapoint on this page, because it carries its own control:

"The failing GPU is always the 5080 (device 1, Blackwell); the 4090 (Ada) never reports an error"

— an RTX 4090 and an RTX 5080 in one machine under --tensor-split 3,1, at build b10665, one build below this page's pin. The same reporter adds that "The issue started after I replaced a RTX 4060 Ti 16G with a RTX 5080 (both as the second GPU next to the 4090); with the previous card this never occurred" — i.e. an in-box swap from the other 16 GB card in this family to this one, with the hang arriving alongside the card.

⚠️ Four limits, and none of them is small. Every report runs Qwen3.8-27B, whose architecture string is qwen35 rather than this model's qwen35moe — the same hybrid attention-plus-SSM family, not the same architecture. None of the reports is an expert-offload run: they are full-offload or tensor-split, so nothing here is conditioned on -ncmoe and nothing says this recipe is more or less exposed than a resident one. The hang is stochastic — "It never happens on the first request", with eleven observed hangs at 4 to 15 minutes from server start, mean 8.6. And the workaround's evidence is statistical rather than a fix: 50 minutes and then 32+ minutes clean on the same boot, which the reporter prices honestly as "the probability of surviving 32 minutes by chance is under 3 percent."

What the reporter ruled out is as useful as what they found, because it stops you chasing the wrong thing: not power (identical at 95 W and 175 W), not thermal (HW Thermal Slowdown counter zero), not a hardware fault (no Xid 79, 63 or 64, no ECC), not MTP (hangs with and without --spec-type draft-mtp), not kernel fusion (the decisive A/B row has fusion enabled and only graphs disabled, and is stable), and not KV quantisation (q8_0 on both sides). It is also not the same thing as the illegal-memory-access reports: this is a hang, the watchdog fires on a timeout and the hardware exception counters stay at zero, which is why its workaround differs from #26609's.

So: if you meet NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! and Xid 8 in dmesg — possibly alongside your X server hitting Xid 8 on its own channel, which the issue also documents — set GGML_CUDA_DISABLE_GRAPHS=1 and see whether it goes away. And if you are about to leave an agent loop running against this server for hours, setting it up front is a defensible trade: you give up CUDA graph optimisation, which this page cannot price for you, against a failure mode that takes the display down with it.

It loads, then runs out of memory when a long prompt lands

Two causes with opposite fixes.

The KV cache grows at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the ladder and either raise -ncmoe or lower -c. On this file, raising -ncmoe by two frees between 0.844 and 0.973 GiB depending on which blocks the steps land on.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means staging those tensors there. Each of this file's expert triples is 0.4219–0.4863 GiB, so prefill peaks materially above the generation footprint the budget table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those ops on the CPU where their weights already are; expect slower prefill in exchange. If the card is also driving a display, start one or two steps deeper than the table and read nvidia-smi after load.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it.

The first is the offload: at -ncmoe 17 this recipe puts 7.752 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache, on by default at 8192 MiB — cache_ram_mib = 8192 at common/common.h:632, exposed as -cram / --cache-ram in common/arg.cpp with -1 for no limit and 0 to disable. So a default server may hold up to 8 GiB beyond the offload — close to 16 GiB of host memory before the operating system gets any, which is why the command above pins it to 2.

There is one direct measurement of that growth on this architecture. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. Closed as completed 2026-08-28 (re-checked 2026-08-31). Two cautions if you cite it: the 126 MiB is the only measured per-entry figure anywhere and it is roughly twice one of this model's recurrent states (65,863,680 bytes = 62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support even though the 126 stands as a measurement; and it was measured on a different model of the same architecture, with a Vulkan rebuild giving +126.0 MiB per request against ROCm's +126.3 — so this is a model-and-request-layer behaviour and not a vendor story.

So 32 GB is the practical floor for this recipe as written and 16 GB is not enough. On a 32 GB box keep -cram 2048, or pass -cram 0 to disable caching entirely and trade prompt-reuse speed for headroom.

Generation is slower than you expected

Check three things, and on this card the order they matter in depends on how full your context is.

Your DIMMs, first, and here they matter more than on a narrow-bus 16 GB card. Single-channel memory doubles the host half of every decode step — 2.71 ms becomes 5.42 on DDR5-6000 — which at this recipe's window is a 1.43× loss overall. Confirm dual channel before anything else; it is the cheapest fix and on this board it is a real one.

Your context depth, second, and it is the largest lever once the window fills. At 131,072 tokens the KV cache is 1.328 GiB of VRAM traffic per generated token, larger than every weight on the card put together, and it is the term that hands the bottleneck from your DIMMs to your GPU somewhere around 55,000 tokens on DDR5-6000. Generation slowing down as a conversation grows is not degradation, it is the roofline. If your agent loop never fills a six-figure window, -c 65536 at -ncmoe 16 is a row in the ladder above and halves that term.

Your offload depth, third and smallest. Each step is 13.5–15.6 MiB more read from RAM per token, about 0.15 ms on dual-channel DDR5-6000 against 0.015 ms saved on the card. Do not go deeper than the ladder requires — on this card, unlike on a slow one, there is nothing to be gained from it.

If it is specifically the wait before the first token that is bad while generation is fine, that is the bus and not the model — see Prefill is the phase the link pays for, and check pcie.link.gen.current before assuming you have the Gen 5 link the card supports. And if you have GGML_CUDA_DISABLE_GRAPHS set anywhere in your environment and did not put it there for the Xid 8 hang, unset it rather than setting it to 0: the variable is read as getenv(...) != nullptr, presence and not value, so =0 switches CUDA graphs off exactly as thoroughly as =1 does. That cuts both ways on this card and the linked section says why.

Finally, if you are on Windows and running an official prebuilt binary, benchmark a source build before you conclude anything — see the measurement under Installation, which is the only sm_120-specific throughput comparison anyone has published.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true at common/common.h:476 — and moves MoE tensors to system memory until it can leave its 1024 MiB per-device margin. That is a reasonable path everywhere except here, and it carries three caveats on this card:

• It records its choice nowhere except the log, so you cannot reproduce a run from the command line alone.
• There is an open regression report against the margin flag on this architecture family — llama.cpp issue #27171, filed 2026-08-16 against Qwen3.6-35B-A3B Q4_K_M with --fit-target, still open and labelled bug-unconfirmed with 3 comments as of 2026-08-31, and its proposed fix (PR #27207) still unmerged.
• And the Blackwell-specific one: #26901 above reports the fitter reading system RAM instead of VRAM on this architecture. A fitter that believes a 16 GB card has 77 GB free will not offload anything.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line means your offload is being honoured, not that something broke.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp drives it with --spec-type draft-mtp — the exact spelling matters. On this card, price it first, because every cost lands on the 380.3 MiB of slack this page reserved:

• blk.40 stops being skipped: +0.444 GiB, and -ncmoe 17 does not touch it — the flag installs overrides for blk.0 through blk.16 only.
• The recurrent state is multiplied by the draft depth: need_n_rs_seq() at common/common.h:394 returns draft.n_max, so --spec-draft-n-max 3 takes 0.061 GiB to 0.184.
• A second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k — so at 131,072 tokens on one layer that is 2,048 bytes per token, 0.250 GiB.

That is 0.817 GiB against 0.371 GiB of slack, so on this configuration it does not fit at all; you would be raising -ncmoe to pay for it, before any throughput has arrived.

And the return is unknown. Unsloth's published guidance for this class of model is that "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs) — 1.15–1.25× at best, before offload is counted. Under offload even that is contested: an owner describing their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" running a different model of this architecture at --n-cpu-moe 25 with the MTP flags on reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" (discussion #18 on that model's GGUF repo), and a second community member there offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model.

⚠️ There is a crash report in this area, and note whose card it is on. llama.cpp #26558 records llama-server aborting with a cublasSgemm CUDA_ERROR_INVALID_VALUE under --spec-type draft-mtp with parallel load and a saturated KV cache — on an RTX 4090 built with CMAKE_CUDA_ARCHITECTURES=89, i.e. Ada, a different compute capability from this card, which is worth saying because the sibling pages for Ada boards can call it same-silicon and this one cannot. It was closed as completed on 2026-08-28 (11 comments, re-checked 2026-08-31) after the issue's own author reported the crash gone following a CUDA toolkit upgrade — a lapsed reproduction, not an identified fix. The reported condition is parallel load, which -np 1 is not, so it is a hazard to recognise rather than one this recipe is exposed to.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps's hardware table names "IQ2_M or IQ3_M" for a 16 GB card, with "IQ2_M full offload; IQ3_M may require hybrid offload". Weigh it against the rest of that card, which is candid: "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." and "Prefer Q4_K_M or higher when system memory permits."

Check the arithmetic before taking it. Against this page's 13.917 GiB ceiling, IQ2_M at 11.682 GiB in the lead repo leaves room for a 131,072-token q8_0 cache and the recurrent state with 0.846 GiB to spare, so it genuinely escapes offload — and it is a 2-bit quantisation of a model whose routed experts are 512 wide, used for tool calling where a malformed argument is a failed task. Q2_K_L at 12.650 GiB does the same at 65,536 tokens. On this card the trade is sharper than it looks in either direction: escaping offload removes the host half of the decode step entirely, which on DDR5-6000 at a full window is 2.71 of 6.28 ms — so a fully resident 2-bit build is genuinely faster, not just tidier, and it is still a large quality bet to avoid a memory purchase. Both — capacity-driven for whether it fits, bandwidth-driven for what it buys.

Choosing a different quant publisher

Conversions of this model are still appearing: an enumeration on 2026-08-31 returned 22 repositories matching Apodex-1.1-mini across 11 publishers, one of which first appeared on 2026-08-30. Re-enumerate before you decide, and check the byte count and the file list rather than the tier name — across publishers a file named Q4_K_M spans more than a gigabyte for the same nominal tier. Within bartowski's own ladder the rungs adjacent to the lead are Q4_K_S at 19.619 GiB and IQ4_XS at 17.955 GiB; each buys back host RAM rather than VRAM, which on this card is the resource you have most of and the one whose bandwidth you are usually waiting on.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, but they are vLLM/SGLang safetensors rather than GGUF and llama.cpp cannot load them at all. They are also far too large for this card before anything else is counted — 22.548 GiB and 22.958 GiB of weights for the first two, against 16 GiB — and the -NVFP4 repo is not a 4-bit checkpoint anyway: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4 (re-read 2026-08-31).

⚠️ Note what their Quick Starts actually launch, because the obvious reading is wrong. All three quantised repos launch on a single GPU: --tp-size 1 on -NVFP4 (line 82) and -GPTQ-Int4 (line 82), --tp 1 on -FP8 (line 81). The --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone (lines 80 and 83). Verified at source 2026-08-31. The reasons to skip these on this card are the format and the size, not an imagined eight-GPU deployment.

There is one thing worth knowing here that is specific to Blackwell and is a genuine capability rather than a caveat: llama.cpp's own build file annotates compute capability 120 as "Blackwell, needs CUDA v12.8, FP4 tensor cores", and this card has hardware FP4 that the Ada 16 GB boards do not. Nothing in the GGUF path this recipe uses spends it — the routed experts here are Q4_K and Q6_K, which are ggml block formats, not FP4 — so it is a capability with nothing to run on it today rather than an advantage this page can price.

Other pages on this exact card worth reading

As published on 2026-08-31, Qwen3.8-27B on RTX 5080 is the useful contrast: a dense model of similar size that fits entirely on the card, so it has no experts to skip and no host half at all. It is also where this catalogue's account of the seven NVIDIA 16 GB cards spanning 288–960 GB/s comes from, and it reaches the same 960 GB/s figure from the same whitepaper table this page read independently. LTX-2.5 on RTX 5080 works the same bus question from the video side. And if you have 32 GB and landed here by accident, Apodex 1.1 mini on RTX 5090, published 2026-08-28, runs the full 262,144-token window at Q5_K_M on the other Blackwell board with none of this page's arithmetic.

The model emits garbage, or stops immediately on a very long prompt

Two different things, and only the second is architecture-specific.

For uniform-probability gibberish, check which backend the report you found is about. The nearest to this recipe is issue #25857, filed 2026-07-18 on a Windows 12 GB NVIDIA box: every token at an identical log-probability at any offload setting. Read its scope first — it is a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins. Rebuild at a current release before inheriting the symptom. On this card, also rule out the driver-side sharedMemPerBlockOptin path in #23385 — it aborts rather than producing garbage, but it is the one failure class that is specifically about Blackwell and specifically about MoE routing.

For a completion that comes back empty with a clean HTTP 200 after a very long prompt, see issue #27756, which documents exactly that failure mode on a 64-layer sibling architecture beyond roughly 98–130k positions, reproduced on CUDA, on CPU and on a fork. As noted above, the same reporter's control run passed at 243k tokens on this model's 30-layer base architecture — so this window is not known to be affected, and that is the strongest statement the evidence supports rather than a guarantee. If you hit it, the signature is tokens_predicted: 1 with stop_type: "eos" and empty content, and it is worth telling us as well as the tracker.

And on measuring any of this, read the server's own counter rather than a stopwatch. llama.cpp #27623 carries an alarming headline about decode throughput collapsing ~25× at context on a hybrid Gated-DeltaNet model, and its own reporter retracted it: "My measurement metric was flawed. The numbers in my matrix were completion_tokens / total request time, which includes prompt processing" — which, he goes on, is why every configuration appeared to collapse to ~1.4 t/s when it was prefill time and not decode. That is precisely the error this page's configuration invites, because here prefill is bus-bound and slow while decode is not, so dividing generated tokens by wall time makes decode look broken when only prefill was slow. The issue is still open with the retraction as the last of six comments (re-checked 2026-08-31), so a reader searching their own symptom meets the headline first. Use llama-server's reported predicted_per_second.

For completeness on the issue that earlier writing about this model cites: llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label (re-verified 2026-08-31 against an authenticated client, with #27171 returning open as a positive control). Anything you read about it that was written before 2026-08-30 will call it open, including this site's own earlier pages for this model — that dates the text rather than changing the advice. It never applied to a text-only run, which is what --no-mmproj makes this one.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well — checked 2026-08-31, with ollama.com/library/qwen3.5 and the registry.ollama.ai manifest for qwen3 both returning 200 as controls. Use llama.cpp directly. If a tag appears later, please tell us.