Apodex 1.1 mini on RTX 5070: a 35B-A3B agent in 12 GB via expert offload on Blackwell

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one 12GB RTX 5070, at a 65,536-token context. The Q4_K_M weights are 19.908 GiB. The card holds 12 GiB. It runs anyway, because 91.24% of those weights are routed experts that llama.cpp can leave in system RAM.

Hardware data: RTX 5070 (12GB VRAM) · 8.891 GiB derived working set at -ncmoe 26, 65,536-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5070. /check/apodex-1-1-mini/rtx-5070 holds zero benchmarks, and so does every other card in our catalogue for this model. Every VRAM figure below is arithmetic over the GGUF's own tensor table and llama.cpp's allocation code, and it is shown so you can check it. There is no throughput number on this page at all — see Results for what exists and why none of it transfers. If you run this, please send the numbers.

🔴 Read this before you run anything. There is an open llama.cpp crash report against this architecture and this technique, and the one machine that reproduces it deterministically is this card. llama.cpp issue #26609 reports CUDA error: an illegal memory access was encountered in ggml_backend_cuda_synchronize, deterministically, on an NVIDIA GeForce RTX 5070 12 GB, serving a qwen35moe model under partial expert offload with --flash-attn on. That is the architecture string this GGUF declares and the offload technique this recipe uses, reproduced deterministically on this board. A second reporter in the same thread hits the same signature on an Ampere RTX 3070 Laptop 8 GB, so the defect is scoped to qwen35moe under partial expert offload rather than to Blackwell or to 12 GB — which makes it more likely to reach you, not less. This page keeps flash attention on, states why, and gives you the complete fallback with its arithmetic — because on a 12 GB card at this context the fallback is not a one-flag change, and doing it the obvious way fails to allocate. See An illegal memory access on the flash-attention path under Troubleshooting before a long session.

🧭 Two more things about this card specifically. First, the RTX 5070 is Blackwell, compute capability 12.0, and llama.cpp's CUDA build has a Blackwell-only branch that a 12 GB Ampere or Ada page cannot tell you about: below CUDA 12.8 there is no Blackwell device code at all, and a mismatched toolkit and runtime is itself a filed crash on this architecture. Second, everything about capacity here is the answer any 12 GB card gets, and everything about speed points away from the GPU: at -ncmoe 26 a decode step reads 0.367 GiB from system RAM against at most 2.670 GiB from VRAM at a full window, and the two halves take equal time only where your card's bandwidth is 7.28× your DIMMs'. That threshold is not a constant — it is 5.47× on an empty context and 9.09× at 131,072 — so it is derived here at three depths rather than quoted as a number.

ℹ️ This recipe is text-only. The checkpoint has a vision tower and the GGUF repo ships a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card contains no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it, against 24 occurrences of agent as a control (counted 2026-08-31). Note that the quant repo's own pipeline_tag is image-text-to-text, so the two disagree; this page follows the vendor. On a card this tight the projector is also 0.838 GiB you cannot spare, so the run command below refuses it explicitly rather than merely omitting the flag.

Requirements

Two version notes, because both have bitten this catalogue. The build tag moves several times a day — b10712 was published 2026-08-31T05:39:28Z, with b10710 and b10711 inside the previous ninety minutes — so this page pins b10666 and does not claim to be current; check the release list rather than trusting a date. And GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, not a bNNNN tag, because every build tag is marked prerelease; use /releases?per_page=N if you are scripting a version check.

Disk size and VRAM residency are different numbers here and they are not interchangeable. The file is 21,864,082,336 bytes — 20.363 GiB — on disk; a normal run loads 21,376,133,632 (19.908 GiB) of it. The gap is 487,948,704 B, and it is worth doing the subtraction properly rather than waving at the MTP head, because two things make it up:

The second row is simply 21,864,082,336 − 21,853,090,304, the difference between the file and the sum of its tensor payloads, which is by construction everything in the file that is not tensor data. It is 10.5 MiB and it changes no decision on this page — but subtracting only the MTP block from the file size does not reach the residency figure, and a budget built on a subtraction nobody has actually done is how a gigabyte goes missing somewhere else. Budget 22 GB of disk and 19.908 GiB of weights.

What NVIDIA publishes about this card, and what it does not. The vendor's own 50-series table gives the RTX 5070 as 6,144 CUDA cores, 2.51 GHz boost / 2.33 GHz base, 12 GB GDDR7, a 192-bit memory interface, 250 W total graphics power and a 650 W recommended system supply, with PCI Express Gen 5 listed under technology support (nvidia.com). Read that table carefully: it is two columns, and the left one is the RTX 5070 Ti (8,960 cores, 16 GB, 256-bit, 300 W). NVIDIA publishes no memory-bandwidth figure, no memory-speed figure and no PCIe lane count for this board — the only occurrence of the word "bandwidth" anywhere on that page is about display bandwidth — so this page states none of the three and does not compute a roofline from a number it cannot source. Read the link off the hardware instead:

# capacity, link generation and width, and the memory clock your board actually runs
nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.max,pcie.link.width.max,clocks.max.memory --format=csv

For the system side, your DIMMs' rated transfer rate gives it directly: dual-channel DDR4-3200 is 3200 MT/s × 8 B × 2 channels = 51.2 GB/s, dual-channel DDR5-6000 is 96 GB/s. That second number is the one that sets your generation rate on this recipe, and Where this card's speed actually goes shows why.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. Use b10630 or newer — the release the lead quant was produced with. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that makes blk.40 optional landed in the bracket (b10211, b10212], so b10630 is comfortably past it.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Now the part that is genuinely different on Blackwell, and it is not the same trap an Ada page has. GGML_CUDA_CC_BLACKWELL is 1200 in ggml/src/ggml-cuda/common.cuh at line 59. In ggml/src/ggml-cuda/CMakeLists.txt the fallback architecture list — the else() branch of the GGML_NATIVE test at :27, i.e. what you get when the native path does not apply — is built in five list(APPEND …) steps. The first, at :30-32, adds 50-virtual 61-virtual 70-virtual under CUDAToolkit_VERSION VERSION_LESS "13" and does not bear on this card; the other four do. The comment block at the top states the requirement outright — # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores:

list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real)          # :34, in every fallback build

if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real 90-virtual)                 # :36-38
endif()

if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real)                          # :40-52
endif()
if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.9")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 121a-real)                          # :53-55
endif()

Three consequences, in order of how much they cost you:

1. Below CUDA 12.8 there is no Blackwell code in that list at all. Your card runs by just-in-time compiling 90-virtual PTX (or 80-virtual, below 11.8). It works. It is not the code the project built for your silicon, and nothing in the log calls that out.
2. 120a-real is architecture-specific and real-only. The file's own comment at :75-79 explains why in one line — "12X is forwards-compatible, 12Xa is not." — and goes on to name the Blackwell FP4 tensor-core instructions as the reason. A foreach loop at :80-92 then rewrites every plain 12X in the list into 12Xa. There is no Blackwell virtual architecture shipped, deliberately.
3. The native path has a Blackwell-only failure mode, and llama.cpp works around it in-tree. GGML_NATIVE defaults ON, and with a toolkit ≥ 11.6 and CMake ≥ 3.24 it sets CMAKE_CUDA_ARCHITECTURES to native. The comment at :94-95 is explicit about what happens next on this card: a native build, it says, "will use the 12X architectures and fail". The rewrite loop above is what saves it. That is fine on a current tree; it is the reason an old llama.cpp checkout can fail to configure on a card that a newer one builds for happily.

Check which list you got — CMake prints it during configure:

nvcc --version | tail -2                       # toolkit >= 12.8?
cmake -B build -DGGML_CUDA=ON 2>&1 | grep CMAKE_CUDA_ARCHITECTURES

You want 120a or a native line resolving to it. A list that ends at 90-virtual means your toolkit is too old for this card and you are running JIT-compiled PTX.

And one more Blackwell build hazard, because it produces a crash rather than a slowdown. llama.cpp issue #25060 — open, bug-unconfirmed, five comments, last updated 2026-08-20 — is a SOFT_MAX failed crash on Blackwell parts where cudaFuncSetAttribute is fed a garbage shared-memory limit. Two commenters independently traced it to a toolkit/runtime mismatch rather than to the GPU. mangrove-one, on an RTX 5080 (SM 12.0), same machine and same driver, built one 15-line probe two ways:

4294967297 is 0x100000001 — bit 32 set on an otherwise-correct 1 — and llama.cpp then hands that to cudaFuncSetAttribute, which rejects it and leaves the limit at zero. github-pmunoz reported the same symptom traced to "CMake FIND_PACKAGE partially resolving the given local cuda 13.1 and mixing it with the system cuda 12.4 libcudart.so", found in CMakeCache.txt, and fixed it with one flag:

cmake -B build -DGGML_CUDA=ON -DCUDAToolkit_ROOT=/usr/local/cuda-13.1   # your path

⚠️ Scope it honestly. Neither report is an RTX 5070 — they are an RTX 5090, an RTX 5080 and an RTX 5070 Ti Laptop, all SM 12.0 — and the third of those (pmsudhi) probed the value directly and found it correct (101376) while still crashing, so the runtime-mismatch cause is well demonstrated but does not explain every instance. The issue body's own patch was, in its author's words, written by asking a model to write it; the two runtime probes below it are the load-bearing evidence. Treat this as a thing to check if SOFT_MAX fails, not as a claim that your build is broken.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes, imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z; the file name and size above were re-fetched from the tree API on 2026-08-31 rather than copied from a sibling page, and the download URL was HEAD-checked the same day (HTTP 200).

Do not start from the publisher's own quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card states plainly that "llama.cpp downloads the mmproj automatically when using" -hf. llama-server is in llama.cpp's mmproj_examples list (common/arg.cpp), no_mmproj defaults to false and mmproj_use_gpu defaults to true (common/common.h:594,596), so that command puts 899,283,488 bytes — 0.838 GiB — of vision projector on a card this recipe has already budgeted to within about a gigabyte. Download the file explicitly, as above, and load it with -m.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 26 \
    -np 1 \
    -c 65536 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Eight of those flags are decisions rather than defaults. Each was re-derived at b10666 for this page, and each one's default is quoted from the source at that pin:

• --no-mmproj refuses the vision projector rather than merely not asking for it — 0.838 GiB, for the reason given under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 26 keeps the routed experts of blocks 0–25 in system RAM. The flag's own help text at this pin is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp:2790), and the implementation matches the wording exactly — see How the offload actually works. 26 is derived in Choosing your own -ncmoe and again in Why 26 and not 24; it is not a measured optimum. Capacity-driven — the constraint is the 12 GiB, not the Blackwell silicon, so this number is the same on any 12 GB card.
• -np 1. llama-server sets params.n_parallel = -1 — "auto by default", common/arg.cpp:1400, with the help string at :2542 reading "number of server slots (default: %d, -1 = auto)". The Gated-DeltaNet recurrent state is allocated per sequence slot, so however many slots auto picks, the 62.8 MiB is multiplied by that number and the KV pool is divided among them. One slot is what a single-user agent loop wants. (This page does not state what auto resolves to at this pin; it did not need to, and asserting an unverified multiplier is how a number becomes wrong.) Capacity-driven.
• -ctk q8_0 -ctv q8_0 halves the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 65,536-token cache from 1.250 GiB to 0.664 GiB. Capacity-driven.
• -fa on makes Flash Attention explicit. The default at this pin is auto — flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO (common/common.h:499) — and pinning it means a future change to what auto decides cannot silently change your memory profile. 🔴 This is the flag with an open crash report against it on this exact card. Read An illegal memory access on the flash-attention path before a long session; the decision to keep it on is argued there rather than assumed here.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader itself warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp). Taking that advice means the 11.742 GiB of expert weights are ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure rather than a cache you can squeeze.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632, exposed as -cram/--cache-ram at common/arg.cpp:1713 — and it is more expensive on this architecture than on a plain transformer. See the host-RAM entry under Troubleshooting.

Sampling values are the model's own. temperature: 1.0, top_p: 0.95 and top_k: 20 are keys inside the GGUF header (general.sampling.temp, general.sampling.top_p, general.sampling.top_k, read directly from the file on 2026-08-31), and repetition_penalty: 1.05 comes from the model card. The card notes that Apodex "follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF and enables Jinja by default, so tool calls and <think> blocks parse the way the vendor intends. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It builds one regex per block and hands them to the model loader as buffer-type overrides (common/common.h:1130):

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, …) pushes "blk\.0" + regex … "blk\.(N-1)" + regex

The loader then does a plain std::regex_search on each tensor name and, on a match, allocates that tensor in a CPU buffer instead of the GPU one (src/llama-model-loader.cpp). Nothing in that path consults the architecture, so it applies to this hybrid recurrent model exactly as it does to a plain MoE. 123 tensors in this file match — three per block across all 41 blocks, counted by reading the GGUF's own tensor-info table — totalling 19,956,498,432 bytes (18.586 GiB), of which 19,503,513,600 (18.164 GiB) sit in the forty blocks a normal run loads. Against the 21,376,133,632 bytes of resident weight that is 91.24%.

What does not move is as important. The offload takes each block's ffn_down_exps, ffn_gate_exps and ffn_up_exps, and leaves on the GPU: the attention path, the recurrent state tensors (ssm_conv1d, ssm_a, ssm_alpha, ssm_beta, ssm_norm, ssm_out), the router (ffn_gate_inp), the shared expert (ffn_*_shexp) and every norm. So routing is decided on the GPU, the Gated-DeltaNet state stays on the GPU, and only the selected experts' matmuls happen elsewhere. Offload everything the regex can reach — -ncmoe 40 — and 1,872,620,032 bytes (1.744 GiB) is still resident.

That last word — elsewhere — means two different places depending on what you are doing, and it is the whole story of this recipe's performance. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights", per the scheduler's own comment (ggml/src/ggml-backend.cpp) — unless a higher-priority backend wants to offload it. CUDA does, and the test is one comparison at ggml/src/ggml-cuda/ggml-cuda.cu:5344 against op_offload_min_batch_size, which is getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(...) : 32 at :5515.

Generating a token is a batch of one, so decode reads the offloaded experts from system RAM and multiplies them on your CPU — they never cross the bus. Prompt processing is a batch of hundreds, so prefill copies those tensors across PCIe into a duplicate the graph allocator sizes for the run. A narrow slot is therefore a time-to-first-token problem, not a tokens-per-second problem, and this page treats the two separately throughout.

The VRAM budget

Weights, KV cache and recurrent state, at the recipe's settings.

The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 26 moves 12,608,077,824 of that (11.742 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks are loaded, and full_attention_interval 4 makes ten of them full-attention: blocks 3, 7, 11, 15, 19, 23, 27, 31, 35 and 39. Each carries head_count_kv 2 at key_length 256 and value_length 256 — 1,024 cache elements per token per layer, so 10,240 across ten layers. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows.
• Recurrent state. Those thirty blocks each allocate two F32 tensors, sized by src/llama-hparams.cpp: n_embd_r() at :204 is (ssm_d_conv − 1) × (ssm_d_inner + 2 × ssm_n_group × ssm_d_state) = 3 × (4096 + 4096) = 24,576 elements, and n_embd_s() at :232 is ssm_d_state × ssm_d_inner = 524,288. That is 2,195,456 bytes per layer per sequence and 65,863,680 — 62.8 MiB — across all thirty. llama-server prints it at startup as RS buffer size.

The two terms that table cannot produce, and what this page reserves for them

The 8.891 GiB is not a peak. Two things sit outside it, and on a 12 GB card they decide whether a configuration loads:

1. The card's usable pool is not 12 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. Our catalogue holds a measured expert-offload run in this VRAM tier with the numbers written down — gemma4-26b on an RTX 3060 at -ncmoe 12 — whose submitter recorded "That peak is 11,179 of the card's 12,288 MiB" on a headless box, adding that the roughly 1.1 GB spare "is why the lower rungs fail rather than merely slow down". That is a different GPU in the same tier and a single community submission stored at confidence 0.5, so read it as the order of the term and not as your board's exact reserve. This page sizes against ~11.0 GiB of usable pool, not 12.0.

There is one same-card reading, and it is consistent without being precise. The RTX 5070 report in #26609 notes its GPU was "active and verified via VRAM usage ~11.5 GiB" during a partial-offload run of a 22 GB model. That is an approximate figure with no tool named, taken on Windows where the WDDM driver can also spill into shared system memory, so it does not license raising the ceiling — it only says the conservative figure is not absurd.

2. The compute buffers. llama.cpp allocates working memory for the graph on top of everything above, and under expert offload it also stages the host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 452,984,832 or 522,190,848 bytes, so that staging term alone is of that order. This page reserves 1024 MiB. That number is llama.cpp's own default margin per device for auto-fit — fit_params_target is initialised to 1024 * 1024*1024 at common/common.h:481 — so it is the runtime's own opinion of how much room it wants, not a figure this page invented.

For scale on what happens if you leave nothing: llama.cpp issue #27698 records a run where the weights loaded, expert offload was honoured, and then context creation failed on a ~504 MiB compute buffer with cudaMalloc failed: out of memory. ⚠️ That report is not about this card and not even about this architecture, and it must not be reported as a qwen35moe observation. The issue has two numbered branches, and the one with the honoured offload and the compute-buffer OOM is its gemma4moe branch (gemma-4-26B-A4B), on an RX 6700 XT, gfx1031, RDNA2, on the ROCm backend, open with zero comments since 2026-08-25. Its qwen35moe branch fails differently — the loader ignores --n-gpu-layers/--n-cpu-moe entirely and segfaults on a full-model allocation. It is quoted here for one thing only: the magnitude of a term every table in this family of recipes excludes, and the shape of the failure it produces.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 11.0 GiB − 1.0 GiB reserve = 10.0 GiB.

At -ncmoe 26 and 65,536 tokens the sub-total is 8.891 GiB, which clears that by 1.109 GiB.

Choosing your own -ncmoe

Every row is weights + KV + recurrent state on a 12 GiB card, one slot. The first block of columns is this recipe's settings. The second is the flash-attention fallback in Troubleshooting, which forces the V half of the cache back to f16 — and a configuration that only fits with the fully quantised cache is a configuration you cannot fall back from. The third is a fully f16 cache, the conservative reading if you would rather not think about which half is quantised.

The -ncmoe 20 and 22 rows are the point of having a ceiling at all: against a naive 12 GiB they look like they have 0.781 and 1.624 GiB to spare, and against the ~11.0 GiB a card in this tier has actually been observed to give, they do not load. The bottom row is the model's entire declared window (context_length 262144) on a 12GB card, and it clears at q8_0 — at four more steps of offload than this recipe, which the speed section prices in milliseconds, and with no flash-attention fallback available at all.

Then stop guessing and measure it. llama.cpp ships a tool that answers this question on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments:
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device line splitting total / free / model / context / compute / unaccounted. That compute column is the number this page reserved 1024 MiB for, and unaccounted is your driver's share — the two terms the table above cannot produce. If you run this, those two figures are the most useful thing you could send us, more useful than a speed number.

Why 26 and not 24

The row above it fits. -ncmoe 24 at 65,536 tokens is 9.799 GiB and clears the ceiling by 0.201 GiB, and it puts two fewer blocks on the host path, which is the only lever this recipe has on generation speed. So it is a fair question why this page does not lead with it, and the answer is one specific fallback that has to stay available — the fallback for the open crash report on this card.

Turning flash attention off is what that report's own author found to work. It also changes the cache: llama.cpp rejects a quantised V cache without flash attention outright — "quantized V cache requires flash_attn to be enabled" at src/llama-context.cpp:3681, returning a null context rather than warning — while the equivalent check on the K half at :3686 runs only when flash attention is not disabled, so -ctk q8_0 -ctv f16 is legal. That is 5,440 + 10,240 = 15,680 bytes per token, which at 65,536 is 0.957 GiB against q8_0's 0.664 and a fully f16 cache's 1.250. Apply it to the rows:

• -ncmoe 24 at q8_0/f16: 10.092 GiB — 0.092 GiB over the ceiling, and 0.385 over with a fully f16 cache. The fallback is not available to you.
• -ncmoe 25: 9.671 GiB — clears by 0.329, but only by 0.036 GiB (37 MiB) if you take the simpler fully-f16 route, and 37 MiB is not a margin.
• -ncmoe 26: 9.184 GiB — clears by 0.816, and by 0.523 even fully f16.

So 26 is the shallowest offload depth at which this recipe survives its own workaround with room to spare, by either route. That fallback is not hypothetical here. #26609 is a report against this architecture string, this quant class and this offload technique — and one of its two reporting cards is this one — which is why it decides the setting rather than merely informing it. It is not a Blackwell defect or a 12 GB defect; the second reporter is on an 8 GB Ampere laptop part. Being architecture-scoped rather than card-scoped is what makes it bear on this recipe at all. The same arithmetic is why the 131,072-token row is not the default either: it fits at q8_0 and goes over at both fallback cache types, so a reader who takes the bigger window has quietly given up the fallback.

⚠️ And the KV cache is not the expensive half of that fallback. Turning flash attention off changes the graph, not just the cache, and on a 12 GB card the graph change is four times larger. The arithmetic is in What -fa off actually costs on a 12 GB card under Troubleshooting; read it before you reach for the flag, because the naive version of the fallback does not allocate at this context.

Where this card's speed actually goes

Here is the conclusion first: on this recipe the RTX 5070's advantage over a 12 GB Ampere card is confined to prompt processing, and even there it is partly spent waiting on the bus. Generation does not run on the GPU at all.

Decode. At batch 1 the CUDA offload test fails, the offloaded expert matmuls stay on the CPU, and their weights never cross the bus. So a decode step splits across two buses, and the split is computable. expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token — and attention re-reads the whole occupied KV cache from VRAM on every generated token, so that term belongs in the same table:

The VRAM rows are an upper bound, because the token-embedding table is read one row at a time while the output head is read in full, and this page has not separated the two in this file. That the bound points upward matters twice below.

The host side is therefore at least 12.08% of the bytes — 394,002,432 ÷ 3,261,000,192 — and that is a floor, not a ceiling, precisely because the VRAM term is an upper bound: separate the embedding table out and the host's share can only rise.

Now put the two on their buses. This page will not tell you the host is more than 12.08% of the time without a bandwidth figure it is not entitled to state — NVIDIA publishes none for this board. So here is the threshold instead, which needs none. The two halves of the decode roofline take equal time exactly when your card's bandwidth is 2,866,997,760 ÷ 394,002,432 = 7.28× your system RAM's. That threshold is a function of how deep your context is, not a constant, because the KV term grows with it and the host term does not:

Compute both sides on your own box — nvidia-smi for the card, your DIMMs' rated transfer rate for the host, both under Requirements — and compare against the row for the depth you actually work at. Two things follow whichever way it lands, and they are not the ones a spec sheet suggests:

• If you are past the threshold, decode is host-bound and gets more host-bound the faster the card is. A faster GPU cannot fix decode on this recipe, because decode was never on the card. DIMM bandwidth can.
• If you are near or below it, decode is VRAM-bound and faster DIMMs buy you less than you expect. A 192-bit board against dual-channel DDR5 is not obviously on either side of 7.28×, which is exactly why this page gives you the threshold rather than an answer. Do not assume the 12 GB tier has one answer: on a 128-bit board in this same round the crossover reached 13.18× at 131,072, above every DDR4 and DDR5 configuration considered, inverting the advice completely.

Whichever side you are on, note the direction: a session that fills its context drifts toward VRAM-bound, because the threshold rises with depth while your hardware does not move.

That gives a ceiling on the host term alone that does not depend on which GPU you own — divide the 0.367 GiB of host traffic by your memory bandwidth:

⚠️ Those are ceilings from one term, not predictions. They exclude the CPU time to actually multiply those experts, kernel launch latency, the VRAM side, and everything else. A measured 12 GB-tier expert-offload run in our own catalogue that states its memory bandwidth — gemma4-26b on an RTX 3060, -ncmoe 12, DDR4-2133 dual-channel, a different model — achieved 37.2 tok/s, comfortably below what a host-bandwidth roofline would allow it. Read the table as "you cannot beat this", never as "you will get this". A single-channel configuration halves every row.

The same record is also the cleanest available demonstration that the mechanism is real, on this exact VRAM tier: the submitter's ladder is -ncmoe 12 = 37.23 tok/s against -ncmoe 30 = 21.20 tok/s. Eighteen more blocks on the host path, on one unchanged card, cost (37.23 − 21.20) ÷ 37.23 = 43% of the generation rate. That is a different model, so the numbers are not yours; the direction and the size of the effect are the point.

Which makes the offload depth the lever, priced per step. Moving one block from card to host frees its whole expert triple from VRAM — 452,984,832 or 522,190,848 bytes — and adds one thirty-second of it to every token's host read: 14,155,776 B (13.5 MiB) for a cheap block, 16,318,464 B (15.6 MiB) for an expensive one, or about 0.28–0.32 ms per token each on dual-channel DDR4-3200. Going from this page's -ncmoe 26 to 30 for the full 262,144-token window adds four blocks and roughly 1.2 ms per token. Going the other way, to 24, buys back about 0.6 ms — and costs the flash-attention fallback, as the previous section shows.

⚠️ Treat that per-step arithmetic as a floor on the cost, not as a prediction of the curve. The only -ncmoe sweep on this architecture family run against stock llama.cpp that this page could find is a comment in the #25859 thread, and generation in it is not monotonic: at fixed batch and KV settings the reporter measures -ncmoe 22 → 36.75 t/s, 21 → 36.66, 20 → 42.18, 19 → 35.33, and notes that "The 20 -> 19 step alone drops generation by about 16%" while explicitly declining to claim a cause — "I did not profile PCIe transfers or GPU idle time. I am only reporting the reproducible behavior." They re-ran a fresh server at 20 and reproduced it, so it is not a one-off. That measurement does not transfer to this card and is not evidence about it: it is an RX 9070 XT 16 GB on the Vulkan backend, a different model (Qwen3.6-35B-A3B UD-Q4_K_M), different batch sizes and q4_0 KV — and Vulkan does not share the CUDA batch-32 offload test this page's whole decode argument rests on. What it does establish is that a smooth bytes-per-token model is the optimistic reading of this flag. Sweep two or three settings on your own box rather than trusting one step of arithmetic, and if you find a cliff on CUDA, that is worth reporting.

Prefill is the opposite case, and it is where this card's extra silicon has something to do. Issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04." running "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26, prefill at pp2048 / ub2048" — and it finds that "prefill is bottlenecked by PCIe transfers, not GPU compute", with GPU idle around 42% of each prefill pass waiting on serial expert-weight H2D copies that do not overlap the matmuls consuming them.

Take that seriously and it bounds what this card's extra compute can buy. NVIDIA's own tables make the RTX 5070 6,144 CUDA cores at 2.51 GHz against the RTX 3060's 3,584 at 1.78 GHz (50-series 5070 family, 30-series 3060 table) — a real gap in arithmetic throughput, and a generational one in tensor cores that no core count captures — but if roughly two fifths of prefill is the GPU waiting on copies that a faster GPU does not accelerate, then the compute advantage applies to the other three fifths only. The report's caveats limit this further and all of them matter: it is a different model, a single box, a custom fork used for A/B, gen4-specific, and the author says the measurement dates to "around the b6xxx era" and may be stale against current master. Treat it as the shape of the answer, not its value — the shape being that a newer card on this recipe is faster at the part that was already not the bottleneck.

Two things you can act on, both of which point away from the GPU: put your DIMMs in dual channel and buy memory bandwidth before you buy silicon; and if time-to-first-token specifically is bad while generation is fine, look at the slot, because prefill is the only phase where the expert weights cross the bus. NVIDIA lists this board as PCI Express Gen 5 and does not publish its lane count, so read it with nvidia-smi rather than assuming ×16.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true at common/common.h:476 — and moves MoE tensors to system memory until it can leave its 1024 MiB margin. That is a reasonable path, with two caveats: it records its choice nowhere except the log, so you cannot reproduce a run from the command line alone; and llama.cpp issue #27171 is open and unfixed against exactly this feature on exactly this architecture. It reports a --fit-target throughput regression measured with llama-bench on a qwen35moe 35B-A3B Q4_K_M — the same architecture string and quant class this page installs — of pp2048 1265.61 → 923.45 t/s and tg256 101.00 → 87.92 t/s across one commit, bisected to 9a688e51e, tagged b10284, whose subject is "fit: Fix memory allocation for MTP layers (#26605)". The open, unmerged PR #27207 that names it describes the problem as failing to "account for MTP/NextN layer positions in n_gpu_layers, even when their tensors are not loaded", with the consequence that "regular layers remain on the CPU and tensor override indices become misaligned". This model has exactly that shape — nextn_predict_layers is 1 and blk.40 is skipped by default — which is why this page pins -ncmoe explicitly rather than letting the fitter choose. Both issues were re-checked open on 2026-08-31.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line in the log means your offload is being honoured, not that something broke.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys you X GB" is wrong here. bartowski's imatrix recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28 and 31, then 34–39 — read out of the file's own tensor-info table, and checkable in one line, since 20 × 522,190,848 + 20 × 452,984,832 = 19,503,513,600 reproduces the offloadable total exactly. The top and bottom of the stack are all expensive and the middle alternates, so the first few steps of -ncmoe and the last few buy more room than the ones in between. Block 25 in particular is an expensive one, which is why the 25 row in the ladder above moves by 0.4863 GiB rather than 0.4219. Read the cumulative column rather than multiplying.

This is a property of the file, not of the model, and the same publisher's IQ4_XS is the control: 19,278,556,064 bytes on disk (17.955 GiB), 18,790,607,360 resident (17.500 GiB), and reading its tensor table the same way gives one expert-triple size across all forty loaded blocks — 427,819,008 bytes, 0.3984 GiB — with only the skipped blk.40 differing. Its ladder really is linear at 0.3984 GiB per step. If you would rather spend quality than throughput, that is the trade — but derive its table from its own tensor list, do not scale this one.

Does the arithmetic survive contact with a real card?

No one has run Apodex on an RTX 5070, and /check/apodex-1-1-mini/rtx-5070 returns unknown with zero benchmarks — as does every other pair for this model. The card itself is not unmeasured in our catalogue: it carries four benchmarks, for Qwen3 8B and Qwen3 14B, but both are dense models resident on the card, so neither says anything about an offloaded MoE. (All four store peak_vram_gb: 12.0, which is the board's capacity rather than an observed reading — two different models cannot both peak at exactly the card's size — so they also say nothing about the usable pool.)

The nearest test of the method is a different model on a 12 GB card with a figure at four offload depths: Qwen3.6-35B-A3B at UD-Q4_K_XL on an RTX 3060. That model is the same architecture shape as this one — 41 blocks, ten full-attention, identical ssm parameters and expert dimensions — so a weight ladder computed from its own tensor table is directly comparable:

⚠️ Read that right-hand column as GiB of weights, and do not read it as a peak. It tracks the derived weight ladder to within 0.16 GiB across a 9 GiB range. Read as decimal GB, every point falls below the weights that setting must load, which is impossible. And a figure that tracks the weights cannot also contain the KV cache and the compute buffers: at the top rung, 11.7 means the interval [11.65, 11.75] GiB against 11.542 GiB of derived weights, leaving at most 0.208 GiB for everything else — while that run's f16 KV cache alone was 0.168 GiB at its 8,832-token bench depth, plus 0.061 of recurrent state. The excluded terms do not fit in what remains, so those numbers are floors, not peaks. Our own RTX 3060 recipe for that model built its context table on the 9.8 figure read as a measured peak when this page was written; that reading does not survive the arithmetic above, and no number from it appears anywhere on this page.

What the comparison does establish is the shape of the ladder, at four settings spanning 9 GiB. It is not four independent measurements — one submitter, one rig, one model, one tool — so every systematic error the method could have is shared across all four points.

Two other pages on this exact card worth reading first

gpt-oss 20B on RTX 5070 is the same card and the same technique on a much smaller model, and it is the cheaper way to find out whether your box's CPU and RAM are up to an offloaded MoE at all — build llama.cpp, run that, then come back. Ornith 1.0 9B on RTX 5070 is the agentic-workload comparison that fits entirely on the card, which is the honest alternative if the offload arithmetic here reads as too tight for you. If instead you have 24 GB and landed here by accident, Apodex 1.1 mini on RTX 3090 fits the whole model on the card with no offload and none of this page's arithmetic applies.

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this file, under llama.cpp, on any NVIDIA card, at any offload depth exists in the space I searched on 2026-08-31, and the space was: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces (re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100 — the count was 21/10 two days ago, so re-run it rather than trusting this sentence; 21 cards were fetched and read, and aykutx21/Apodex-1.1-mini-GGUF has no README.md at all); the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 273 for qwen35moe and 729 for n-cpu-moe on the same query shape; and our own /check, where this model has zero benchmarks on all three cards it is linked to. Nothing in those 21 cards mentions this card at all — 5070, Blackwell, GB205, sm_120 and RTX 50 match zero files, where the same shape of query for 7900, gfx1100 and 395 matches five. Figures for the model do exist and every one of them is a different artifact on a different engine:

  • kingjones777 publishes a table covering four of his own repacks — 64.87 / 63.30 / 45.05 / 32.54 tok/s — in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (gfx1151), which is not a catalogue card. ⚠️ The 64.87 belongs to STRIX_LEAN, the one repo of the four whose own card carries no measurement paragraph: its hand-written speed section shows a dash beside "Full-offload speed being measured on an idle box, card will be updated".
  • ghazni101 reports "generation measured at 226 tok/s end-to-end for MQ4R". Three disqualifications, each sufficient: it is a .mq4r file under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF; and it is an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim. (The repo ships both an mq4r and an mq4rp file, so the figure does measure something the repo hosts; the "~10% RP decode penalty" sentence beside it is a prediction about the other one. The two other numbers on that card, ~235 and ~212 tok/s, are not Apodex at all: the card introduces them as the Ornith reference builds, and says those "builds of the same recipes measured ~235 tok/s (MQ4R) vs ~212 tok/s (MQ4RP)" — a different model.)

  One figure on this pair would be worth more than all of the above. If you run this, contribute it.

• VRAM usage: 8.891 GiB derived sub-total at -ncmoe 26 and a 65,536-token context, plus the compute buffers and driver share this page reserves 1024 MiB and ~1 GiB for respectively but cannot derive. See /check/apodex-1-1-mini/rtx-5070 for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: with 26 of 40 blocks' experts on the host, decode is paced by your CPU and system-memory bandwidth and not by this card — see the crossover table above, work out which side of it your box sits on, and treat the ceiling table as a ceiling. Prefill is where the RTX 5070's compute has something to do, and even there a profiled run of this shape spent about two fifths of the time waiting on the bus.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated — the GGUF header names its own calibration set (quantize.imatrix.dataset, 590 chunks, 510 entries) — which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. Note that the publisher stores the MTP layers at Q4_0 in the imatrix quants — "since imatrix calibration does not exercise them", per their own card, and the file agrees: every tensor in blk.40 is Q4_0 or F32 — a detail that only matters if you turn the MTP head on. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-5070.

Troubleshooting

An illegal memory access on the flash-attention path, under expert offload

llama.cpp issue #26609 — open, four comments, opened 2026-08-05, record last updated 2026-08-29 (the most recent comment is 2026-08-19) — reports CUDA error: an illegal memory access was encountered in ggml_backend_cuda_synchronize (cudaStreamSynchronize). The reporter's hardware block is "Windows 11 Pro, NVIDIA GeForce RTX 5070 12 GB, driver 610.47". The model is Qwen3.6-35B-A3B-UD-Q4_K_M.gguf, arch qwen35moe, and the server args are --ctx-size 32768 --parallel 1 --n-gpu-layers 99 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0, with experts of layers 16–39 pushed to the CPU by --override-tensor because "model is 22 GB, VRAM is 12 GB". Line for line, that is this recipe on a different checkpoint.

It names the RTX 5070 by model, runs qwen35moe by architecture string, and uses partial expert offload by technique — all three. Of the 104 issues in that tracker matching the phrase "RTX 5070" (a query polluted by matches on the issue number 5070, so read that count as an upper bound on what is relevant), it is the only one I found that does, which is why it leads this section rather than sitting at the bottom of it. That is a statement about the tracker, not about the defect: what the bug is actually scoped to is a separate question, and item 4 below answers it — not this card.

What is established. The crash is deterministic — reproduced 5+ times, on the second request of a fixed two-request sequence — and it survives a one-variable-at-a-time matrix that rules out most of the obvious suspects: --slot-prompt-similarity 0, cache_prompt: false, sending all experts to the CPU, whole-layer offload instead of tensor overrides, f16 KV with flash attention still on, grammar-constrained output, GGML_CUDA_DISABLE_GRAPHS=1, and context size (32,768 and 131,072 behave the same). CPU-only inference never crashes. --flash-attn off is the discriminant: the follow-up matrix of 2026-08-18 records it as "passes 3/3 on both builds", and the original report adds that the same arm completed "an 87-request campaign" cleanly.

The reporter flags -fa off as a "declared double axis" because it also forces the V cache to f16 — and their own probe F closes that gap: --cache-type-k/v f16 with flash attention still on crashes. So the cache type is not the variable; flash attention is.

Read the scope before you inherit the symptom, because four things limit it:

1. It is a different model — Qwen3.6-35B-A3B UD-Q4_K_M, and a different Qwen 3.6 35B MoE finetune in the second reporter's case. It is the same architecture string as this file, which is a closer match than most such reports, but it is not Apodex.
2. It is strongly input-dependent, and the reporter is explicit about how strongly: a synthetic prompt of the same length and block structure passes, and so does "a shape-preserving scramble of the triggering prompt (every alphanumeric word replaced by a random same-class, same-length word, punctuation/whitespace untouched)", on the exact configuration where the real prompt crashes deterministically. Most prompts do not trip it, which is also why no public reproducer is attached.
3. The newest build it is confirmed on is b10488 (9d77fa172, verified as that tag's commit), which is older than the b10630 this page states as a floor and the b10666 it pins. The reporter's own wording is "not fixed as of that build". Nobody has retested at b10630 or later, so "still broken today" is not established either way.
4. The second reporter's hardware is an RTX 3070 Laptop GPU 8 GB on Linux, and describes it as "spottily in a way I can't make deterministic" — a weaker, different observation. They first reported koboldcpp as immune and then retracted that two days later.

What this page does, and why. It keeps -fa on. Three reasons, and you should weigh them against your own workload rather than take them:

• The trigger is a specific token sequence, not a configuration. Most sessions never hit it.
• The confirmed-crashing builds are all older than this page's floor. That is not evidence of a fix, but it does mean nobody has tested the configuration this page actually recommends.
• On a 12 GB card at 65,536 tokens the fallback is not free and not a one-flag change — see the next section. A reader who adds -fa off and nothing else will fail to allocate, and will reasonably conclude the recipe is broken.

What would change it: a reproduction on a b10630-or-newer build, or any report of this crash against Apodex specifically. If you are running an unattended agent loop where a server death costs you a task rather than a retry, invert the default and start from the fallback below — the cost is bounded and this page's -ncmoe 26 was chosen so that it fits.

One counter-observation, offered for scope rather than reassurance: our own gemma4-26b run on an RTX 3060 is a 12 GB expert-offload run with "flash-attn on" recorded in its notes and no crash reported — but that is a gemma4 model, not qwen35moe, and #26609 is specific to the latter.

What -fa off actually costs on a 12 GB card

Not 0.293 GiB of KV cache. About 2 GiB of compute buffer, and it is the reason the naive fallback fails to allocate at this recipe's context.

With flash attention on, llm_graph_context::build_attn_mha calls ggml_flash_attn_ext and the attention scores are never materialised. With it off, the same function at :2607 builds ggml_tensor * kq = ggml_mul_mat(ctx0, k, q) and immediately pins its precision with ggml_mul_mat_set_prec(kq, GGML_PREC_F32), under a comment reading "note: this op tends to require high floating point range". That tensor's shape is [n_kv, n_tokens, n_head] at 4 bytes an element, and this model's attention.head_count is 16 (read from the GGUF header). The worst-case graph is reserved at n_tokens = std::min(cparams.n_ctx, cparams.n_ubatch) against the full cache (src/llama-context.cpp:595,609), so it sizes to your whole -c, not to how much of it you have filled:

Every cell is n_kv × n_ubatch × 16 × 4 bytes, so they are exact rather than estimated. This page reserves 1024 MiB for the whole compute buffer, and at -c 65536 -ub 512 this one tensor is twice that reserve on its own — before the mask, the softmax result, the expert staging buffers and everything else in the graph. That is why the fallback is three flags:

# the #26609 workaround, sized so it allocates on a 12 GB card
./build/bin/llama-server -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj -ngl 99 -ncmoe 26 -np 1 -c 65536 \
    -fa off -ctk q8_0 -ctv f16 -ub 128 \
    --load-mode none -cram 2048 \
    --host 127.0.0.1 --port 8080

• -fa off is the discriminant from #26609.
• -ctv f16 is not optional: llama.cpp refuses a quantised V cache without flash attention outright ("quantized V cache requires flash_attn to be enabled", src/llama-context.cpp:3681) and returns a null context rather than a warning. -ctk q8_0 is still legal — the block-size check on the K half at :3686 only runs when flash attention is enabled — so keep it and pay 0.957 GiB instead of 1.250.
• -ub 128 brings the kq tensor to 0.500 GiB, inside the 1024 MiB this page already reserves. Lower it further if the log still shows a compute buffer you cannot afford; raise it only after reading the table above.

Total: 9.184 GiB of weights + KV + recurrent state, clearing the 10.0 GiB ceiling by 0.816, with the largest single graph tensor accounted for inside the reserve. Expect slower prefill — a smaller micro-batch means more passes — which is the trade you are making for a server that does not die mid-session.

One corroboration worth noting, because it is the issue's own data. Every passing -fa off arm in #26609 is at --ctx-size 32768; the 131,072 arms were all run with flash attention on. At 32,768 and the reporter's --ubatch-size 512 the kq tensor is 1.000 GiB rather than 2.000 — the top-left cell of the table above. Their working configuration is consistent with this arithmetic, which is a mild check on it rather than a confirmation, since they did not report a memory breakdown.

If turning flash attention off makes the crash go away for you, you have reproduced #26609's discriminant on a new model and a new build, and that is worth telling both us and the issue.

SOFT_MAX failed on a Blackwell build

If you take the -fa off route above, this is the failure mode to know about, because that route is the one that puts a softmax kernel in the attention path at all. See Build llama.cpp with CUDA for the full treatment of #25060; the short version is that on Blackwell parts a mismatched CUDA toolkit and runtime makes cudaGetDeviceProperties return 4294967297 for sharedMemPerBlockOptin, llama.cpp feeds that to cudaFuncSetAttribute, the call is rejected, and any kernel asking for shared memory dies. Check CMakeCache.txt for two different CUDA paths and rebuild with -DCUDAToolkit_ROOT= pointed at one of them. Note that one commenter on an RTX 5070 Ti Laptop probed the value, found it correct, and still crashed — so a clean probe does not clear you.

Out of memory once a long prompt lands, but not at load

Two different causes, and they need opposite fixes.

The KV cache grows with the prompt at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the table above and either raise -ncmoe or lower -c.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means copying those tensors into a duplicate the graph allocator sizes for the run. Each of this file's expert triples is 452,984,832–522,190,848 bytes, so prefill peaks materially above the generation footprint the table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those operations on the CPU where their weights already are; expect slower prefill in exchange, and note that this is the one place a wider slot was helping you.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it, not one.

The first is the offload itself: at -ncmoe 26 this recipe puts 11.742 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second is the one that catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache and it is on by default at 8192 MiB — cache_ram_mib = 8192 in common/common.h:632, exposed as -cram / --cache-ram at common/arg.cpp:1713 with -1 for no limit and 0 to disable (added by PR #16391, "server : host-memory prompt caching", which the flag's own help text links). So a default llama-server may hold up to 8 GiB of RAM beyond the offload — with this recipe's 11.742 GiB that is close to 20 GiB of host memory before the operating system gets any, which is why the command above pins it lower.

And an entry here has a floor. Saving a prompt calls llama_state_seq_get_data_ext on the whole sequence state (tools/server/server-context.cpp), and on a hybrid model llama_memory_hybrid::state_write writes both children — the attention cells and the recurrent memory, the latter unconditionally. So every cached prompt carries its tokens' KV at your -ctk/-ctv type plus a full copy of the 62.8 MiB recurrent state, whatever its length; the fixed copy is the larger half of an entry until 65,863,680 ÷ 10,880 ≈ 6,054 tokens, where the per-token half overtakes it.

There is one direct measurement of this growth and it is worth reading before you size your box. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. It is closed as completed, 2026-08-28, re-checked 2026-08-31. Two cautions if you cite it yourself: the 126 MiB is the only measured per-entry figure this round found on any surface, and it is roughly twice one recurrent state (62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support even though the 126 stands as a measurement; and it was measured on a different model of the same architecture and on ROCm — the reporter rebuilt with -DGGML_VULKAN=ON and got +126.0 MiB per request against ROCm's +126.3, so this is a model-and-request-layer behaviour and not a vendor story.

So: 32GB is the practical floor for this recipe as written, and 16GB is not enough. On a 32 GB box keep -cram at 2048 as above, or pass -cram 0 to turn caching off entirely and trade prompt-reuse speed for headroom. Lowering -ncmoe moves weight back onto the card and off the host, and the one lever that helps both sides at once is a smaller quant.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE. Check llama-server --version against the release list and move to b10630 or newer. On this card, check the CUDA toolkit in the same pass — see Build llama.cpp with CUDA, where 12.8 is the floor for emitting Blackwell device code at all.

And note what a build floor buys you here beyond "the model loads". llama.cpp issue #25162 — "Performance regression on Turing GPUs after 9e58d4d69 — Qwen35 SSM kernels affected", closed as completed 2026-08-03 — recorded a 24–42% loss in ssm-scan.cu / ssm-conv.cu, the kernels behind 30 of this model's 40 layers, while "Gemma4 (no SSM layers) is unaffected". The commit's subject is "Avoid PDL race conditions by disabling __restrict__ when PDL is used" — but read the issue's own root-cause section before inferring what changed on a non-PDL card. It says 9e58d4d69 moved the qualifier "from function parameters to local variables" across fourteen CUDA files, and that "nvcc generates different (worse) code when restrict is on local variables vs function parameters". On Turing the keyword was never removed — the macro still resolves to __restrict__ there — it moved.

And what is known about the recovery is thinner than it looks. The issue closed on the reporter's own one-line retest at 0b14b87d7, a commit whose subject is a server port-change notice rather than a fix, and the thread never identifies a cause; the reporter's attempt to put the qualifier back on function parameters "did not restore performance". So the honest statement is: performance had recovered by the tree of 2026-08-03, cause unidentified in the thread. b10630 is well past that tree, which is the reason to be on a current build — not evidence that a fix landed.

⚠️ On this card, read that branch carefully, because Blackwell sits on the other side of it from Turing. At b10666, ggml/src/ggml-cuda/common.cuh:1650-1654 — sitting under a comment at :1649 reading "PDL and restrict need to be mutually exclusive" — expands GGML_CUDA_RESTRICT to nothing when GGML_CUDA_USE_PDL && __CUDA_ARCH__ >= GGML_CUDA_CC_HOPPER, and to __restrict__ otherwise. GGML_CUDA_CC_HOPPER is 900 and GGML_CUDA_CC_BLACKWELL is 1200, so:

• Established: on a Blackwell build those SSM kernels compile without __restrict__, while Turing (750), Ampere (860) and Ada (890) keep it. Note that this is a different change from the one Turing measured — there the keyword stayed and only moved from parameters to locals. That arch guard is not new either: it is already present, byte for byte, in the blamed commit itself, whose own third bullet reads "Add hopper to PDL restrict fix".
• Not established: what that costs on Blackwell, or whether it costs anything. The trade is deliberate: this is the generation that actually gets programmatic dependent launch in exchange, which Turing did not, and that is why the guard exists. No Blackwell measurement of it appears in #25162 or its thread, and I found none elsewhere in that tracker. Do not read this as a claim that this card is slow on those kernels, and do not upgrade the Turing reporter's closing request into a Blackwell measurement. If you own this card and a Turing or Ampere one, a llama-bench comparison on this file would be genuinely new and we would like to see it.

The model emits !!!! or uniform-probability gibberish

Searching the llama.cpp tracker for this architecture string turns up several gibberish-shaped reports, and the first thing to do with any of them is check which backend it is about. The nearest one to this recipe is issue #25857, filed 2026-07-18 on a Windows 12GB NVIDIA box: every token at an identical log-probability of −12.42 — the uniform distribution over the vocabulary — at any offload setting including pure CPU. Read its scope before you inherit the symptom. It is against a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins; it remains open with the stale label and zero comments as of 2026-08-31; and a separate publisher reports coherent output from an Apodex Q4_K_M on an older build still (vcruz305 smoke-verified -ngl 80 on b9835, on an RTX 6000 24GB). The other reports in that neighbourhood are on other backends — #24168 is SYCL on an Intel Arc card, #23321 is Vulkan with --no-kv-offload and is closed — so neither describes a CUDA build. Rebuild at a current release first.

A different failure worth knowing about at depth, because this recipe's window is a third of the model's declared one: llama.cpp #27756 — open, four comments, re-checked 2026-08-31 — reports "silent instant-EOS beyond ~130k context" on a 64-layer Qwen3.5-hybrid. Its cross-model control is what makes it worth citing here rather than alarming: "The same needle test at 243k tokens PASSES on Qwen3.5-35B-A3B (30 GDN layers) on the same build, same server, same machine". That is this model's exact base and exact Gated-DeltaNet layer count. The reporter's hypothesis is a depth × linear-layer-count effect and they note "The failure is not monotonic in depth", so read it as one box's evidence that the 30-layer shape holds well past this recipe's 65,536 rather than as a guarantee.

For completeness on the one issue this catalogue's older Apodex pages point at: llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label. Anything you read about it that was written before that date will call it open, including this site's own earlier pages for this model — that dates the text rather than changing the advice. It also never applied to a text-only run, which is what --no-mmproj makes this one. It is a separate report from #26609 above, which is about the flash-attention path on text requests.

Throughput far below what your CPU should manage

Check three things before suspecting the model, and note that none of them is the GPU. -ncmoe too high pushes more blocks onto the host path than you need — each step is 13.5–15.6 MiB more read from RAM per token, about 0.3 ms on dual-channel DDR4-3200. Single-channel system memory halves the bandwidth those reads get, which halves the ceiling table above. And if you are using auto-fit rather than an explicit -ncmoe, pin the offload yourself rather than passing --fit-target, because of issue #27171.

If it is specifically the wait before the first token that is bad while generation itself is fine, look at the slot rather than the model: prefill is the only phase where the expert weights cross the bus, so a card in a chipset-fed ×4 slot or on a riser pays there and nowhere else.

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

⚠️ If you have set GGML_CUDA_DISABLE_GRAPHS while debugging, unset it rather than setting it to 0. At b10666 the check is getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr (common.cuh:1258) — presence, not value — so =0 disables CUDA graphs exactly as thoroughly as =1 does. Note that #26609's reporter tried GGML_CUDA_DISABLE_GRAPHS=1 against their crash and it did not help, with the caveat that their build printed no acknowledgement that the variable was honoured.

Should you turn on the MTP head?

On this card, the short answer is no, and unlike every sibling page this one has a card-specific reason.

llama.cpp issue #25318 — open, one comment, bug-unconfirmed, re-checked 2026-08-31 — is titled "RTX 5070 CUDA drivers crash with MTP" and its hardware block reads "GPU: NVIDIA GeForce RTX 5070 (Blackwell, 12 GB)", Ubuntu 24.04, driver 595.71.05, CUDA 13.2. Running an MTP GGUF with --spec-type draft-mtp --spec-draft-n-max 6, prompt processing proceeds normally and then CUDA error: unspecified launch failure arrives in ggml_backend_cuda_synchronize(), after which the kernel log carries NVRM: GPU lost from the bus [NV_ERR_GPU_IS_LOST], the fan ramps to 100%, nvidia-smi stops responding, and "The GPU cannot be recovered without rebooting the system." The reporter notes the same machine runs non-MTP models without trouble. One commenter reports the same on an RTX 5090D.

⚠️ Scope it before you treat it as settled. The model is a 9B (Qwythos-9B-Claude-Mythos-5-1M-MTP-Q6_K), not qwen35moe; the build is version 9873 (a4107133a), far below this page's b10630 floor; the command also carries --fit on, -c 200000 and q4_1 KV, so the variable is not isolated the way #26609's is. It is one report plus one confirmation. What makes it worth acting on anyway is the failure mode: an unrecoverable GPU on the card this page is about is a different risk class from a server that exits.

The cost side argues the same way even if that report is unrelated to this model, because MTP lands on the scarcest resource you have:

• blk.40 stops being skipped: +476,956,672 bytes (0.444 GiB), and note that -ncmoe 26 does not touch it. The flag installs overrides for blk.0 through blk.25; only -ncmoe 41 would reach block 40's experts.
• The recurrent state is multiplied by the draft depth, and it is worth reading the two source lines together rather than one. need_n_rs_seq() at :394 returns draft.n_max, and src/llama-memory-recurrent.cpp:101 then sizes the allocation as mem_size * (1 + n_rs_seq) — with src/llama-memory-recurrent.h:73 saying so in words: "tensors are widened to (1 + n_rs_seq) groups". So --spec-draft-n-max 3 gives four groups, not three: 0.061 GiB becomes 0.245.
• A second KV cache is allocated for the draft context. It is f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k / -ctkd. blk.40 is a single attention block, so at this geometry that is 2 × 256 × 2 (K and V) × 2 B = 2,048 bytes per token, 0.125 GiB at 65,536.

That is 476,956,672 + 197,591,040 + 134,217,728 = 808,765,440 bytes — 0.753 GiB — out of the 1.109 GiB of slack this page reserved, before any throughput arrives. It also leaves the flash-attention fallback of Why 26 and not 24 with 0.063 GiB, i.e. 64 MiB, which is not a margin. There is also an open interaction between the two flags this recipe cares about: #27717 (open, six comments, re-checked 2026-08-31) reports that --n-cpu-moe below a threshold crashes the MTP draft model load with invalid vector subscript, on a different MoE — so -ncmoe and draft-mtp are not independent settings.

The return is unknown here: no MTP measurement of this model on any card turned up in the space named under Results, and the mechanism argues against it. Speculative decoding wins when the GPU is waiting on itself; under expert offload it is waiting on your RAM, and the draft pass has to walk the same host-resident experts the target pass does. The nearest evidence agrees and is on a 12 GB card, though on a different model — an owner who describes their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" and runs Qwen3.6-35B-A3B at --n-cpu-moe 25 with --spec-type draft-mtp --spec-draft-n-max 2 reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" in discussion #18 on that GGUF repo, and a second community member in the same thread offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model, so read it as a warning rather than a result. If you try it anyway, do it on a machine you can reboot, and please tell us what you get.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps publishes a hardware-guidance table whose 12 GB row names "IQ1_M or IQ2_M" with the note "IQ1_M full offload; IQ2_M may require hybrid offload". Weigh it with the rest of that card, which is candid about the cost — "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." — and which ends the same table with "Prefer Q4_K_M or higher when system memory permits."

That closing sentence is this recipe. A 1.75-bit quantisation of a model whose routed experts are 512 wide, used for tool-calling where a malformed argument is a failed task rather than a clumsy sentence, is a large quality bet to avoid a memory purchase. And check what the smaller quant actually buys before taking it. IQ2_M is 12,543,404,960 bytes — 11.682 GiB — in the lead repo (re-fetched 2026-08-31), which on a card whose usable pool this page sizes at ~11.0 GiB does not fit at all, let alone leave room for a KV cache; that is what their own table means by "may require hybrid offload". Below Q4 the trade is quality for a shallower -ncmoe, not quality for no -ncmoe at all. Only IQ1_M — 8,821,679,936 bytes, 8.216 GiB, which that publisher's own table rounds to 8.82 GB — escapes offload entirely, and it is the one its own card tells you to avoid when you can.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well (checked 2026-08-31, with ollama.com/library/qwen3.5 and the registry.ollama.ai manifest for qwen3 both returning 200 as controls). Use llama.cpp directly. If a tag appears later, please tell us.