Apodex 1.1 mini on RTX 4070: a 35B-A3B agent in 12 GB via expert offload on Ada

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one 12GB RTX 4070, at a 65,536-token context. The Q4_K_M weights are 19.908 GiB. The card holds 12 GiB. It runs anyway, because 91.2% of those weights are routed experts that llama.cpp can leave in system RAM.

Hardware data: RTX 4070 (12GB VRAM) · 8.891 GiB derived working set at -ncmoe 26, 65,536-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 4070. /check/apodex-1-1-mini/rtx-4070 holds zero benchmarks, and so does every other card in our catalogue for this model. Every VRAM figure below is arithmetic over the GGUF's own tensor table and llama.cpp's allocation code, and it is shown so you can check it. There is no throughput number on this page at all — see Results for what exists and why none of it transfers. If you run this, please send the numbers.

🧭 Two things about this card specifically, before the arithmetic. First, the RTX 4070 is Ada, compute capability 8.9, and unlike the 30-series that is not covered by llama.cpp's unconditional CUDA architecture list — the build section below is genuinely different from a 12 GB Ampere page, and getting it wrong costs you performance silently rather than loudly. Second, everything about capacity here is the same answer any 12 GB card gets, and everything about speed points away from the GPU: at -ncmoe 26 a decode step reads 0.367 GiB from system RAM against at most 1.945 GiB from VRAM, and the two halves take equal time only where your card's bandwidth is 5.30× your DIMMs'. No discrete GPU in this catalogue is anywhere near that ratio, so your generation rate is set by your memory, not by this card. If you came here from our own Qwen3.6-35B-A3B on RTX 4070 page, read The one measured run on this card — this page disagrees with it about which phase the bottleneck is in, and says why.

ℹ️ This recipe is text-only. The checkpoint has a vision tower and the GGUF repo ships a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card contains no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-30). On a card this tight the projector is also 0.838 GiB you cannot spare, so the run command below refuses it explicitly rather than merely omitting the flag.

Requirements

Two version notes, because both have bitten this catalogue. The build tag moves several times a day — b10701 was published 2026-08-30T18:56:31Z, three tags inside one hour — so this page pins b10666 and does not claim to be current; check the release list rather than trusting a date. And GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, not a bNNNN tag, because every build tag is marked prerelease; use /releases?per_page=N if you are scripting a version check.

Disk size and VRAM residency are different numbers here and they are not interchangeable. The file is 21,864,082,336 bytes — 20.363 GiB — on disk; a normal run loads 21,376,133,632 (19.908 GiB) of it. The gap is 487,948,704 B, and it is worth doing the subtraction properly rather than waving at the MTP head, because two things make it up:

The second row is simply 21,864,082,336 − 21,853,090,304, the difference between the file and the sum of its tensor payloads. It is 10.5 MiB and it changes no decision on this page — but subtracting only the MTP block from the file size does not reach the residency figure, and a budget built on a subtraction nobody has actually done is how a gigabyte goes missing somewhere else. Budget 22 GB of disk and 19.908 GiB of weights.

What NVIDIA publishes about this card, and what it does not. The vendor's own 40-series table gives the RTX 4070 as 5,888 CUDA cores, 2.48 GHz boost, 192-bit memory interface, 200 W total graphics power, with PCI Express Gen 4 listed under technology support (nvidia.com). It publishes no memory-bandwidth figure, no memory-speed figure and no PCIe lane count, so this page states none of the three. It also lists the standard memory config as 12 GB GDDR6 / 12 GB GDDR6X — the vendor itself does not resolve which memory a given RTX 4070 board carries, so you cannot look your own board's bandwidth up. Read it off the hardware instead:

# capacity, link, and the memory clock your board actually runs
nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.max,pcie.link.width.max,clocks.max.memory --format=csv

Multiply the reported memory clock by 2 (GDDR is double-data-rate) and by 24 bytes (192 bits ÷ 8) to get bytes per second. Do the same for your DIMMs from their rated transfer rate: dual-channel DDR4-3200 is 3200 MT/s × 8 B × 2 channels = 51.2 GB/s, dual-channel DDR5-6000 is 96 GB/s. Both numbers matter below and the second one matters more.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. Use b10630 or newer — the release the lead quant was produced with. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that makes blk.40 optional landed in the bracket (b10211, b10212], so b10630 is comfortably past it.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Now the part that is not the same on this card as on a 12 GB Ampere board. The RTX 4070 is Ada — GGML_CUDA_CC_ADA_LOVELACE is 890 in ggml/src/ggml-cuda/common.cuh at line 55 — and llama.cpp's CUDA architecture list treats 8.6 and 8.9 differently. In ggml/src/ggml-cuda/CMakeLists.txt, the fallback list appends 86-real unconditionally, and then:

# 86     == RTX 3000, needs CUDA v11.1
# 89     == RTX 4000, needs CUDA v11.8
...
list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real)

if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real 90-virtual)
endif()

Read that carefully, because the consequence is quiet. An RTX 3060 or 3080 Ti gets a real, architecture-specific binary out of that list no matter which toolkit you have. An RTX 4070 gets one only on CUDA 11.8 or newer, and the list contains no 86-virtual — so on an older toolkit your Ada card falls back to just-in-time compiling from 80-virtual PTX. It still runs. It is not the code the project built for your silicon, and nothing in the log calls that out.

Most people never reach that branch, and it is worth knowing why: GGML_NATIVE defaults ON (ggml/CMakeLists.txt, GGML_NATIVE_DEFAULT is ON unless you are cross-compiling or have SOURCE_DATE_EPOCH set), and with a toolkit ≥ 11.6 and CMake ≥ 3.24 that sets CMAKE_CUDA_ARCHITECTURES to native, which detects your actual card. The fallback list is what you get when any of those conditions fails — a container build with SOURCE_DATE_EPOCH set, a cross-compile, an explicit -DGGML_NATIVE=OFF, an older CMake, or an older toolkit. Check which one you got; CMake prints it during configure:

nvcc --version | tail -2                       # toolkit >= 11.8?
cmake -B build -DGGML_CUDA=ON 2>&1 | grep CMAKE_CUDA_ARCHITECTURES

You want 89 or native in that line. If you see a list ending at 86-real, your toolkit is too old for this card and you are running JIT-compiled PTX.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes, imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z; the file name and size above were re-fetched from the tree API on 2026-08-30 rather than copied from a sibling page, and the download URL was HEAD-checked the same day.

Do not start from the publisher's own quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card states plainly that "llama.cpp downloads the mmproj automatically when using" -hf. llama-server is in llama.cpp's mmproj_examples list (common/arg.cpp), no_mmproj defaults to false and mmproj_use_gpu defaults to true (common/common.h:594,596), so that command puts 0.838 GiB of vision projector on a card this recipe has already budgeted to within about a gigabyte. Download the file explicitly, as above, and load it with -m.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 26 \
    -np 1 \
    -c 65536 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Eight of those flags are decisions rather than defaults. Each was re-derived at b10666 for this page, and each one's default is quoted from the source at that pin:

• --no-mmproj refuses the vision projector rather than merely not asking for it — 0.838 GiB, for the reason given under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 26 keeps the routed experts of blocks 0–25 in system RAM. The flag's own help text at this pin is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp), and the implementation matches the wording exactly — see How the offload actually works. 26 is derived in Choosing your own -ncmoe and again in Why 26 and not 24; it is not a measured optimum. Capacity-driven — the constraint is the 12 GiB, not the Ada silicon, so this number is the same on any 12 GB card.
• -np 1. llama-server sets params.n_parallel = -1 — "auto by default" — and tools/server/server.cpp resolves that under if (params.n_parallel < 0) to params.n_parallel = 4, logging "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true". The Gated-DeltaNet recurrent state is allocated per sequence, so four slots quadruple it — 0.245 GiB instead of 0.061 — and split your context four ways. One slot is what a single-user agent loop wants. Capacity-driven.
• -ctk q8_0 -ctv q8_0 halves the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 65,536-token cache from 1.250 GiB to 0.664 GiB. Capacity-driven.
• -fa on makes Flash Attention explicit. The default at this pin is auto — flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO (common/common.h:499) — and pinning it means a future change to what auto decides cannot silently change your memory profile. ⚠️ This flag has an open, architecture-specific crash report against it under expert offload — read An illegal memory access on the flash-attention path under Troubleshooting before you run a long session. This recipe keeps it on, and the section explains what that choice costs and what the fallback costs.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader itself warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp). Taking that advice means the 11.742 GiB of expert weights are ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure rather than a cache you can squeeze.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632 — and it is more expensive on this architecture than on a plain transformer. See the host-RAM entry under Troubleshooting.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05). The card notes that Apodex "follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF and enables Jinja by default, so tool calls and <think> blocks parse the way the vendor intends. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It builds one regex per block and hands them to the model loader as buffer-type overrides (common/common.h:1130):

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, …) pushes "blk\.0" + regex … "blk\.(N-1)" + regex

The loader then does a plain std::regex_search on each tensor name and, on a match, allocates that tensor in a CPU buffer instead of the GPU one (src/llama-model-loader.cpp). Nothing in that path consults the architecture, so it applies to this hybrid recurrent model exactly as it does to a plain MoE. 123 tensors in this file match — three per block across all 41 blocks — totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads. That is 91.2% of the 19.908 GiB resident weight.

What does not move is as important. The offload takes each block's ffn_down_exps, ffn_gate_exps and ffn_up_exps, and leaves on the GPU: the attention path, the recurrent state tensors (ssm_conv1d, ssm_a, ssm_alpha, ssm_beta, ssm_norm, ssm_out), the router (ffn_gate_inp), the shared expert (ffn_*_shexp) and every norm. So routing is decided on the GPU, the Gated-DeltaNet state stays on the GPU, and only the selected experts' matmuls happen elsewhere. Offload everything the regex can reach — -ncmoe 40 — and 1.744 GiB is still resident.

That last word — elsewhere — means two different places depending on what you are doing, and it is the whole story of this recipe's performance. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights", per the scheduler's own comment (ggml/src/ggml-backend.cpp) — unless a higher-priority backend wants to offload it. CUDA does, and the test is one comparison against op_offload_min_batch_size, which is getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(...) : 32 in ggml/src/ggml-cuda/ggml-cuda.cu.

Generating a token is a batch of one, so decode reads the offloaded experts from system RAM and multiplies them on your CPU — they never cross the bus. Prompt processing is a batch of hundreds, so prefill copies those tensors across PCIe into a duplicate the graph allocator sizes for the run. A narrow slot is therefore a time-to-first-token problem, not a tokens-per-second problem, and this page treats the two separately throughout.

The VRAM budget

Weights, KV cache and recurrent state, at the recipe's settings.

The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 26 moves 12,608,077,824 of that (11.742 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks are loaded, and full_attention_interval 4 makes ten of them full-attention: blocks 3, 7, 11, 15, 19, 23, 27, 31, 35 and 39. Each carries head_count_kv 2 at key_length 256 and value_length 256 — 1,024 cache elements per token per layer, so 10,240 across ten layers. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows.
• Recurrent state. Those thirty blocks each allocate two F32 tensors, sized (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements (src/llama-hparams.cpp). That is 2,195,456 bytes per layer per sequence and 65,863,680 across all thirty. llama-server prints it at startup as RS buffer size.

The two terms that table cannot produce, and what this page reserves for them

The 8.891 GiB is not a peak. Two things sit outside it, and on a 12 GB card they decide whether a configuration loads:

1. The card's usable pool is not 12 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. Our catalogue holds one measured expert-offload run in this VRAM tier with the numbers written down — gemma4-26b on an RTX 3060 at -ncmoe 12 — whose submitter recorded "That peak is 11,179 of the card's 12,288 MiB — roughly 1.1 GB spare, which is why the lower rungs fail rather than merely slow down", on a headless box. That is a different GPU in the same tier and a single community submission stored at confidence 0.5, so read it as the order of the term and not as your board's exact reserve. This page sizes against ~11.0 GiB of usable pool, not 12.0.

2. The compute buffers. llama.cpp allocates working memory for the graph on top of everything above, and under expert offload it also stages the host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 0.4219 or 0.4863 GiB, so that staging term alone is of that order. This page reserves 1024 MiB. That number is llama.cpp's own default margin per device for auto-fit — fit_params_target is initialised to 1024 * 1024*1024 at common/common.h:481 — so it is the runtime's own opinion of how much room it wants, not a figure this page invented.

For scale on what happens if you leave nothing: llama.cpp issue #27698 records a run where the weights loaded, expert offload was honoured, and then context creation failed on a ~504 MiB compute buffer. ⚠️ That report is not about this card and does not transfer — it is an RX 6700 XT, gfx1031, RDNA2, on the ROCm backend, open with zero comments since 2026-08-25. It is quoted for one thing only: the magnitude of a term every table in this family of recipes excludes, and the shape of the failure it produces — a clean load followed by an OOM the moment context is allocated.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 11.0 GiB − 1.0 GiB reserve = 10.0 GiB.

At -ncmoe 26 and 65,536 tokens the sub-total is 8.891 GiB, which clears that by 1.109 GiB.

Choosing your own -ncmoe

Every row is weights + KV + recurrent state on a 12 GiB card, one slot, with Flash Attention on — which is what the recipe runs. The fallback in When the fallback does not fit has a different and much larger cost, priced in its own table there rather than as a column here, because that cost is dominated by a term this one does not contain.

The -ncmoe 20 and 22 rows are the point of having a ceiling at all: against a naive 12 GiB they look like they have 0.781 and 1.624 GiB to spare, and against the ~11.0 GiB a card in this tier has actually been observed to give, they do not load. The bottom row is the model's entire declared window (context_length 262144) on a 12GB card, and it clears — at four more steps of offload than this recipe, which the speed section prices in milliseconds.

Then stop guessing and measure it. llama.cpp ships a tool that answers this question on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments:
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device line splitting total / free / model / context / compute / unaccounted. That compute column is the number this page reserved 1024 MiB for, and unaccounted is your driver's share — the two terms the table above cannot produce. If you run this, those two figures are the most useful thing you could send us, more useful than a speed number.

Why 26 and not 24 — and what the flash-attention fallback really costs

-ncmoe 24 at 65,536 tokens is 9.799 GiB, clears the 10.0 GiB ceiling by 0.201 GiB, and puts two fewer blocks on the host path — the only lever this recipe has on generation speed. So why not lead with it?

Two reasons, and the second one is the section's real content because turning Flash Attention off is far more expensive than a cache change, and the expensive part is invisible in the ladder above.

Reason one: 0.201 GiB is inside the error bar. The 10.0 GiB ceiling is 11.0 usable minus a 1024 MiB reserve, and that 11.0 comes from a run on a different GPU — one community submission at confidence 0.5. A 0.201 GiB margin is smaller than the uncertainty in the number it is measured against. 1.109 GiB is not.

Reason two, and it corrects an assumption this page made in an earlier revision. The workaround for the crash report in Troubleshooting is --flash-attn off, and -fa off changes two things, in opposite directions:

• The cache gets cheaper than you would expect. -fa off rejects a quantised V cache, not a quantised K. In src/llama-context.cpp the V guard is if (ggml_is_quantized(params.type_v) && params.flash_attn_type != LLAMA_FLASH_ATTN_TYPE_ENABLED), while the K check immediately below it is gated on params.flash_attn_type != LLAMA_FLASH_ATTN_TYPE_DISABLED and therefore never runs with FA off. The one guard that would forbid mixing K and V types fires only for hparams.is_mla() or LLM_ARCH_DEEPSEEK4, and this file is neither — it carries no MLA keys at all. So -ctk q8_0 -ctv f16 is legal, at 15,680 B/token: 0.957 GiB at 65,536 tokens, not the 1.250 GiB a naive "FA off means f16" reading gives.
• The graph gets much more expensive, and this is the term that decides everything. Without Flash Attention, src/llama-graph.cpp materialises the attention scores as a real tensor — ggml_tensor * kq = ggml_mul_mat(ctx0, k, q); followed by ggml_mul_mat_set_prec(kq, GGML_PREC_F32), with the source's own explanation that "this op tends to require high floating point range". That tensor is [n_kv, n_tokens, n_head] at F32. This model's n_head is 16, and the GGUF header says so directly: qwen35moe.attention.head_count is a UINT32 with value 16. Read that key with care, because it is a strict prefix of qwen35moe.attention.head_count_kv (value 2) — a substring probe sees one key where there are two, and only the u64 length prefix in front of the name tells them apart, 30 bytes against 33. The tensor table agrees independently: blk.3.attn_output.weight is [4096, 2048] against an attention.key_length of 256, giving 4096 ÷ 256 = 16. Do not derive it from blk.3.attn_q.weight, which is [2048, 8192] and divides to 32 — that projection is exactly twice as wide because it carries the attention gate concatenated alongside q, and taking 32 here would double every figure in the table below. The graph is reserved at n_tokens = min(n_ctx, n_ubatch), so at -c 65536 and the default -ub 512:

2.000 GiB, from one tensor, against a 1024 MiB reserve. Put both corrections together — cheaper cache, far dearer graph — and the fallback's budget is weights + KV(q8_0/f16) + recurrent state + that tensor, against the ~11.0 GiB usable pool:

Read the first column before you read anything else: at the default -ub 512 the fallback does not allocate at any offload depth this page recommends — not at 26, and not at 24 — and that is before a single other compute buffer. So the lever for the fallback is -ub, not -ncmoe. Drop the micro-batch to 256 and -ncmoe 26 clears by 0.816 GiB; at -ncmoe 24 you would need -ub 128 and would still clear by only 0.408.

That is the honest answer to "why 26": not that 24 makes the fallback impossible, but that 26 keeps it one flag away while 24 makes it a two-flag climbdown with less room at the end of it. At -c 131072 the tensor doubles again and the fallback is out of reach at every setting here, which is a better reason to leave the 128K row alone than the one this page gave before.

And there is corroboration in the issue itself, which nobody had computed. The reporter's frozen arguments are --ctx-size 32768 … --ubatch-size 512, where this tensor is 1.000 GiB rather than 2.000. Read their isolation matrix by rows and the two arms that pass with flash attention off are both at n_ctx 32768 — b10107 | 32768 | yes | off (KV f16) | pass 3/3 and b10488 | 32768 | yes | off (KV f16) | pass — while every row in that table carrying 131072 runs flash attention on. So their data never exercises a large FA-off graph on a 12 GB card. That is consistent with a ceiling nobody in the thread worked out, and it means the workaround is better evidenced at their context than at this recipe's.

If your board is headless, you have confirmed you are not hitting the crash, and you want the throughput, -ncmoe 24 at 65,536 with FA on is a row in the table rather than a different recipe — just know which fallback you are further from.

Where this card's speed actually goes

The arithmetic in this section is not specific to this card, and saying so is the point. Every byte figure below is a property of this file at -ncmoe 26 — how much expert weight ends up on which side of the bus — so any 12 GB card running this recipe moves exactly the same bytes per token, and derives exactly the same crossover. What is specific to the RTX 4070 is the conclusion you should draw from those numbers, and it will disappoint you, so here it is first: on this recipe the RTX 4070's advantage over a 12 GB Ampere card is confined to prompt processing, and even there it is partly spent waiting on the bus. Generation does not run on the GPU at all.

Decode. At batch 1 the CUDA offload test fails, the offloaded expert matmuls stay on the CPU, and their weights never cross the bus. So a decode step splits across two buses, and the split is computable. expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token:

The VRAM row is an upper bound, because the token-embedding table is read one row at a time while the output head is read in full, and this page has not separated the two in this file. That the bound points upward matters twice below.

The host side is therefore at least 15.9% of the bytes — 0.367 ÷ (0.367 + 1.945) — and that is a floor, not a ceiling, precisely because the VRAM term is an upper bound: separate the embedding table out and the host's share can only rise.

Now put the two on their buses. This page will not tell you the host is more than 15.9% of the time without a bandwidth figure it is not entitled to state — so here is the threshold instead, which needs none. The two halves of the decode roofline take equal time exactly when your card's bandwidth is 1.945 ÷ 0.367 = 5.30× your system RAM's, and because the VRAM figure is an upper bound the true crossover is lower still. Compute both sides with the two arithmetic recipes under Requirements and compare. No discrete GPU in this catalogue is anywhere near that threshold — a 192-bit board against dual-channel DIMMs is far past it — and past it the host term dominates, and dominates more the faster the card. On this recipe, a faster card cannot fix decode, because decode was never on the card.

That gives a ceiling that does not depend on which GPU you own at all — divide the 0.367 GiB of host traffic by your memory bandwidth:

⚠️ Those are ceilings from one term, not predictions. They exclude the CPU time to actually multiply those experts, kernel launch latency, the VRAM side, and everything else. The one measured 12 GB-tier expert-offload run this catalogue holds with a stated memory bandwidth — gemma4-26b on an RTX 3060, -ncmoe 12, DDR4-2133 dual-channel, a different model — achieved 37.2 tok/s, comfortably below what a host-bandwidth roofline would allow it. Read the table as "you cannot beat this", never as "you will get this". A single-channel configuration halves every row.

The same record is also the cleanest available demonstration that the mechanism is real, on this exact VRAM tier: the submitter's ladder is -ncmoe 12 = 37.23 tok/s against -ncmoe 30 = 21.20 tok/s. Eighteen more blocks on the host path, on one unchanged card, cost 43% of the generation rate. That is a different model, so the numbers are not yours; the direction and the size of the effect are the point.

Which makes the offload depth the lever, priced per step. Moving one block from card to host frees its whole expert triple from VRAM — 0.4219 or 0.4863 GiB — and adds one thirty-second of it to every token's host read: 13.5 MiB for a cheap block, 15.6 MiB for an expensive one, or about 0.28–0.32 ms per token each on dual-channel DDR4-3200. Going from this page's -ncmoe 26 to 30 for the full 262,144-token window adds four blocks and roughly 1.2 ms per token. Going the other way, to 24, buys back about 0.6 ms — and costs the flash-attention fallback, as the previous section shows.

⚠️ Treat that per-step arithmetic as a floor on the cost, not as a prediction of the curve. The only -ncmoe sweep on this architecture family run against stock llama.cpp that this page could find is a comment in the same issue #25859 thread, and generation in it is not monotonic: at fixed batch and KV settings the reporter measures -ncmoe 22 → 36.75 t/s, 21 → 36.66, 20 → 42.18, 19 → 35.33, and notes that "The 20 -> 19 step alone drops generation by about 16%" while explicitly declining to claim a cause — "I did not profile PCIe transfers or GPU idle time. I am only reporting the reproducible behavior." They also re-ran a fresh server at 20 and reproduced it, so it is not a one-off. That measurement does not transfer to this card and is not evidence about it: it is an RX 9070 XT 16 GB on the Vulkan backend, a different model (Qwen3.6-35B-A3B UD-Q4_K_M), different batch sizes and q4_0 KV — and Vulkan does not share the CUDA batch-32 offload test this page's whole decode argument rests on. What it does establish is that a smooth bytes-per-token model is the optimistic reading of this flag. Sweep two or three settings on your own box rather than trusting one step of arithmetic, and if you find a cliff on CUDA, that is worth reporting.

Prefill is the opposite case, and it is where this card's extra silicon has something to do. llama.cpp issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04." running "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26, prefill at pp2048 / ub2048" — and it finds that "prefill is bottlenecked by PCIe transfers, not GPU compute", with GPU idle around 42% of each prefill pass waiting on serial expert-weight H2D copies that do not overlap the matmuls consuming them.

Take that seriously and it bounds what this card's extra compute can buy. NVIDIA's own tables make the RTX 4070 5,888 CUDA cores at 2.48 GHz against the RTX 3060's 3,584 at 1.78 GHz (40-series 4070 family, 30-series 3060 table) — a real gap in arithmetic throughput — but if roughly two fifths of prefill is the GPU waiting on copies that a faster GPU does not accelerate, then the compute advantage applies to the other three fifths only. The report's caveats limit this further and all of them matter: it is a different model, a single box, a custom fork used for A/B, and the author says the measurement dates to "around the b6xxx era" and may be stale against current master. Treat it as the shape of the answer, not its value — the shape being that a newer card on this recipe is faster at the part that was already not the bottleneck.

Two things you can act on, both of which point away from the GPU: put your DIMMs in dual channel and buy memory bandwidth before you buy silicon; and if time-to-first-token specifically is bad while generation is fine, look at the slot, because prefill is the only phase where the link is on the critical path.

The one measured run on this card, and why you should not copy its command

Our catalogue holds a published expert-offload recipe with a measured number for this architecture family on two 12 GB cards, and this is one of them: Qwen3.6-35B-A3B on RTX 4070, 80.8 tok/s at a 128K context, from benchmark id 2. The other is on an RTX 3060 at 38.9 tok/s, and this page leans on it two sections down. If you own this card you have probably read the 4070 one. Three things about it bear on this page, and two of them are corrections.

What separates the two records is how the offload was chosen, and that is what makes the rest of this section apply here and not there. The RTX 4070 benchmark used auto-fit — its note reads "Using -fitt 1536 to balance GPU/CPU load", i.e. --fit-target 1536 rather than an explicit -ncmoe. The RTX 3060 record pins its offload by hand, and records the command that did it: -ncmoe 24. So a regression in the fitter lands on this card's published number and leaves the sibling's untouched — and there is one.

And auto-fit has since regressed on this architecture. llama.cpp issue #27171 is an open, bug-unconfirmed report of a --fit-target throughput regression measured with llama-bench on a qwen35moe 35B-A3B Q4_K_M — the same architecture string and quant class this page installs — reporting pp2048 1265.61 → 923.45 t/s and tg256 101.00 → 87.92 t/s across one commit. The reporter bisected it to 9a688e51e, tagged b10284, whose subject is "fit: Fix memory allocation for MTP layers (#26605)", and the open, unmerged PR #27207 that names #27171 describes the problem as failing to "account for MTP/NextN layer positions in n_gpu_layers, even when their tensors are not loaded", with the consequence that "regular layers remain on the CPU and tensor override indices become misaligned". This model has exactly that shape — nextn_predict_layers is 1 and blk.40 is skipped by default — which is why this page pins -ncmoe explicitly rather than letting the fitter choose. Our catalogue's record of the 80.8 figure is dated 2026-05-13, roughly three months before b10284 (2026-08-05), so it predates the regression: a reader who copies that command onto a current build is not reproducing that measurement.

Its explanation of the bottleneck is wrong, and this page contradicts it deliberately. As that page read on 2026-08-30, its troubleshooting said "The bottleneck for MoE CPU-offload is PCIe latency moving expert tensors, not raw CPU speed." (it may since have been corrected — the sentence is quoted here for the mechanism, not to keep score) For prefill that is right, and #25859 above is the profiling evidence for it. For decode it is not: at batch 1 the offload test in ggml_backend_cuda_device_offload_op fails against op_offload_min_batch_size 32, the expert matmuls run on the CPU, and those weights never cross PCIe at all. The advice that follows from the two readings is opposite — one sends you shopping for a wider slot, the other for faster DIMMs — so it is worth being explicit about which phase you are trying to fix.

And its 12GB peak is the card, not a measurement. That benchmark record stored peak_vram_gb: 12.0 when this page was written, which is the capacity of the board rather than an observed peak; the one 12 GB-tier record in this catalogue that reports an actual reading gives 11,179 of 12,288 MiB. Do not size a budget against 12.

None of this makes the 80.8 wrong. It is a real figure for a real run on this card, and it is consistent with everything above — a 35B-A3B under partial offload can reach that rate if the host memory can feed it, which is the whole point of the ceiling table.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true at common/common.h:476 — and moves MoE tensors to system memory until it can leave its 1024 MiB margin. That is a reasonable path, with two caveats: it records its choice nowhere except the log, so you cannot reproduce a run from the command line alone; and #27171 above is open and unfixed against exactly this feature on exactly this architecture, with its fix still unmerged as of 2026-08-30.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line in the log means your offload is being honoured, not that something broke.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys you X GB" is wrong here. bartowski's imatrix recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28 and 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates, so the first few steps of -ncmoe and the last few buy more room than the ones in between. Block 25 in particular is an expensive one, which is why the 25 row in the ladder above moves by 0.4863 GiB rather than 0.4219. Read the cumulative column rather than multiplying.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 19,278,556,064 bytes — 17.955 GiB on disk — and its ladder is linear at 0.3984 GiB per step. If you would rather spend quality than throughput, that is the trade — but derive its table from its own tensor list, do not scale this one.

Does the arithmetic survive contact with a real card?

No one has run Apodex on an RTX 4070, and /check/apodex-1-1-mini/rtx-4070 returns unknown with zero benchmarks — as does every other pair for this model. The nearest test of the method is a different model on a 12 GB card with a figure at four offload depths: Qwen3.6-35B-A3B at UD-Q4_K_XL on an RTX 3060. That model is the same architecture shape as this one — 41 blocks, ten full-attention, identical ssm parameters and expert dimensions — so a weight ladder computed from its own tensor table is directly comparable:

⚠️ Read that right-hand column as GiB of weights, and do not read it as a peak. It tracks the derived weight ladder to within 0.16 GiB across a 9 GiB range. Read as decimal GB, every point falls below the weights that setting must load, which is impossible. And a figure that tracks the weights cannot also contain the KV cache and the compute buffers: at the top rung, 11.7 means the interval [11.65, 11.75] GiB against 11.542 GiB of derived weights, leaving at most 0.208 GiB for everything else — while that run's f16 KV cache alone was 0.168 GiB at its 8,832-token bench depth, plus 0.061 of recurrent state. The excluded terms do not fit in what remains, so those numbers are floors, not peaks. Our own RTX 3060 recipe for that model built its context table on the 9.8 figure read as a measured peak when this page was written (2026-08-30); that reading does not survive the arithmetic above, and no number from it appears anywhere on this page.

What the comparison does establish is the shape of the ladder, at four settings spanning 9 GiB. It is not four independent measurements — one submitter, one rig, one model, one tool — so every systematic error the method could have is shared across all four points.

Two other pages on this exact card worth reading first

gpt-oss 20B on RTX 4070 is the same card and the same technique on a much smaller model, and it is the cheaper way to find out whether your box's CPU and RAM are up to an offloaded MoE at all — build llama.cpp, run that, then come back. If instead you have 24 GB and landed here by accident, Apodex 1.1 mini on RTX 3090 fits the whole model on the card with no offload and none of this page's arithmetic applies.

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this file, under llama.cpp, on any NVIDIA card, at any offload depth exists in the space I searched on 2026-08-30, and the space was: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces (re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100; the count was 21/10 two days ago, so re-run it rather than trusting this sentence — 21 cards were fetched and read, and aykutx21/Apodex-1.1-mini-GGUF has no README.md at all); the discussions tab of every one of those repos, which holds one thread between them, a community announcement rather than a measurement; the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 271 for qwen35moe and 727 for n-cpu-moe on the same query shape; and our own /check, where this model has zero benchmarks on all three cards it is linked to. Figures for the model do exist and every one of them is a different artifact on a different engine:

  • kingjones777 publishes a table covering four of his own repacks — 64.87 / 63.30 / 45.05 / 32.54 tok/s — in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (gfx1151), which is not a catalogue card. ⚠️ The 64.87 is the one figure of the four whose own repo card carries no measurement paragraph, and that card's hand-written speed section shows a dash beside "Full-offload speed being measured on an idle box, card will be updated".
  • ghazni101 reports "generation measured at 226 tok/s end-to-end for MQ4R". Three disqualifications, each sufficient: it is a .mq4r file under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF; and it is an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim. (The repo does ship both an mq4r and an mq4rp file, so the figure does measure something the repo hosts; the "~10% RP decode penalty" sentence beside it is a prediction about the other one.)

  One figure on this pair would be worth more than all of the above. If you run this, contribute it.

• VRAM usage: 8.891 GiB derived sub-total at -ncmoe 26 and a 65,536-token context, plus the compute buffers and driver share this page reserves 1024 MiB and ~1 GiB for respectively but cannot derive. See /check/apodex-1-1-mini/rtx-4070 for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: with 26 of 40 blocks' experts on the host, decode is paced by your CPU and system-memory bandwidth and not by this card — see the ceiling table above, and treat it as a ceiling. Prefill is where the RTX 4070's compute has something to do, and even there a profiled run of this shape spent about two fifths of the time waiting on the bus.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated, which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. Note that the publisher stores the MTP layers at Q4_0 in the imatrix quants — "since imatrix calibration does not exercise them", per their own card — which is a detail that only matters if you turn the MTP head on. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-4070.

Troubleshooting

An illegal memory access on the flash-attention path

This is the one open report that lands squarely on this recipe's configuration, and as of 2026-08-30 no other page in this catalogue cited it.

llama.cpp issue #26609 — open, four comments, last updated 2026-08-29 — reports CUDA error: an illegal memory access was encountered in ggml_backend_cuda_synchronize while serving a qwen35moe model under partial expert offload on a 12 GB card with --flash-attn on. The reporter ran a one-variable-at-a-time matrix and --flash-attn off is the discriminant: it "passes 3/3 on both builds" where every arm with flash attention on crashes. They also rule out several plausible culprits — the crash survives --slot-prompt-similarity 0, cache_prompt: false, all-experts-to-CPU, whole-layer offload instead of tensor overrides, f16 KV instead of q8_0, and GGML_CUDA_DISABLE_GRAPHS=1.

Read the scope before you inherit the symptom, because four things limit it:

1. It is a different model — Qwen3.6-35B-A3B UD-Q4_K_M, and a Qwen3.6 finetune in the second reporter's case. It is the same architecture string as this file, which is a closer match than most such reports, but it is not Apodex.
2. It is a different card — an RTX 5070 12 GB for the primary reporter, an RTX 3070 Laptop 8 GB for the second. Same VRAM tier for the first; no RTX 4070 report exists.
3. It is strongly input-dependent. The reporter states that a synthetic prompt of the same length and block structure, and even "a shape-preserving scramble of the triggering prompt", both pass on the exact configuration where the real prompt crashes deterministically — which is also why no public reproducer is attached.
4. The newest build it is confirmed on is b10488, which is older than the b10630 this page pins. Nobody has retested at b10630 or later. The reporter's own wording for the newest confirmed arm is "not fixed as of that build", so that is what is established; "still broken today" is not.

What to do. Keep -fa on as this recipe has it — it is what makes the q8_0 KV cache available, and the report may not apply to your model or your prompts. If you hit an illegal memory access mid-session, the diagnostic is one flag:

# -fa off rejects a quantised V but NOT a quantised K, so keep -ctk q8_0.
# -ub 256 is not optional here: at the default 512 the F32 attention-score
# tensor is 2.000 GiB and this will not allocate. See "Why 26 and not 24".
./build/bin/llama-server -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj -ngl 99 -ncmoe 26 -np 1 -c 65536 \
    -fa off -ctk q8_0 -ctv f16 -ub 256 \
    --load-mode none -cram 2048 \
    --host 127.0.0.1 --port 8080

Price that properly before you run it, because the obvious reading of the cost is wrong twice over. The cache gets cheaper than "FA off means f16" suggests — a quantised K survives, so -ctk q8_0 -ctv f16 is 0.957 GiB rather than 1.250. ⚠️ The published workaround over-corrects here, so do not copy it verbatim. #26609's probe G sets --flash-attn off and annotates it "+ KV f16, required" — the reporter moved both halves of the cache to f16, where only V requires it. On this model and a 65,536-token window that is 0.293 GiB spent for nothing, on a card that has none to spare. Two separate guards in src/llama-context.cpp throw on a quantised V without Flash Attention — one at :464, "quantized V cache was requested, but this requires Flash Attention", and one at :3675 — and neither says anything about K. But the graph gets far dearer: without Flash Attention the attention scores are materialised as an F32 [n_kv, n_tokens, n_head] tensor, 2.000 GiB at -c 65536 and the default -ub 512. That is why -ub 256 is in the command above and is not optional — at 512 this configuration does not allocate at all. The full arithmetic, including what happens at -ncmoe 24, is in Why 26 and not 24. If the crash goes away, you have reproduced #26609's discriminant on a new model and a new card, and that is worth telling both us and the issue.

One counter-observation, offered for scope rather than reassurance: our own gemma4-26b run on an RTX 3060 is a 12 GB expert-offload run with "flash-attn on" recorded in its notes and no crash reported — but that is a gemma4 model, not qwen35moe, and #26609 is specific to the latter.

Out of memory once a long prompt lands, but not at load

Two different causes, and they need opposite fixes.

The KV cache grows with the prompt at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the table above and either raise -ncmoe or lower -c.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means copying those tensors into a duplicate the graph allocator sizes for the run. Each of this file's expert triples is 0.4219–0.4863 GiB, so prefill peaks materially above the generation footprint the table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those operations on the CPU where their weights already are; expect slower prefill in exchange, and note that this is the one place a wider slot was helping you.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it, not one.

The first is the offload itself: at -ncmoe 26 this recipe puts 11.742 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second is the one that catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache and it is on by default at 8192 MiB — cache_ram_mib = 8192 in common/common.h:632, exposed as -cram / --cache-ram with -1 for no limit and 0 to disable (added by PR #16391, "server : host-memory prompt caching", whose body says the cache "is stored in regular RAM"). So a default llama-server may hold up to 8 GiB of RAM beyond the offload — with this recipe's 11.742 GiB that is close to 20 GiB of host memory before the operating system gets any, which is why the command above pins it lower.

And an entry here has a floor. Saving a prompt calls llama_state_seq_get_data_ext on the whole sequence state (tools/server/server-context.cpp), and on a hybrid model llama_memory_hybrid::state_write writes both children — the attention cells and the recurrent memory, the latter unconditionally. So every cached prompt carries its tokens' KV at your -ctk/-ctv type plus a full copy of the 62.8 MiB recurrent state, whatever its length; the fixed copy is the larger half of an entry until roughly 6,000 tokens, where 10,880 B/token overtakes it.

There is one direct measurement of this growth and it is worth reading before you size your box. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. It is closed as completed, 2026-08-28. Two cautions if you cite it yourself: the 126 MiB is the only measured per-entry figure anywhere and it is roughly twice one recurrent state (62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support even though the 126 stands as a measurement; and it was measured on a different model of the same architecture and on ROCm — the reporter rebuilt with -DGGML_VULKAN=ON and got +126.0 MiB per request against ROCm's +126.3, so this is a model-and-request-layer behaviour and not a vendor story.

So: 32GB is the practical floor for this recipe as written, and 16GB is not enough. On a 32 GB box keep -cram at 2048 as above, or pass -cram 0 to turn caching off entirely and trade prompt-reuse speed for headroom. Lowering -ncmoe moves weight back onto the card and off the host, and the one lever that helps both sides at once is a smaller quant.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE. Check llama-server --version against the release list and move to b10630 or newer. On this card, check the CUDA toolkit in the same pass — see Build llama.cpp with CUDA, where 11.8 is the floor for emitting sm_89 at all.

And note what a build floor buys you here beyond "the model loads". llama.cpp issue #25162 — "Performance regression on Turing GPUs after 9e58d4d69 — Qwen35 SSM kernels affected", closed as completed 2026-08-03 — recorded a 24–42% loss in ssm-scan.cu / ssm-conv.cu, the kernels behind 30 of this model's 40 layers, while "Gemma4 (no SSM layers) is unaffected". The b10630 floor is well past the commit the reporter retested at, so a current build carries the fix and an older one loses speed silently. ⚠️ Do not read that as an Ada measurement. The regression was measured on Turing (SM75). At b10666 GGML_CUDA_RESTRICT is emptied only under GGML_CUDA_USE_PDL && __CUDA_ARCH__ >= GGML_CUDA_CC_HOPPER (900) in ggml/src/ggml-cuda/common.cuh, and Ada is 890 — so this card compiles those kernels down the same __restrict__-retaining branch Turing did, the codegen effect is an nvcc property, and nobody has measured whether it reaches Ada. Move to a current build anyway; just do not expect a specific number back.

The model emits !!!! or uniform-probability gibberish

Searching the llama.cpp tracker for this architecture string turns up several gibberish-shaped reports, and the first thing to do with any of them is check which backend it is about. The nearest one to this recipe is issue #25857, filed 2026-07-18 on a Windows 12GB NVIDIA box: every token at an identical log-probability of −12.42 — the uniform distribution over the vocabulary — at any offload setting including pure CPU. Read its scope before you inherit the symptom. It is against a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins; it remains open with the stale label and zero comments as of 2026-08-30; and a separate publisher reports coherent output from an Apodex Q4_K_M on an older build still (vcruz305 smoke-verified -ngl 80 on b9835). The other reports in that neighbourhood are on other backends — #24168 is SYCL on an Intel Arc card, #23321 is Vulkan with --no-kv-offload and is closed — so neither describes a CUDA build. Rebuild at a current release first.

For completeness on the one issue this catalogue's older Apodex pages point at: llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label. It was open when our RTX 3090 and RTX 5090 pages were written and it is not now. It also never applied to a text-only run — which is what --no-mmproj makes this one. It is a separate report from #26609 above, which is about the flash-attention path on text requests.

Throughput far below what your CPU should manage

Check three things before suspecting the model, and note that none of them is the GPU. -ncmoe too high pushes more blocks onto the host path than you need — each step is 13.5–15.6 MiB more read from RAM per token, about 0.3 ms on dual-channel DDR4-3200. Single-channel system memory halves the bandwidth those reads get, which halves the ceiling table above. And if you are using auto-fit rather than an explicit -ncmoe, pin the offload yourself rather than passing --fit-target, because of issue #27171.

If it is specifically the wait before the first token that is bad while generation itself is fine, look at the slot rather than the model: prefill is the only phase where the expert weights cross the bus, so a card in a chipset-fed ×4 slot or on a riser pays there and nowhere else.

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

⚠️ If you have set GGML_CUDA_DISABLE_GRAPHS while debugging, unset it rather than setting it to 0. At b10666 the check is getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr — presence, not value — so =0 disables CUDA graphs exactly as thoroughly as =1 does.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp will drive it with --spec-type draft-mtp — the exact spelling matters. On a 24 GB card that is a reasonable experiment. On this one, price it first, because the costs land on the scarcest resource you have:

• blk.40 stops being skipped: +0.444 GiB, and note that -ncmoe 26 does not touch it. The flag installs overrides for blk.0 through blk.25; only -ncmoe 41 would reach block 40's experts.
• The recurrent state is multiplied by the draft depth — need_n_rs_seq() at :394 returns draft.n_max — so --spec-draft-n-max 3 takes 0.061 GiB to 0.245.
• A second, single-layer KV cache is allocated for the draft context. It is f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k / -ctkd — so at 65,536 tokens on one layer it is 2,048 bytes per token, 0.125 GiB.

That is 0.753 GiB out of the 1.109 GiB of slack this page reserved, before any throughput arrives — and it stacks on top of a fallback path that is already tight: Why 26 and not 24 shows the -fa off configuration needs -ub 256 merely to allocate, and MTP's costs come out of the same pool. The return is unknown here: nobody has measured MTP on this model on any card, and the mechanism argues against it. Speculative decoding wins when the GPU is waiting on itself; under expert offload it is waiting on your RAM, and the draft pass has to walk the same host-resident experts the target pass does. The nearest evidence agrees and is on a 12 GB card, though on a different model — an owner who describes their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" and runs Qwen3.6-35B-A3B at --n-cpu-moe 25 with --spec-type draft-mtp --spec-draft-n-max 2 reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" in discussion #18 on that GGUF repo, and a second community member in the same thread offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model, so read it as a warning rather than a result. Measure both ways on your own prompts before keeping it, and please tell us what you get.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps publishes a hardware-guidance table whose 12 GB row names "IQ1_M or IQ2_M" with the note "IQ1_M full offload; IQ2_M may require hybrid offload". Weigh it with the rest of that card, which is candid about the cost — "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." — and which ends the same table with "Prefer Q4_K_M or higher when system memory permits."

That closing sentence is this recipe. A 1.75-bit quantisation of a model whose routed experts are 512 wide, used for tool-calling where a malformed argument is a failed task rather than a clumsy sentence, is a large quality bet to avoid a memory purchase. And check what the smaller quant actually buys before taking it. IQ2_M is 12,543,404,960 bytes — 11.682 GiB — in the lead repo, which on a card whose usable pool this page sizes at ~11.0 GiB does not fit at all, let alone leave room for a KV cache; that is what their own table means by "may require hybrid offload". Below Q4 the trade is quality for a shallower -ncmoe, not quality for no -ncmoe at all. Only IQ1_M — 8.216 GiB in that publisher's ladder — escapes offload entirely, and it is the one its own card tells you to avoid when you can.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well (checked 2026-08-30, with ollama.com/library/qwen3.5 and the registry.ollama.ai manifest for qwen3 both returning 200 as controls). Use llama.cpp directly. If a tag appears later, please tell us.