Apodex 1.1 mini on RTX 5070 Ti: a 36B agent at 128K, and where DDR5 flips the bottleneck

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one RTX 5070 Ti, at a 131,072-token working context, from a Q4_K_M GGUF that is larger than the card.

The weights load 19.908 GiB. The card holds 16. It runs because the model is a sparse Mixture of Experts and llama.cpp will leave the routed experts of the first N blocks in system RAM: 123 tensors carry 18.586 GiB of the file, and across the forty blocks a normal run loads that is 18.164 GiB of the 19.908 GiB resident — 91.2%. Everything the card must keep — attention, the Gated-DeltaNet mixers, the shared experts, the router, the embedding table and the output head — is 1.744 GiB.

Hardware data: RTX 5070 Ti (16 GB GDDR7, 256-bit, Blackwell) · 13.546 GiB derived working set at -ncmoe 17, 131,072-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5070 Ti. /check/apodex-1-1-mini/rtx-5070-ti returns unknown with zero benchmarks. Every figure below is arithmetic over this GGUF's own tensor table, over NVIDIA's published specification for this board, and over llama.cpp's allocation code at a pinned release — written out so you can check it. There is no throughput measurement on this page; Results names the space searched and what does exist. If you run it, please send the numbers.

🧭 The conclusion that is specific to this card, before the arithmetic that gets there. Every offload page in this family ends with your system memory sets the pace, not the GPU. On this card that answer depends on your DIMMs, and the line runs right through the middle of what people own. At -ncmoe 17 a decode step reads 0.260 GB from system RAM and 2.00–3.43 GB from VRAM, depending on how full your context is; the two halves cost the same wall-clock only when the card's bandwidth is 7.70× to 13.18× your DIMMs'. Against the 896 GB/sec NVIDIA publishes for this board, dual-channel DDR4-3200 gives 17.50× and is host-bound at every depth — the usual answer. Dual-channel DDR5-6000 gives 9.33×, which is host-bound on a short prompt and VRAM-bound at this recipe's own 131,072-token window. At full context the switch happens at about 4,250 MT/s dual-channel, i.e. below the slowest DDR5 that ships. Where the decode time actually goes prices it out. Everything about capacity on this page is what any 16 GB card gets; everything about speed is not.

⚠️ The single most likely thing to bite you on this card is not the model. Five independent reporters on five different Blackwell boards — including a GB203-based RTX 5090 Laptop, an RTX 5080 and an RTX PRO 4000 — have logged an Xid 8 GPU-channel hang under sustained load on this model's sibling architecture, with GGML_CUDA_DISABLE_GRAPHS=1 as a complete A/B-tested workaround (llama.cpp #27330, open). The workload that triggers it is a long-running coding-agent session, which is exactly what this model is for. It is not in the run command below, because it costs throughput and nobody has reproduced it on qwen35moe; it is the first thing to try if your GPU locks up. See The GPU locks up minutes into a long agent session.

ℹ️ This recipe is text-only, deliberately. The checkpoint has a vision tower and the lead GGUF repo ships two projectors for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-31). The smaller projector is 0.838 GiB, which on this budget is nearly two full steps of -ncmoe spent on a capability the recipe does not use, so the run command refuses it explicitly rather than merely omitting the flag.

⚠️ Three floors, and only the first is about whether the model loads.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990 (it is the { LLM_ARCH_QWEN35MOE, "qwen35moe" } entry on line 42 at the pin below), so it arrived in the interval (b7989, b7990]. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that keeps blk.40 off the card — int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0; at src/models/qwen35moe.cpp:45 — arrived in (b10211, b10212]. Both are intervals bracketed by adjacent published tags, which is as fine as a tag-granularity probe resolves. Use b10630 or newer, the release the lead quant was produced with.
• A toolkit floor that is specific to Blackwell. A non-native build emits no sm_120 device code below CUDA Toolkit 12.8, and there is no Blackwell PTX to fall back on either. Build llama.cpp with CUDA has the four lines of CMake that decide it.
• A throughput floor. llama.cpp issue #25162 — "Performance regression on Turing GPUs after 9e58d4d69 — Qwen35 SSM kernels affected", closed as completed 2026-08-03 — records a 24–42% loss in ssm-scan.cu / ssm-conv.cu, the kernels behind thirty of this model's forty layers, with "Gemma4 (no SSM layers) is unaffected". It closed on a third party's one-line retest at 0b14b87d7 reporting 27.47 tok/s, above the pre-regression figure. That commit is 426 commits behind the tree pinned here and zero ahead, so performance had recovered by this build — but no fix was ever identified: PR #25185 was asked about in the thread and never confirmed, the original reporter notes that restoring parameter-level __restrict__ alone did not restore performance, and the issue carries the stale label. Treat b10630 as a tree past which the loss was no longer reproduced, not as a tree carrying a named fix. ⚠️ That regression was measured on Turing (SM75), and this card is on the other side of the switch that caused it — see the note under the build step. What Blackwell does on those kernels is unmeasured, in either direction.

Every source line quoted on this page was read at release b10666 (4e97ac86ebe2c4cb8212d98d2641ad6768810896), and line numbers are given for that tree only. The tag moves several times a day — b10712 was published 2026-08-31T05:39:28Z, with four more tags inside the preceding two hours — so this page pins rather than claiming currency. Note also that GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, because every bNNNN tag is marked prerelease; use /releases?per_page=N if you script a version check.

Requirements

What the vendor publishes about this board, and what it does not. From NVIDIA's own RTX 5070 family page and its full comparison table, read 2026-08-31:

The bandwidth figure on this page is a vendor number, and that is worth saying because it usually is not. NVIDIA's 40-series product pages publish an interface width and no bandwidth at all; the 50-series comparison table publishes both, and it gives 896 GB/sec here against 960 for the RTX 5080 and 448 for the RTX 5060 Ti — three 16 GB Blackwell cards, one capacity, three different buses. Everything in this page's capacity arithmetic is identical on all three. Everything in its speed arithmetic is not.

What it does not publish is the PCIe lane count. The strings lanes, x8 and x16 occur zero times on that page; the vendor states the generation only. This page asserts no lane count, and the one command that settles it on your machine is:

nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.max,pcie.link.width.max,pcie.link.gen.current,pcie.link.width.current --format=csv

For your DIMMs, do the arithmetic from their rated transfer rate: dual-channel DDR4-3200 is 3200 MT/s × 8 B × 2 = 51.2 GB/s, dual-channel DDR5-6000 is 96 GB/s, and a single-channel configuration halves whichever you have. Both sides of that comparison decide the answer on this page, and unusually for this card, neither side dominates outright.

On units, because this is where the arithmetic below would go wrong. VRAM is binary: a "16 GB" card is 16 GiB = 16,384 MiB, and llama.cpp agrees — ggml-cuda.cu reports device memory as prop.totalGlobalMem / (1024 * 1024) in MiB. Bandwidth is decimal: 896 GB/sec is 896 × 10⁹ bytes per second. Every division on this page that mixes the two takes bytes on the numerator, never GiB — feeding a rounded GiB figure into a decimal GB/s divisor is a silent 7.4% error.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered as LLM_ARCH_QWEN35MOE in src/llama-arch.cpp.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

The part that is genuinely different on Blackwell. GGML_CUDA_CC_BLACKWELL is 1200 at ggml/src/ggml-cuda/common.cuh:59, and ggml/src/ggml-cuda/CMakeLists.txt builds its fallback architecture list in four gated steps. Its own comment block states the requirement outright — # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores at :19:

list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real)   # :34, unconditional
if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real 90-virtual)          # :36-38
endif()
if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real)                   # :40-52
endif()
if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.9")
    list(APPEND CMAKE_CUDA_ARCHITECTURES 121a-real)                   # :53-55
endif()

Three things follow, and the first is the one that costs you.

1. Below CUDA 12.8 there is no Blackwell entry in that list at all, and no Blackwell PTX either. 120a-real is real-only, and the file explains at :41-50 that the virtual form was deliberately not shipped — "until the release of e.g. Rubin there is no benefit to shipping virtual architectures for Blackwell." So an old-toolkit build leaves this card just-in-time compiling 90-virtual PTX. It works, it is silent, and it is not the code the project built for this silicon.
2. 120a is architecture-specific on purpose. A foreach loop at :80-92 rewrites every plain 12X in the list into 12Xa, under the comment "12X is forwards-compatible, 12Xa is not" and a note naming the Blackwell FP4 tensor-core instructions as the reason.
3. GGML_NATIVE saves most readers, and you should know when it does not. It defaults ON — ggml/CMakeLists.txt:105-109 sets GGML_NATIVE_DEFAULT to OFF only if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}), and :123 wires that into the option — and with a toolkit ≥ 11.6 and CMake ≥ 3.24 it sets CMAKE_CUDA_ARCHITECTURES to native, which resolves to your actual card. So an ordinary source build on the machine that holds the GPU escapes the whole issue. A container build, a cross-compile, a reproducible build that exports SOURCE_DATE_EPOCH, or an explicit -DGGML_NATIVE=OFF does not.

Check which list you got. CMake prints it during configure at :100:

nvcc --version | tail -2                                    # toolkit >= 12.8?
cmake -B build -DGGML_CUDA=ON 2>&1 | grep "Using CMAKE_CUDA_ARCHITECTURES"

You want 120a in that line, or a native build resolving to it. A list that stops at 90-virtual means your toolkit is too old for this card.

⚠️ And a related switch this card sits on the far side of, which is a fact about the build rather than a measurement. GGML_CUDA_RESTRICT is defined empty when defined(GGML_CUDA_USE_PDL) && defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= GGML_CUDA_CC_HOPPER and as __restrict__ otherwise (common.cuh:1649-1654, under the comment // PDL and __restrict__ need to be mutually exclusive). GGML_CUDA_CC_HOPPER is 900 and Blackwell's __CUDA_ARCH__ is 1200, so this card compiles the branch with no __restrict__ and with programmatic dependent launch, while every Ada, Ampere and Turing card compiles the other one. That is the same switch #25162 above is about — and #25162 measured only the __restrict__ branch, on a 2080 Ti. Nobody has measured the SSM kernels on Blackwell. This is stated so you do not carry an Ada page's conclusion here; it is not a claim that anything is slow.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes (20.363 GiB), imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z. The file list, the byte count and the download URL were re-fetched and HEAD-checked on 2026-08-31 rather than copied from a sibling page — the resolve URL 302s to a CDN object whose x-linked-size is that same byte count.

Disk size and VRAM residency are different numbers here, and the gap has two parts. The file is 21,864,082,336 bytes (20.363 GiB) on disk; a normal run loads 21,376,133,632 (19.908 GiB). The difference is 487,948,704 bytes, and only most of it is the model. 476,956,672 of it is blk.40, the multi-token-prediction head, created with TENSOR_SKIP unless you ask for it. The remaining 10,992,032 bytes (10.5 MiB) is the file's non-tensor content — the GGUF header, the key-value metadata, the tensor-info table and the alignment padding — and it is exactly 21,864,082,336 − 21,853,090,304, the file size minus the sum of its tensor payloads. So the MTP block alone does not account for the gap. Budget 22 GB of disk and 19.908 GiB of weights, and do not substitute one for the other.

Do not start from the publisher's quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card itself says "llama.cpp downloads the mmproj automatically when using" -hf. It does: find_best_mmproj at :632 resolves a sibling by filename keyword, this repo ships two projectors (899,283,488 and 902,822,432 bytes), and no_mmproj defaults to false while mmproj_use_gpu defaults to true (common/common.h:594,596). So the quickstart puts 0.838 GiB of vision projector on a card this recipe has already budgeted to within about a third of a gigabyte. Download the file explicitly, as above, and load it with -m.

Q4_K_M on a 16 GB card is a choice, and the alternative is narrower than it looks. Ten of bartowski's rungs are under 16 GiB on disk, so "small enough to load" is not the constraint — what has to sit beside the weights is. Hold this page's ceiling (derived below: 13.917 GiB for weights plus KV plus recurrent state) and a 32,768-token q8_0 cache, and a fully resident configuration needs weights at or below 13.524 GiB. The whole 3-bit tier misses that: IQ3_XXS is 14.287 GiB and Q3_K_S is 14.886. The escape from offload is not "drop to 3-bit", it is "drop to 2-bit" — Q2_K_L at 12.650 GiB, or IQ2_M at 11.682. That is exactly where a second publisher draws the line: abenzerps's hardware table gives the 16 GB row as "IQ2_M full offload; IQ3_M may require hybrid offload" and closes the section with "Prefer Q4_K_M or higher when system memory permits." This recipe follows that closing sentence, because the model is an agent that runs tool calls and multi-step plans, and a malformed argument is a failed task rather than a clumsy sentence.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 17 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

Nine of those flags are decisions rather than defaults. Each default was read at b10666 for this page rather than carried across from another card, and each decision is labelled capacity-driven — meaning it is the same on any 16 GB card — or bandwidth-driven, meaning it is not.

• --no-mmproj refuses the vision projector rather than merely not asking for it: 0.838 GiB, for the reason under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point; -ncmoe then walks part of it back off. Capacity-driven — it is the baseline the offload is measured back from, and it is also what keeps the KV cache and the recurrent state on the card.
• -ncmoe 17 keeps the routed experts of blocks 0–16 in system RAM. The flag's help text at this pin is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp:2789) and the implementation matches the wording exactly. 17 is derived in Choosing your own -ncmoe as the lowest rung that clears this page's ceiling at 131,072 tokens — it is not a measured optimum, and it is not inherited. Capacity-driven.
• -np 1. llama-server sets params.n_parallel = -1; // auto by default (common/arg.cpp:1400) and resolves that to four slots. The Gated-DeltaNet recurrent state is allocated per sequence, so four slots take it from 0.061 GiB to 0.245 and split your context four ways. One slot is what a single-user agent loop wants. Capacity-driven.
• -c 131072 is half the model's declared context_length of 262144. The ladder below prices the other half; it costs three more steps of -ncmoe. Both, and on this card it is the largest single speed decision on the page — capacity-driven for the 1.328 GiB the cache occupies, and bandwidth-driven because that cache is re-read out of VRAM on every generated token. Context depth moves the crossover from 7.70× on an empty window to 13.18× on a full one, and on this board that range straddles the DDR5 line.
• -ctk q8_0 -ctv q8_0 quantises the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 131,072-token cache from 2.500 GiB to 1.328 — a saving of 1.172 GiB, which is worth between two and three steps of -ncmoe you do not have to spend (a step is 0.4219 or 0.4863 GiB, so two come to 0.844–0.973 and three to 1.266–1.459). Capacity-driven — and on this card it buys speed too, because the KV cache is read out of VRAM on every decode step and halving it halves the largest single term in the decode budget.
• -fa on makes Flash Attention explicit. The default at this pin is auto — flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO (common/common.h:499) — and auto would turn it on anyway here, because a quantised V cache requires it: src/llama-context.cpp:3675-3683 logs "enabling flash_attn since it is required for quantized V cache" under AUTO and returns a null context under DISABLED. Pinning it means a future change to what auto decides cannot silently change your memory profile. Capacity-driven. ⚠️ There is an open, well-isolated crash report on this path for this architecture — see An illegal memory access on the flash-attention path.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp:1194). Taking that advice makes the 7.752 GiB of host-side experts ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure and not a cache you can squeeze. Bandwidth-driven, on the host side: the point is the rate the CPU reads those experts at during decode.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632 — and it costs more on this architecture than on a plain transformer. See System thrashing. Capacity-driven, but on host RAM rather than on VRAM — it does not appear anywhere in the card's budget.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05), whose text notes that "Apodex follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction: "Do not inline tool descriptions in the system prompt — pass them via tools=".

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It appends one buffer-type override per block index 0 … N-1, each built from a regex in common/common.h:1130:

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, LLM_FFN_EXPS_REGEX, params.tensor_buft_overrides)

The loader matches that against tensor names and, on a hit, gives the tensor a CPU buffer type at tensor-creation time, before any buffer exists — so those bytes are allocated in host memory and never enter VRAM at all. Nothing in the path consults the architecture, which is why it works identically on this hybrid recurrent model and on a plain transformer MoE. 123 tensors in this file match, three per block across all 41 blocks, totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads.

Three things it does not move, all of which stay on the card and all of which show up in the decode budget later:

• The attention and Gated-DeltaNet weights. A block's attn_* tensors and its ssm_* tensors do not match _exps.
• The shared expert and the router. ffn_*_shexp and ffn_gate_inp do not match either, so the dense path every token takes stays resident, and routing is decided on the GPU.
• The KV cache and the recurrent state. Both take their buffer type from the layer's device, which -ngl alone decides; -ncmoe rewrites weight tensors and never touches the layer assignment. A block whose experts are in RAM still keeps its recurrent state in VRAM — which is what makes the budget below decomposable at all.

Offloading everything the regex can reach, -ncmoe 40, still leaves 1.744 GiB resident: 0.655 GiB of token_embd, output and output_norm, and 1.089 GiB of everything else across forty blocks.

Does that actually happen on this architecture? The source says so, and there is an independent report that the offload takes effect on a hybrid MoE: llama.cpp issue #27698 records a run with --fit off --n-gpu-layers 46 --n-cpu-moe 6 where "the weights load (expert offload is honored)" and the failure comes later. ⚠️ That report is an RX 6700 XT (gfx1031, RDNA2) on ROCm and the branch with the honoured offload is a gemma4moe model rather than this one, so nothing about its failure transfers to this card — it is cited for the one clause quoted, plus a magnitude in the next section.

The VRAM budget

Weights, KV cache and recurrent state at the recipe's settings. Every figure is derived from the GGUF's own tensor table and from llama.cpp's allocation code at b10666.

The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 17 moves 8,323,596,288 of that (7.752 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks load; full_attention_interval 4 makes ten of them full-attention — blocks 3, 7, 11, 15, 19, 23, 27, 31, 35, 39. Each carries head_count_kv 2 at key_length and value_length 256, so 1,024 cache elements per token per layer and 10,240 across the ten. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows with context.
• Recurrent state. Those thirty blocks each allocate n_embd_r() = (ssm_d_conv − 1) × (ssm_d_inner + 2 × ssm_n_group × ssm_d_state) = 3 × (4096 + 2×16×128) = 24,576 elements and n_embd_s() = ssm_d_state × ssm_d_inner = 128 × 4096 = 524,288 elements, both F32 (src/llama-hparams.cpp:183,211). That is 2,195,456 bytes per layer per sequence and 65,863,680 across thirty. llama-server prints it at startup as RS buffer size.

The two terms that table cannot produce, and what this page reserves for them

13.546 GiB is not a peak. Two things sit outside it, and on a card this tight they decide whether a configuration loads.

1. A 16 GB card does not give you 16 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. The only expert-offload run in our catalogue with the numbers written down is gemma4-26b on an RTX 3060 at -ncmoe 12, whose submitter recorded a peak of "11,179 of the card's 12,288 MiB" on a headless box — so at most 1,109 MiB was unavailable to the model there. ⚠️ That is a different board in a different VRAM tier, on Ampere rather than Blackwell, and the term is a context plus a driver reservation rather than a percentage, so borrowing the absolute 1,109 MiB here is a transfer of an absolute quantity and not a measurement of this one. It is the best number in the catalogue and it is the direction this page errs in: this page reserves 1,109 MiB, and a Windows box driving a monitor should expect to want more.

2. The compute buffers. llama.cpp allocates graph working memory on top of everything above, and under expert offload it also stages host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 0.4219 or 0.4863 GiB, so that staging term alone is of that order. This page reserves 1024 MiB, which is llama.cpp's own per-device auto-fit margin: fit_params_target is initialised to std::vector<size_t>(llama_max_devices(), 1024 * 1024*1024) at common/common.h:481. It is the runtime's opinion of how much room it wants, not a figure this page invented. For scale on leaving nothing: #27698 records a clean weight load followed by context creation failing on a ~504 MiB compute buffer with cudaMalloc failed: out of memory — a different card, backend and model, quoted only for the magnitude of a term every table in this family excludes.

⚠️ One item inside that reserve is bigger on this card than on an Ada one, and it is worth knowing the direction. llama.cpp sizes its cuBLAS workspace by compute capability: cublas_workspace_sizes[device] = (cc >= GGML_CUDA_CC_HOPPER) ? 32 * 1024 * 1024 : 4 * 1024 * 1024; at common.cuh:1509, allocated per device per stream. Blackwell is 1200 and Hopper is 900, so this card takes the 32 MiB branch where an RTX 4080 or 4060 Ti takes the 4 MiB one. It is 28 MiB per handle — small against a 1024 MiB reserve, and it is the only term on this page that is larger here purely because the card is newer.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 16,384 − 1,109 − 1,024 MiB = 14,251 MiB = 13.917 GiB.

At -ncmoe 17 and 131,072 tokens the sub-total is 13.546 GiB, which clears that by 0.371 GiB.

⚠️ This is a stricter reserve than our published page for a fully resident model on this same card, and the difference is deliberate rather than a disagreement: Qwen3.8-27B on RTX 5070 Ti reserved 1.410 GiB when it was written, for a model that lives entirely on the card. An offloaded run additionally stages expert tensors on the GPU during prefill, which a resident model never does, so it needs more room for the same nominal context.

Read your own numbers off the server's startup log rather than trusting mine: llama.cpp prints model buffer size once per buffer, so under -ncmoe you get a CUDA line and a host line and the split above is directly checkable, followed by KV buffer size, RS buffer size and compute buffer size. If you do, those figures are the most useful thing you could send us — more useful than a speed number, because they are the two terms nobody in this family of recipes has been able to derive.

Choosing your own -ncmoe

Every row is weights + KV + recurrent state, q8_0 KV, one slot. Free of 16 GiB is what the naive arithmetic says; Slack is what is left once the driver share and the compute reserve are taken out, and the two columns disagree by more than 2 GiB, which is the whole point of the previous section.

Two rows are worth reading twice. -ncmoe 16 at 131,072 tokens has 1.968 GiB "free" and should not be attempted — that is what the ceiling is for. And -ncmoe 17 is the recipe's setting because it is the lowest rung that clears the ceiling at this context, which is a derivation and not a preference: 16 misses by 0.115 GiB. Going deeper is always safe and always slower; the bottom row is the model's entire declared 262,144-token window, which this card can hold at -ncmoe 20.

Then stop guessing and measure it. llama.cpp ships a tool that answers this on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 131072

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 131072 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device split of total / free / model / context / compute / unaccounted. That compute column is what this page reserved 1024 MiB for, and unaccounted is your driver's share. ⚠️ On this card there is one reason to sanity-check whatever those tools report against the number on the box — see llama.cpp thinks this card has far more memory than it does.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys X GB" is wrong here. bartowski's recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28, 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates. Read the cumulative column above rather than multiplying. Nine of the seventeen blocks this recipe offloads are the expensive kind, which is why -ncmoe 17 moves 7.752 GiB rather than seventeen equal shares of anything.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 17.955 GiB on disk and its ladder is linear. If you would rather spend quality than offload depth, that is the trade — but derive its table from its own tensor list; do not scale this one.

Where the decode time actually goes

This is the section that is not the same on another 16 GB card, and it is why the capacity numbers above are interchangeable across the tier while the performance advice is not.

The mechanism first. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights" (ggml/src/ggml-backend.cpp:942) — unless the scheduler finds "a backend with higher prio wants to offload the op" (:960). CUDA answers that with a single comparison:

// ggml/src/ggml-cuda/ggml-cuda.cu:5341-5344 at b10666
static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
    ...
    return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size;
}
// :5515 — const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;

Generating a token is a batch of one, and one is less than 32, so at decode the offloaded expert matmuls stay on the CPU and those weights never cross PCIe at all. Prompt processing is a batch of hundreds, so at prefill those same tensors are copied host→device. The two phases are bound by different things and this page treats them separately.

The decode roofline, in bytes

expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token. Everything else on the card is read whole. At -ncmoe 17:

Two rows there deserve a note.

The KV cache is decode traffic, not just decode capacity. Every generated token reads the whole occupied cache back out of VRAM. At a full 131,072-token window that is 1,426,063,360 bytes — the single largest term in the table, larger than every weight the card holds put together. It is easy to price the KV cache as a capacity line and leave it out of the bandwidth arithmetic — the budget table above treats it as capacity — and on this card leaving it out changes the answer rather than merely rounding it.

The output.weight head is separated from the embedding table. They are usually lumped as "0.655 GiB of embeddings", but only one of them is read per token: output.weight is 417,177,600 bytes of Q6_K read in full to produce logits over 248,320 tokens, while token_embd.weight is read one row at a time — 1,152 bytes, since 2,048 Q4_K elements are eight 144-byte superblocks. Splitting them takes the non-expert VRAM read from a loose upper bound of 1.744 GiB down to a near-exact 1.4776 GiB.

The crossover, and where this card sits

The two halves of a decode step take equal wall-clock exactly when the card's bandwidth is VRAM bytes ÷ host bytes times your DIMMs'. That ratio needs no bandwidth figure at all, which is why it is the form this page states first. Every ratio here includes the recurrent state, and the second and third rows add the KV cache at the stated occupancy — a crossover is only meaningful with its terms named, and this page's are those.

Those three numbers are properties of this file at this offload depth, not of the card — an RTX 4080 and an RTX 4060 Ti at -ncmoe 17 have exactly the same three. What the card decides is where its own ratio falls against them. Using the 896 GB/sec NVIDIA publishes for this board:

Read that table by row and then by column, because the two readings are different advice. On DDR4 this card behaves like every other offload page in the family: the host is the slower half at every depth, and faster DIMMs are the upgrade. On DDR5 it does not. Solve for the boundary directly and the depth is what decides it:

At this recipe's own window, ~4,250 MT/s dual-channel is below the slowest DDR5 that ships. So on any DDR5 desktop, at full context, the RTX 5070 Ti's own memory is the larger half of a decode step — and the reason is not that the card is slow. It is that 896 GB/s is fast enough to make 1.328 GiB of KV re-read per token the cheap half of nothing.

Priced out as a roofline — the two halves treated as serial, which they broadly are, since within a layer the CPU's expert matmul consumes what the GPU's router produced:

⚠️ That last column is a ceiling from two bandwidth terms, and you will not reach it. It excludes the CPU time to actually multiply the experts — which on this path is the work, not an overhead — plus kernel launch latency, attention arithmetic, sampling, and every scheduling gap between the two halves, and it credits both memories with perfect efficiency. For a sense of the gap, the one expert-offload run in our catalogue with real numbers, gemma4-26b on an RTX 3060 at -ncmoe 12 on DDR4-2133, measured 37.23 tok/s. Read the column as an upper bound that no configuration can beat, never as a rate to expect.

The practical consequence, stated as a purchase decision. Moving from DDR4-3200 to DDR5-6000 improves this card's 131K bandwidth ceiling by 1.36×; moving from DDR5-6000 to DDR5-8000 improves it by 1.12×, because by then the card is already doing 58% of the work. Faster DIMMs pay well on this card up to DDR5, and then stop paying. The lever that helps the VRAM half is already in the run command: at a full window -ctk q8_0 -ctv q8_0 reads 1,426,063,360 bytes per token where f16 would read 2,684,354,560, so the flag removes 1,258,291,200 bytes — 1.40 ms. The 3.83 ms above is the figure with the flag already applied; without it the VRAM half is 5.23 ms.

The offload depth is the smallest lever, and it is priced per step. Moving one block from card to host frees its whole expert triple from VRAM — 0.4219 or 0.4863 GiB — and adds one thirty-second of it to every token's host read: 13.5 MiB for a cheap block, 15.6 MiB for an expensive one. On dual-channel DDR5-6000 that is 0.15–0.17 ms per token added to the host side against 0.02 ms saved on the card side. Going from 17 to 20 for the full 262,144-token window costs about 0.5 ms per token in offload and adds 1.328 GiB of KV traffic, about 1.59 ms — three times more. At this recipe's depth, context is a far more expensive throughput decision than offload depth is.

Prefill is the phase the link pays for

At batch sizes of 32 or more the scheduler hands the expert matmuls to the GPU, which means copying host-resident expert tensors across PCIe. llama.cpp issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0", "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26" — and it finds "prefill is bottlenecked by PCIe transfers, not GPU compute", with the GPU idle around 42% of each pass on expert H2D copies that run serially with the matmuls consuming them. ⚠️ Carry its caveats: a different model, one box, gen-4-specific, and the author dates the measurement to "around the b6xxx era", so the idle fraction may be stale. It is the strongest evidence in circulation that the shape of this path is real, and it is not a measurement of Apodex.

If the whole offloaded set crosses per physical batch — the shape that report describes — then at -ncmoe 17 that is 8,323,596,288 bytes per -ub batch, and -ub defaults to 512 (common/common.h:452; -b defaults to 2048 at :451). This is the one place where this card's newer bus is worth something on this recipe. NVIDIA publishes PCI Express Gen 5 for this board, which carries about 3.94 GB/s per lane after 128b/130b encoding, so the transfer term alone bounds prompt processing at roughly:

Theoretical link rates, with real achieved throughput lower. The card being Gen 5 does not make your slot Gen 5, and this page asserts no lane count for the board either — NVIDIA publishes the generation only. Read your own from the nvidia-smi line under Requirements: if gen.current reads 4 where gen.max reads 5, you are on the third row, silently. Either way it is a time-to-first-token problem and not a tokens-per-second one, because at decode batch 1 those weights do not move. If the link turns out to be your constraint, --no-op-offload disables the offload rule globally, keeping the expert matmuls on the CPU in both phases and removing the transfers entirely — a real trade, untested here, and one flag to try.

Does the arithmetic survive contact with a real card?

Nobody has run Apodex on an RTX 5070 Ti, so the honest answer is that this page is derived and the derivation has three partial checks, none of them on this pair.

The offload ladder has a measured analogue two tiers down. gemma4-26b on an RTX 3060 is the only expert-offload run in our catalogue with a written-down ladder: -ncmoe 12 gave 37.23 tok/s at an 11,179 MiB peak and -ncmoe 30 gave 21.20 tok/s on the same unchanged card — eighteen more blocks on the host path costing 43% of the generation rate — with -ncmoe 8 and below failing to load outright. Different model, different card, DDR4-2133 dual channel, headless, llama.cpp b10088. The direction and the magnitude of the effect are the point; none of those numbers is yours.

The prefill claim has the profiling report above, with the caveats stated there.

And the 131,072-token window has a control run on this model's base architecture. llama.cpp issue #27756 is a long-context failure report against Qwen3.8-27B — a different 64-layer member of this family, 48 of whose layers are Gated-DeltaNet — which emits EOS as its first generated token beyond roughly 98–130k positions on CUDA, on CPU and on a fork. Its reporter ran a cross-model control: "The same needle test at 243k tokens PASSES on Qwen3.5-35B-A3B (30 GDN layers) on the same build, same server, same machine" — which is exactly this model's base and exactly this model's thirty Gated-DeltaNet layers. ⚠️ It is one box, one control run reported in passing, a needle-recall test rather than a quality evaluation, and the base model rather than this fine-tune. It is not a guarantee. It is the only evidence found in the space searched (the llama.cpp tracker and the 22 model cards enumerated under Results) that this architecture's recurrent state holds up past the window this page documents, and it points the right way.

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this GGUF, under stock llama.cpp, on any NVIDIA card, at any offload depth exists in the space searched on 2026-08-31. That space was: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces, re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100 and fetched one by one (21 of the 22 have a card at all; aykutx21/Apodex-1.1-mini-GGUF returns 404 for its README); the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 273 for qwen35moe and 729 for n-cpu-moe on the same query shape; and our own /check, which returns unknown with zero benchmarks. Figures for the model do exist, and every one is a different artifact, a different engine, or both:

  • kingjones777 publishes four of his own repacks at 64.87 / 63.30 / 45.05 / 32.54 tok/s, in ROCmFP4/ROCmFPX tensor formats that exist only in the ROCmFPX fork of llama.cpp — his own card says so under a heading reading "Stock llama.cpp will not load this file" — on a Ryzen AI MAX+ 395 (gfx1151) unified-memory APU at full offload — a machine with no expert offload and no PCIe hop, which is the entire subject of this page. Read the top of that range as unsettled: that build's own card states the run is still pending on an idle box while the shared variants table prints 64.87 for it, under a caption reading "A dash means I haven't measured that one yet". The same cards also carry the only partial-offload numbers anywhere for this model, and they are not what they look like — see Troubleshooting.
  • ghazni101 reports "generation measured at 226 tok/s end-to-end for MQ4R". Three disqualifications, all verified: it is a .mq4r file under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF; and it is an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim.

  One measurement on this pair would be worth more than all of the above. If you run it, contribute it.

• VRAM usage: 13.546 GiB derived sub-total at -ncmoe 17 and 131,072 tokens, plus the ~1,109 MiB driver share and 1024 MiB compute reserve this page sizes against but cannot derive. 7.752 GiB of expert weights sit in system RAM alongside. See /check/apodex-1-1-mini/rtx-5070-ti for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: decode reads 0.26 GB from your DIMMs and 2.0–3.4 GB from the card per token. On a DDR4 box your memory sets the pace; on a DDR5 box, at this recipe's context depth, the card does. Prefill is the phase the host link is on the critical path for, and it is the only phase where the board's PCIe generation buys anything on this recipe.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512), which is part of why the sub-4-bit rungs give up more than their file sizes suggest. The lead quant is imatrix-calibrated and the publisher ships both the calibration corpus and the imatrix beside the ladder, which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-5070-ti.

Troubleshooting

The GPU locks up minutes into a long agent session

This is the best-corroborated hazard on this card, and it is not about this model. llama.cpp issue #27330 — open, four comments, last updated 2026-08-29 — reports that under sustained inference the GPU stops servicing a command channel, the NVIDIA driver's Robust Channel watchdog fires after seven seconds, and the kernel logs NVRM: Xid (…): 8, name=llama-server. The server dies with CUDA error: the launch timed out and was terminated, usually inside ggml_backend_cuda_synchronize.

Four things make it worth leading with here:

1. It is Blackwell. The original report is an "NVIDIA GeForce RTX 5090 Laptop GPU (GB203M), 24463 MiB, compute capability 12.0" — the mobile part of the same GB203 die this board uses — and it has since been reproduced on an RTX PRO 4000 Blackwell, on two desktop RTX 5090s, and on an RTX 5080. In the last of those the reporter runs a mixed rig and states the discriminant outright: "The failing GPU is always the 5080 (device 1, Blackwell); the 4090 (Ada) never reports an error", adding that it began when they replaced "a RTX 4060 Ti 16G with a RTX 5080".
2. It is this model's sibling architecture. Every reproduction is Qwen3.8-27B, arch qwen35 — hybrid attention plus SSM, the dense member of the family this MoE belongs to.
3. The workaround is A/B tested, not guessed. The reporter's table has one row that settles the mechanism: a process with kernel fusion enabled and only CUDA graphs disabled ran "32+ minutes clean and still running", against a graphs-on failure rate the reporter gives as a minimum of 4 minutes, a mean of 8.6 and a maximum of 15, "Across 11 observed hangs". "So this is not the fusion path, it is the graph path." MTP, KV quantisation, thermals, power and hardware faults are each ruled out in the body.
4. The trigger is this recipe's workload. The reproduction steps are "Drive it with a real coding agent workload (I used OpenCode) so the slot stays busy with long prompts and long generations." and then, as the next step, "The hang is stochastic, not deterministic. It never happens on the first request."

⚠️ One scope limit, on the axis this recipe differs on: none of the five reports is an expert-offload run. Every one of them loads the model whole — -ngl 999, or --tensor-split 3,1 across two cards — and the strings --n-cpu-moe, -ncmoe, --cpu-moe and --override-tensor appear zero times across the issue body and all four comments (checked 2026-08-31; the same grep returns four hits on #26609, and the flags these reporters do pass are plainly present, so the pattern can match this corpus). What that does to the transfer is genuinely two-sided and unresolved: expert offload puts a CPU backend into the graph on every layer, which changes what the CUDA graph capture spans, and nobody has reported either outcome. Read it as a hazard whose trigger conditions on this card are known and whose interaction with -ncmoe is not.

The fix, if you hit it:

GGML_CUDA_DISABLE_GRAPHS=1 ./build/bin/llama-server -m ./apodex-1.1-mini/... # rest as above

⚠️ Unset that variable to turn graphs back on. Do not set it to 0. ggml/src/ggml-cuda/common.cuh:1258 reads static const bool disable_cuda_graphs_due_to_env = (getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr); — presence, not value — so =0 disables CUDA graphs exactly as thoroughly as =1 does. And note that graphs are on by default in a llama.cpp CUDA build: the root CMakeLists.txt:169-170 sets GGML_CUDA_GRAPHS_DEFAULT to ON when the option is not already defined.

Why it is not in the run command above. It costs decode throughput, every reproduction is on a different architecture string, and nobody has hit it on qwen35moe — because, as Results records, nobody has run this model on a Blackwell card at all. Run without it, and reach for it the moment your session locks up rather than suspecting the model.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE, which arrived in (b7989, b7990] — so this means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

An illegal memory access on the flash-attention path

llama.cpp issue #26609 — open, four comments, last updated 2026-08-29 — is the best-isolated defect anywhere on this exact architecture, and its hardware is one model name from this card. llama-server dies with CUDA error: an illegal memory access was encountered in ggml_backend_cuda_synchronize while processing a request, deterministically, on Qwen3.6-35B-A3B-UD-Q4_K_M (arch qwen35moe) with partial expert offload.

What the reporter's isolation matrix establishes — one variable per probe, same request sequence, on "Windows 11 Pro, NVIDIA GeForce RTX 5070 12 GB":

• Flash attention is the discriminant. "--flash-attn off (which also forces KV f16 — declared double axis) passes 3/3 on both builds."
• The offload is not the cause. Probe D moved all experts to CPU and probe E used whole-layer offload with no --override-tensor at all; both still crashed.
• The build is not the cause. "Still reproducible on b10488 (commit 9d77fa172, today's official win-cuda-13.3-x64 release) — not fixed as of that build."
• It is strongly input-dependent. "a synthetic prompt of the same length and block structure, and even a shape-preserving scramble of the triggering prompt (every alphanumeric word replaced by a random same-class, same-length word, punctuation/whitespace untouched), both PASS on the exact configuration where the real prompt crashes deterministically."
• GGML_CUDA_DISABLE_GRAPHS=1 does not prevent it — which is what separates it from #27330 above, a different failure with a different fix.

Scope it honestly, in both directions. The reporting card is an RTX 5070 12 GB: same Blackwell generation and the same compute capability 12.0 as this board, a different die and a different capacity. But a second commenter hits the same signature on an RTX 3070 Laptop 8 GB — Ampere — running --cpu-moe on self-built current master, "spottily in a way I can't make deterministic". So this is not a Blackwell defect; it is a qwen35moe plus CUDA-FlashAttention defect that happens to have been isolated on a Blackwell card. And the highest build tested is b10488, which is below the b10630 this page pins, so its status at this pin is unknown — not fixed, and not confirmed broken. If you reproduce it at b10630 or later, that is genuinely new information for the thread and for us.

What -fa off actually costs on this card

If you take that workaround, price it before you type it, because it is not just a KV-cache change.

It changes the graph. On the non-FA path src/llama-graph.cpp:2607 builds ggml_tensor * kq = ggml_mul_mat(ctx0, k, q); and :2612 immediately calls ggml_mul_mat_set_prec(kq, GGML_PREC_F32). That tensor is sized n_kv × n_tokens × n_head at F32 with this model's head_count of 16, and the worst-case graph is reserved at n_tokens = std::min(cparams.n_ctx, cparams.n_ubatch) (src/llama-context.cpp:595). So:

At this recipe's context with the default -ub, that one tensor is 4.000 GiB — four times the entire compute reserve, before the mask, the softmax or expert staging. The fallback does not allocate unless you shrink -ub as well.

And it costs less on the cache than you would guess. -fa off forces only the V half to f16, not both: src/llama-context.cpp:3675 rejects a quantised V without flash attention, while the K check at :3686 is gated if (params.flash_attn_type != LLAMA_FLASH_ATTN_TYPE_DISABLED && …) and never runs when FA is off. So -ctk q8_0 -ctv f16 is legal at 15,680 bytes per token — 1.914 GiB at 131,072 rather than the 2.500 that f16 on both sides would cost.

Three configurations that actually fit, all with -ctk q8_0 -ctv f16:

Note what the last row costs: keeping the full window under -fa off means three more steps of -ncmoe (9.082 GiB in host RAM instead of 7.752) and a physical batch of 64, which will make prompt processing markedly slower. If you can live at 32,768 tokens, the first row is much the better trade.

SOFT_MAX failed on a Blackwell build

Relevant mainly if you took the -fa off route, since that is what puts a softmax kernel in the attention path. llama.cpp issue #25060 — open, bug-unconfirmed, five comments, last updated 2026-08-20 — collects SOFT_MAX failed crashes on Blackwell parts, and two commenters independently trace them to a mismatched CUDA toolkit and runtime rather than to the GPU: cudaGetDeviceProperties returns a nonsense sharedMemPerBlockOptin, llama.cpp hands it to cudaFuncSetAttribute, the call is rejected, and any kernel asking for shared memory dies. Check CMakeCache.txt for two different CUDA paths and rebuild with -DCUDAToolkit_ROOT= pointed at one of them. ⚠️ One commenter on an RTX 5070 Ti Laptop probed the value, found it correct, and still crashed — so a clean probe does not clear you, and the reports are on other SM 12.0 boards rather than on this one.

llama.cpp thinks this card has far more memory than it does

llama.cpp issue #26901 — open, zero comments, filed 2026-08-11 — is the only tracker entry that names this exact board, and it is worth knowing about because of what it does to the auto-fitter. Its title is "Blackwell discrete GPUs (RTX 5080/5070Ti) misclassified as iGPU — breaks multi-GPU tensor split on Linux", and the reporter's second consequence is the one that reaches a single-card reader: "the UMA memory path reads system RAM (~77 GB on a 96 GB system) instead of actual VRAM (15.8 GB), causing common_params_fit_impl to believe a single GPU has enough memory for the entire model."

The code path is real and checkable at the pin. ggml_backend_cuda_device_get_type returns prop.integrated ? GGML_BACKEND_DEVICE_TYPE_IGPU : GGML_BACKEND_DEVICE_TYPE_GPU at ggml/src/ggml-cuda/ggml-cuda.cu:4820-4822, and ggml_backend_cuda_device_get_memory computes bool is_uma = prop.integrated > 0 || uma_env; at :4793 and then, if (is_uma), replaces the free-memory figure with MemAvailable from /proc/meminfo — under the comment "For UMA systems (like DGX Spark), use system memory info". That whole branch is guarded #if defined(__linux__) && !defined(GGML_USE_HIP) at :4786, so it is a Linux + CUDA behaviour only.

⚠️ What is verified and what is not. The two source behaviours above were read at b10666 and are exactly as described. That a consumer Blackwell card sets prop.integrated > 0 is the reporter's claim, uncorroborated by anyone in the thread, and this page cannot test it. Treat it as a thing to check rather than as a property of your machine.

Why it matters here. This recipe's entire subject is how much of the model to leave on the host. If the fitter is reading your system RAM as the card's free memory, then leaving -ncmoe off and letting auto-fit decide is not merely opaque — it is answering the question with the wrong number. Pinning -ncmoe by hand, as above, sidesteps it completely. If you want to check, common/fit.cpp:349 logs a line of the form "projected to use" N "MiB of device memory vs." M "MiB of free device memory" at trace level, so run once with -lv 4 and confirm that the second figure looks like a 16 GB card and not like your DIMMs.

It loads, then runs out of memory when a long prompt lands

Two causes with opposite fixes.

The KV cache grows at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the ladder and either raise -ncmoe or lower -c. On this file, raising -ncmoe by two frees between 0.844 and 0.973 GiB depending on which blocks the steps land on.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means staging those tensors there. Each of this file's expert triples is 0.4219–0.4863 GiB, so prefill peaks materially above the generation footprint the budget table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those ops on the CPU where their weights already are; expect slower prefill in exchange. If the card is also driving a display, start one or two steps deeper than the table and read nvidia-smi after load.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it.

The first is the offload: at -ncmoe 17 this recipe puts 7.752 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache, on by default at 8192 MiB — cache_ram_mib = 8192 at common/common.h:632, exposed as -cram / --cache-ram at common/arg.cpp:1713 with -1 for no limit and 0 to disable, added by PR #16391. So a default server may hold up to 8 GiB beyond the offload — close to 16 GiB of host memory before the operating system gets any, which is why the command above pins it to 2.

There is one direct measurement of that growth on this architecture. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. Closed as completed, 2026-08-28. Two cautions if you cite it: the 126 MiB is the only measured per-entry figure anywhere and it is roughly twice one of this model's recurrent states (65,863,680 bytes = 62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support even though the 126 stands as a measurement; and it was measured on a different model of the same architecture, with a Vulkan rebuild giving +126.0 MiB per request against ROCm's +126.3 — so this is a model-and-request-layer behaviour and not a vendor story.

So 32 GB is the practical floor for this recipe as written and 16 GB is not enough. On a 32 GB box keep -cram 2048, or pass -cram 0 to disable caching entirely and trade prompt-reuse speed for headroom.

Generation is slower than you expected

Check three things, and on this card which one matters depends on your DIMMs.

Your memory configuration, first. Single-channel memory doubles the host half of every decode step. Confirm dual channel before anything else; it is the cheapest fix. If you are on DDR4, it is also the largest one.

Your context depth, second, and on DDR5 it is the largest. At a full 131,072-token window the KV cache is 1,426,063,360 bytes of VRAM traffic per generated token — larger than every weight on the card put together. Generation slowing down as a conversation grows is not degradation, it is the roofline. If your agent loop never fills a six-figure window, -c 65536 at -ncmoe 16 is a row in the ladder above and halves that term.

Your offload depth, third and smallest. Each step is 13.5–15.6 MiB more read from RAM per token — about 0.15 ms on dual-channel DDR5-6000. Do not go deeper than the ladder requires, but do not expect much back from going shallower either.

If it is specifically the wait before the first token that is bad while generation is fine, that is the bus and not the model — see Prefill is the phase the link pays for.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true in common/common.h — and moves MoE tensors to system memory until it can leave its 1024 MiB per-device margin.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp:484 throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings.

⚠️ That warning is benign here for a specific reason, and it is not benign in general. It is safe on this page because every parameter the fitter would have chosen is one the command above sets explicitly — the offload depth, the context, the slot count and the cache types. llama.cpp issue #27872, filed 2026-08-28 and uncommented, is what happens when that is not true: a reader passes --cpu-moe, the fitter bails on the same message, "the server continues loading with the original, unfitted parameters", and the run dies later at compute-buffer allocation with cudaMalloc failed: out of memory — "the actual failure is much less legible than the root cause". Different card and a different architecture, quoted for the failure shape. Seeing that warning means your overrides are being honoured; it does not mean anything else you were relying on the fitter for still is.

There is also an open regression report against the fitter on this architecture family — llama.cpp issue #27171, filed 2026-08-16 against Qwen3.6-35B-A3B Q4_K_M with --fit-target, still open and labelled bug-unconfirmed as of 2026-08-31, with a proposed fix (PR #27207) still open and unmerged. And on this card there is the memory-reporting question above. Between the three, pin the number by hand.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp drives it with --spec-type draft-mtp — the exact spelling matters. On this card, price it first, because every cost lands on the 0.371 GiB of slack this page reserved:

• blk.40 stops being skipped: +0.444 GiB, and -ncmoe 17 does not touch it — the flag installs overrides for blk.0 through blk.16 only.
• The recurrent state is multiplied by the draft depth: need_n_rs_seq() at common/common.h:394 returns draft.n_max when a draft-MTP type is selected, so --spec-draft-n-max 3 takes 0.061 GiB to 0.184.
• A second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k — so at 131,072 tokens on one layer that is 2,048 bytes per token, 0.250 GiB.

That is 0.817 GiB against 0.371 GiB of slack, so on this configuration it does not fit at all; you would be raising -ncmoe to pay for it, before any throughput has arrived.

And the return is unknown. Unsloth's published guidance for this class of model is that "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs) — 1.15–1.25× at best, before offload is counted. Under offload even that is contested: an owner describing their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" running a different model of this architecture at --n-cpu-moe 25 with the MTP flags on reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" (discussion #18 on that model's GGUF repo), and a second community member there offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model.

⚠️ Two reports in this area are worth knowing about, and neither is on this pair. llama.cpp #25318 — open, filed 2026-07-05 — reports that on an "NVIDIA GeForce RTX 5070 (Blackwell, 12 GB)" running an MTP GGUF at --spec-type draft-mtp, "Running MTP (Multi-Token Prediction) GGUF models causes the NVIDIA driver to lose the GPU completely." and "The GPU becomes unrecoverable until the machine is rebooted.", while the same box runs non-MTP models without issue — a different model and a different board, on a Blackwell card. And #27717 — open, six comments — records --n-cpu-moe below a threshold crashing the MTP draft load outright with invalid vector subscript, bisected to "between 27 and 28 on this setup" on an RTX 3060 with a nemotron_h_moe model. Different architecture, different card, and the point is only that -ncmoe and MTP interact in ways nobody has characterised on this file.

The 1.183 tok/s figure you will find, and should not use

Searching for "Apodex partial offload" turns up a number that looks catastrophic and is not a measurement of anything you are about to do. Three of kingjones777's model cards publish per-card functional checks of 1.183 t/s, 0.292 t/s and 0.163 t/s, and the same author retracts all three in the same documents:

"My first published number (1.183 tok/s) was measured with PARTIAL offload on a box already serving 8 models — that was my harness's fault, not the model's."

His original write-up is equally explicit: the run is "17 / 40 layers on ROCm0, rest CPU-mmap (-ngl 17 -c 2048)" on a box where "8 live llama-server seats" held about 107 of 125 GiB of unified memory, and he adds that "These t/s numbers are limited by streaming the CPU-resident layers, not by the ROCm path — treat them as load-and-generate proof with the server's own timing, not as the speed you will get on an idle box."

-ngl 17 is not -ncmoe 17, and the resemblance is the trap. -ngl 17 puts seventeen whole layers on the GPU and leaves twenty-three entire layers — attention, Gated-DeltaNet, shared experts and all — on the CPU behind mmap. -ncmoe 17 keeps all forty layers on the GPU and moves only the routed experts of the first seventeen blocks to host RAM. Those are different configurations that happen to share a number.

The general form of this trap is worth carrying, because an offload page invites it. llama.cpp #27623 is titled "decode throughput collapses ~25x at context" and is retracted by its own reporter in the sixth of its six comments: "My measurement metric was flawed. The numbers in my matrix were completion_tokens / total request time, which includes prompt processing (~43s for a 68K fill at ~1500 t/s pp)." — which, he goes on to say in the same sentence, is why every configuration looked as though it had collapsed to about 1.4 t/s when what was being measured was prefill time rather than decode. The issue is still open, so the alarming headline is what a search shows you first. On this recipe prefill is bus-bound and slow while decode is not, so dividing generated tokens by wall time makes decode look broken when only prefill was. Read llama-server's own predicted_per_second instead, and if it disagrees with what you expected, tell us — it would be the first number anyone has for this pair.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps's hardware table names "IQ2_M or IQ3_M" for a 16 GB card, with "IQ2_M full offload; IQ3_M may require hybrid offload". Weigh it against the rest of that card, which is candid: "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." and "Prefer Q4_K_M or higher when system memory permits."

Check the arithmetic before taking it. Against this page's 13.917 GiB ceiling, IQ2_M at 11.682 GiB in the lead repo leaves room for a 131,072-token q8_0 cache and the recurrent state with 0.846 GiB to spare, so it genuinely escapes offload — and it is a 2-bit quantisation of a model whose routed experts are 512 wide, used for tool calling where a malformed argument is a failed task. Q2_K_L at 12.650 GiB does the same at 65,536 tokens. Below Q4 the trade on this card is quality for no -ncmoe, which is a cleaner trade than the 12 GB tier gets, and it is still a large quality bet to avoid a memory purchase.

Choosing a different quant publisher

Conversions of this model are still appearing: an enumeration on 2026-08-31 returned 22 repositories matching Apodex-1.1-mini across 11 publishers, unchanged in count from the previous day but with a new publisher having appeared the day before that. Re-enumerate before you decide, and check the byte count and the file list rather than the tier name — across publishers a file named Q4_K_M spans more than a gigabyte for the same nominal tier. Within bartowski's own ladder the rungs adjacent to the lead are Q4_K_S at 19.619 GiB and IQ4_XS at 17.955 GiB; each buys back host RAM rather than VRAM, which on this card is the resource you have most of.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, and on a Blackwell card the first of those looks like the obvious answer. It is not, for three separate reasons. They are vLLM/SGLang safetensors rather than GGUF and llama.cpp cannot load them at all. They are far too large for this card before anything else is counted — 22.548 GiB and 22.958 GiB of weights for the first two, against 16 GiB. And the -NVFP4 repo is not a 4-bit checkpoint anyway: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantised layer entries, of which 250 are FP8 and only 40 — the routed experts — are NVFP4, with kv_cache_quant_algo FP8 as well (counted from the file on 2026-08-31). Both formats are native on Blackwell, so the design is sound; it is simply not what the repo name says. Note also that their Quick Starts launch on a single GPU: --tp-size 1 on both -NVFP4 and -GPTQ-Int4 and --tp 1 on -FP8; the --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone. For one RTX 5070 Ti the GGUF route above is the path.

Other pages worth reading

Apodex 1.1 mini on RTX 3090 fits the whole model on the card at this same 131,072-token context with none of this page's arithmetic, and Apodex 1.1 mini on RTX 5090 does it at a higher quant and the full 262,144-token window. One caveat if you read either. llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label — verified 2026-08-31. Anything written about that issue before 2026-08-30 will call it open, including this site's own earlier pages for this model; that dates the text rather than changing the advice, and it never applied to a text-only run, which is what --no-mmproj makes this one.

For a resident-model contrast on this exact board, Qwen3.8-27B on RTX 5070 Ti is a dense model of similar size that fits entirely in 16 GB — the useful comparison being that a dense model reads all of its weights on every token where this one reads one thirty-second of its experts, which is why an offloaded 36B can plausibly out-decode a resident 27B on the same card.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex, /apodex-1.1-mini and /apodex1.1 all return 404, and the registry.ollama.ai manifest for apodex returns 404 as well — checked 2026-08-31, with ollama.com/library/qwen3.5, ollama.com/library/qwen3 and the registry.ollama.ai manifest for qwen3 all returning 200 as controls. Use llama.cpp directly. If a tag appears later, please tell us.