Apodex 1.1 mini on RTX 4070 Ti Super: a 36B agent in 16 GB via llama.cpp expert offload

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single RTX 4070 Ti Super, at a 131,072-token working context, from a Q4_K_M GGUF that is larger than the card.

The Q4_K_M file loads 19.908 GiB of weights and this card holds 16 GiB, so nothing here is a fit calculation in the usual sense. It runs because the model is a sparse Mixture-of-Experts and llama.cpp will leave the routed expert tensors of the first N blocks in system RAM: 123 tensors carry 18.586 GiB of the file, and across the forty loaded blocks that is 18.164 GiB of the 19.908 GiB resident — 91.24%. Everything the card actually has to keep — attention, the Gated-DeltaNet mixers, the shared experts, the router, the embeddings and the output head — is 1.744 GiB. The recipe is a choice of how much of that 91.24% to push across, and what the choice buys you in context.

Hardware data: RTX 4070 Ti Super (16 GB VRAM) · 13.546 GiB derived working set at -ncmoe 17, 131,072-token context · See benchmark data

⚠️ Read this before you read anything else on this page: the offload arithmetic below is a property of the file and of 16 GiB, not of this card. -ncmoe 17, the ladder, the budget table and the 131,072-token window come out identical on every 16 GB card, and this page says so rather than dressing them up as a discovery. What is genuinely specific to the RTX 4070 Ti Super is further down, and it is not arithmetic — it is that three llama.cpp threads run CPU-side offload of a Qwen3.x hybrid model on this exact board — one using this recipe's own -ncmoe flag, one at this recipe's own 131,072-token window, and one written by the author of the sibling --n-cpu-ffn option, for whom this card is the development machine. None of the three is Apodex and none of them is all three of those things at once. Those are in Three llama.cpp threads that run this technique on this card, and they are the reason to read this page rather than a tier-mate's.

⚠️ This model does not fit 16 GB on its own, and system RAM is a prerequisite rather than a footnote. At the setting below, 7.752 GiB of expert weights are resident in host RAM for as long as the server runs, and llama-server's prompt cache can commit several GiB more on top (see System RAM). bartowski's own sizing rule for this file is to "add both your system RAM and your GPU's VRAM together" and pick a quant a couple of GB below the sum (model card); with 20.363 GiB of weights that is satisfied on paper by 16 GB of RAM, and in practice you want 32.

ℹ️ This recipe is text-only, deliberately. The checkpoint contains a vision tower and the GGUF repo ships a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions at all — the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. Every number below is for text generation, and the run command refuses the projector explicitly rather than merely omitting --mmproj. Our RTX 3090 page sets out the same reasoning at length.

⚠️ Two version floors, about 2,200 builds apart. Both are properties of the model rather than of the card, and both were bisected against adjacent published tags.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990, so it arrived in (b7989, b7990].
• The qwen35moe-specific MTP wiring — the load_mtp / TENSOR_SKIP handling in src/models/qwen35moe.cpp that keeps block 40 off the card until you ask for it — absent at b10211, present at b10212, so it arrived in (b10211, b10212].

Each is written as an interval because a tag-granularity probe cannot resolve finer. Use b10630 or newer — the build the lead quant was produced with. That is a sufficient floor, not a bisected minimum. Every source reference on this page was read at release b10666 (4e97ac86ebe2c4cb8212d98d2641ad6768810896, published 2026-08-28), and every line number quoted below is that tree's. Line numbers move between releases; the current tag when this page was written was b10715 (2026-08-31), and llama.cpp published four tags in the six hours before it. If you check a line here against a newer tree, expect the offset to have drifted and the code not to have.

Requirements

Installation

1. Build llama.cpp with CUDA, and check your toolkit version

The architecture string baked into the GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. A release binary at b10630 or newer works; to build from source:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

One build gate matters on Ada silicon and it is silent when it bites. This card is compute capability 8.9. In ggml/src/ggml-cuda/CMakeLists.txt, the non-native branch appends 75-virtual 80-virtual 86-real unconditionally at :34, and 89-real 90-virtual only inside if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8") at :36-37. There is no 86-virtual. So a pre-11.8 toolkit gives an sm_89 card no real cubin and no near-neighbour PTX either — it JIT-compiles from 80-virtual, with no error and no warning. GGML_NATIVE defaults on and :27 takes the native path when the toolkit is ≥ 11.6 and CMake ≥ 3.24, so an ordinary local build escapes this; container builds, cross-compiles and anything passing -DGGML_NATIVE=OFF do not. Check nvcc --version before you conclude the card is slow.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

That is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf, 21,864,082,336 bytes (20.363 GiB), imatrix-calibrated and produced on llama.cpp b10630. Re-read from the repo's own blob listing on 2026-08-31; the repo's lastModified is 2026-08-26T09:14:35Z.

⚠️ Disk size and VRAM residency are different numbers here, and the difference is not only the MTP block. The file is 21,864,082,336 B on disk; the sum of all tensor payloads is 21,853,090,304 B; the resident weight set with MTP off is 21,376,133,632 B. Disk minus resident is 487,948,704 B, of which blk.40 — the multi-token-prediction block, TENSOR_SKIPped unless you ask for it — is 476,956,672 B and the remaining 10,992,032 B (10.5 MiB) is the file's non-tensor content: the GGUF header, the key-value metadata, the tensor-info table and inter-tensor alignment padding. That residual is exactly 21,864,082,336 − 21,853,090,304, i.e. everything in the file that is not tensor payload, by construction. Budget 19.908 GiB of VRAM-side weights and free ~22 GB of disk; neither figure substitutes for the other.

Q4_K_M on a 16 GB card is a deliberate choice, and the obvious alternative is narrower than it looks. Ten of bartowski's rungs are under 16 GiB on disk, so "small enough to load" is not the constraint. The constraint is what has to sit beside the weights: at this page's ceiling (derived below) a fully resident configuration at 32,768 tokens of q8_0 KV needs resident weights at or below 13.524 GiB. ⚠️ The rung sizes below are on-disk figures from the HF tree, so they are the wrong side of the axis and each over-states what that rung would actually hold in VRAM — by its own MTP block plus non-tensor content, which is 0.454 GiB on the Q4_K_M and is not measured per rung. The screen is therefore conservative, and the conclusion survives the correction with room to spare: Q2_K_L at 12.650 GiB on disk clears the budget either way, while IQ3_XXS at 14.287 and Q3_K_S at 14.886 miss it by more than 0.454 even before residency is subtracted, so the whole 3-bit tier is out on both readings. So the escape from offload is not "drop to 3-bit", it is "drop to 2-bit", and that is where the publisher of a second GGUF set draws the line: abenzerps suggests IQ2_M or IQ3_M as a 16 GB starting point and then closes the section with "Prefer Q4_K_M or higher when system memory permits." This recipe follows that advice, because the model is an agent that runs tool calls and multi-step plans, and 2-bit weights are the wrong place to save on a workload where a single malformed argument ends the run. A 3-bit rung with a shallow -ncmoe is a reasonable middle and this page does not measure it either.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 17 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --load-mode none \
    --cache-ram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05), whose text notes that "Apodex follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF by default.

Eight of those flags are load-bearing and none of them is a default.

• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 17 is this recipe's subject; the next three sections derive the number.
• --no-mmproj — llama-server honours the projector auto-download, and both no_mmproj (false) and mmproj_use_gpu (true) default that way in common/common.h, so a bare -hf …-GGUF fetches a sibling projector and puts it on the GPU. The lead repo ships two, at 899,283,488 and 902,822,432 bytes (0.838 and 0.841 GiB). On this card that is just under the 0.844 GiB that the two cheapest steps of -ncmoe would free, spent on a capability the recipe does not use.
• -np 1 — and the reason is worth three files, because "the default is 1" is true of the struct and false of this program. common/common.h:455 initialises n_parallel to 1; common/arg.cpp:1399-1400 then overrides it per example — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default }; and tools/server/server.cpp:152-156 turns that negative into params.n_parallel = 4; params.kv_unified = true;. So omitting -np on llama-server gives you four slots and unified KV, not one slot. The Gated-DeltaNet recurrent state is per sequence slot, so four slots quadruple it, from 0.061 GiB to 0.245 GiB. One slot is right for a single-user agent loop, and passing it explicitly also pins you against the -1 = auto behaviour changing.
• -c 131072 — the GGUF declares context_length 262144. Half of it is what this recipe budgets; the ladder below prices the other half.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 is a 2-byte scale plus 32 one-byte quants, i.e. 34 bytes per 32 elements — 8.5 bits per element rather than 16. It takes the 131,072-token cache from 2.500 GiB to 1.328 GiB, which on this card is a saving of 1.172 GiB — worth between two and three steps of -ncmoe you do not have to spend, since a step costs 0.4219 or 0.4863 GiB so two come to 0.844–0.973 and three to 1.266–1.459. It also decides a flag you are not passing; see What -fa off would cost here.
• --load-mode none — llama.cpp asks for this itself. When a CPU tensor override meets memory mapping, src/llama-model-loader.cpp:1194 prints "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance". An overridden tensor is given the first workable buffer type from a list built in the order "ACCEL -> GPU host -> CPU extra -> CPU" in src/llama-model.cpp, whose GPU-host entry exists precisely because "storing the tensors in a host buffer is useful when the processing of large batches" is offloaded to a GPU — pinned memory the card can DMA from. The loader downgrades that choice to ordinary CPU memory whenever mmap is on. So mmap silently costs you the pinned buffer this whole path was built around.
• --cache-ram 2048 — see System RAM. The default is 8192 MiB (common/common.h:632) and on this architecture that is not a small number.

How the offload actually works

-ncmoe N / --n-cpu-moe N matches tensor names, not architectures. The flag appends one buffer-type override per block index i in 0 … N-1, each built from LLM_FFN_EXPS_REGEX at common/common.h:1130:

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";

In this GGUF exactly 123 tensors match — ffn_down_exps, ffn_gate_exps and ffn_up_exps, three per block across all 41 blocks. The override is applied at tensor-creation time, in the loader, before any buffer exists: a matching tensor is given a CPU-side buffer type and is then allocated and read into it. Those bytes never enter VRAM at all. Nothing about the architecture is consulted, which is why the flag works identically on a hybrid recurrent model and on a plain transformer MoE.

Three things it does not move, all of which stay on the card:

• The Gated-DeltaNet mixers and the attention weights. A GDN block's ssm_* tensors, its attn_qkv and attn_gate, and a full-attention block's attn_q/k/v/output are untouched by the regex.
• The shared expert and the router. ffn_down_shexp, ffn_gate_shexp, ffn_up_shexp, ffn_gate_inp and ffn_gate_inp_shexp do not match _exps, so the dense path every token takes stays resident.
• The KV cache and the recurrent state. Both pick their buffer type from the layer's device, which -ngl decides; -ncmoe rewrites weight tensors and never touches the layer assignment. So a block whose experts are in RAM still keeps its recurrent state in VRAM — which is what makes the budget below decomposable at all.

Add it up and -ncmoe 40 leaves 1.744 GiB on the card: 0.655 GiB of token_embd and output, and 1.089 GiB of everything else across forty blocks.

Does the offload really happen on a hybrid recurrent architecture? The source above says the override is applied by name at creation time and is architecture-blind, which is an argument rather than an observation. The observation exists and it is on this card: on llama.cpp #24886 a commenter running a Qwen 3.5 MoE on an "RTX 4070 Ti SUPER (16 GB, sm_89)" reports ngl 48 -ncmoe 43 occupying 14.7 GB of that card while ngl 48 -ncmoe 40 puts it ">16 GB" — GPU occupancy moving with the flag, in the direction and roughly the magnitude the tensor table predicts, on this silicon. A second commenter in the same thread reproduced the surrounding bug on Qwen3.5-35B-A3B, this model's own base. Neither is Apodex and neither is a peak measurement; they establish that the flag moves bytes off this card, not how many it moves for this file.

The VRAM budget

Every figure here is derived from the artifact's own tensor table and from llama.cpp's allocation code at b10666. Nothing on this page was measured on an RTX 4070 Ti Super with this model.

A 16 GB card does not give you 16 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. The only expert-offload run in our catalogue with the numbers written down is gemma4-26b on an RTX 3060 at -ncmoe 12, whose submitter recorded a peak of "11,179 of the card's 12,288 MiB" on a headless box — so at most 1,109 MiB was unavailable to the model there. ⚠️ That is a different board in a different VRAM tier, and the term is a driver reservation plus a CUDA context rather than a percentage of capacity, so borrowing the absolute 1,109 MiB here transfers a quantity rather than measuring one. It is the best number in the catalogue and it errs in the direction this page wants: this page reserves 1,109 MiB, and a Windows box driving a monitor should expect to want more.

The second reserve is llama.cpp's own. Auto-fit's default margin per device is 1024 MiB — fit_params_target = std::vector<size_t>(llama_max_devices(), 1024 * 1024*1024) at common/common.h:481 — which is the runtime's own estimate of what compute buffers, the CUDA context and the scheduler need beyond the weights and the caches. This page adopts it, so:

weights + KV + recurrent state ≤ 16,384 − 1,109 − 1,024 MiB = 14,251 MiB = 14,943,256,576 B = 13.917 GiB.

At -ncmoe 17, -c 131072, q8_0 KV and one sequence slot:

Each GiB figure is rounded from its own byte count and the total is rounded once from the byte sum, which is why the GiB column adds to 13.545 while the total reads 13.546. The bytes are the arithmetic; the GiB are a convenience.

The KV row is 10 full-attention layers × head_count_kv 2 × key_length 256 × 2 (K and V) × 8.5 bits — block_q8_0 is 34 bytes per 32 elements, so 8.5 bits per element rather than 8. Ten of forty blocks carry full attention: full_attention_interval is 4 and the GGUF's tensor list puts them at [3, 7, 11, 15, 19, 23, 27, 31, 35, 39]. The other thirty carry Gated-DeltaNet recurrent state, which does not grow with context.

Choosing your own -ncmoe

The ceiling above minus the KV cache and the recurrent state leaves a weights budget, and the ladder answers it. Both figures either side of the chosen rung are printed, because the margins are what tell you how much slack you have:

All four rows use -ctk q8_0 -ctv q8_0 and -np 1. All four are arithmetic. The middle row is what the command above runs, and it is chosen so that this page and the RTX 3090 page document the same context — the whole difference between a 24 GB card and this one is that 7.752 GiB of experts have moved off the board and into RAM.

Note how thin the 32,768 row is: -ncmoe 14 clears its budget by 0.037 GiB, or 38 MiB. That is not headroom, it is a rounding error, and it is why a shallow-context configuration on this card should start at 15 rather than at the first rung that arithmetically fits. The 131,072 row's 0.371 GiB is the comfortable one, which is an odd thing to be able to say about the deeper window — and it is not a property of the depth. The margin is simply whatever is left over after the last whole step, and a step is 0.4219 or 0.4863 GiB, so where a given context lands inside a step is close to arbitrary. Read the margin, not the rung number.

Going the other way is legitimate and this page will not pretend otherwise: if your agent loop never fills a six-figure window, -ncmoe 14 at 32,768 leaves less in RAM and less to stream.

The full ladder, weights only — no KV cache, no recurrent state, no compute buffers:

The left column is a system-RAM requirement, and it is the number to size your host against.

Why the steps are uneven

The ladder is not linear. bartowski upcasts ffn_down_exps to Q6_K in 20 of the 40 loaded blocks — 0, 1, 2, 3, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, and then 34 through 39 — so one step of -ncmoe costs either 0.4219 GiB (Q4_K/Q4_K/Q4_K) or 0.4863 GiB (Q6_K/Q4_K/Q4_K) depending on which block it lands on. Any sentence of the form "a step of -ncmoe costs X GB" is wrong on this file; read the cumulative column instead. The check is that the two block sizes reproduce the total exactly: 20 × 522,190,848 + 20 × 452,984,832 = 19,503,513,600, which is the offloadable total across blocks 0–39, and 19,503,513,600 / 21,376,133,632 = 91.24%.

Blocks 0–4 are all the expensive kind, which is why -ncmoe buys a lot early, and 34–39 are too, which is why the deep end of the ladder is steep again.

⚠️ This has a consequence this page found nowhere in the sources it read, and it is specific to files like this one. -ncmoe takes blocks sequentially from 0, so which blocks you get is decided for you. llama.cpp #27987, an open feature request filed on 2026-08-29 by the author of the --n-cpu-ffn option — running an "RTX 4070 Ti Super 16 GB; 32 GB DDR5; i5-13600KF" — proposes selecting the band by size instead, and reports of the MoE case that "I also tested --n-cpu-moe on Qwen3.6-35B-A3B: its experts are all the same size, so largest does nothing there" That is true of Qwen3.6-35B-A3B and false of this file, whose expert triples differ by 15.3% between the two kinds. Offloading the six expensive blocks 34–39 instead of six cheap ones frees 2.918 GiB rather than 2.531 GiB — the same count of blocks, 0.387 GiB more VRAM. Today that needs --override-tensor by hand and this page does not document it, because nobody has measured whether the extra depth costs more in host reads than it saves in VRAM. It is the clearest thing on this card worth measuring, and if you do, please send the numbers.

What this card shares with the RTX 4080 — and what it does not

This is the section that decides how much of the rest of this page you should trust as yours rather than as the tier's, and NVIDIA's own spec pages settle most of it:

Sources: the RTX 4070 family, RTX 4080 family and RTX 4060 / 4060 Ti spec pages, read 2026-08-31.

Read the middle two columns against each other and the honest summary is short. On every vendor-published term that enters this recipe's arithmetic — capacity, memory type, interface width and PCIe generation — this card and the RTX 4080 are the same card. The one published difference is 8448 shaders against 9728, and shader count is the term this recipe is least bound by: prompt processing is bounded by expert transfers over the host link rather than by GPU compute (#25859, discussed below), and at generation the offloaded expert matmuls do not run on the GPU at all. So the -ncmoe 17 answer above is not merely similar to the RTX 4080's — it is the same number reached from the same inputs, and this page would be lying if it dressed that up as a separate finding.

Two things follow for how to read the rest:

• Capacity conclusions transfer across the whole 16 GB tier. The ladder, the budget, the four-row -ncmoe table and the 131,072-token window are properties of a 20.363 GiB file against a 16 GiB board. They come out identical on an RTX 4060 Ti 16GB, on this card and on an RTX 4080, because none of the inputs is a property of the silicon.
• Bandwidth conclusions do not transfer, and the 128-bit column is why. At equal memory clock the RTX 4060 Ti reads its own VRAM at half this card's rate, which is enough to flip which half of a decode step is the slow one. That is the next section, and it is the one place where being on a 256-bit board changes the advice.

⚠️ NVIDIA publishes neither a memory speed nor a memory bandwidth for any of these three boards, and it publishes the PCIe generation without a lane count. This page therefore asserts neither, and states the one bandwidth conclusion it draws as a threshold you check against your own board's published memory speed.

Where the decode time actually goes

The mechanism first. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights" (ggml/src/ggml-backend.cpp:942) — unless the scheduler finds "a backend with higher prio wants to offload the op" (:960). CUDA answers that with a single comparison in ggml/src/ggml-cuda/ggml-cuda.cu:

// :5341-5344 at b10666
static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
    ...
    return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size;
}
// :5515 — const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;

Generating a token is a batch of one, and one is less than 32, so at decode the offloaded expert matmuls stay on the CPU and those weights never cross PCIe at all. Prompt processing is a batch of hundreds (-ub defaults to 512 at common/common.h:452, -b to 2048 at :451), so at prefill those same tensors are copied host→device. The two phases are bound by different things and this page treats them separately.

The decode roofline, in bytes

expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token. Everything else on the card is read whole. At -ncmoe 17:

Two lines in that table are worth a sentence each. The KV cache is decode traffic, not just decode capacity — attention re-reads the whole occupied cache out of VRAM on every generated token, and at a full 131,072-token window that is 1,426,063,360 bytes, larger than every weight the card holds put together. And output.weight is separated from token_embd: they are usually lumped as "0.655 GiB of embeddings", but only one is read per token — the Q6_K head is read in full to produce logits, while the embedding table is read one 1,152-byte row at a time. The split is checkable: 417,177,600 + 286,064,640 + 8,192 + 1,169,369,600 = 1,872,620,032, which is the 1.744 GiB floor exactly.

The crossover, stated as a number you can look up

The two halves of a decode step take equal wall-clock exactly when your card's bandwidth is VRAM bytes ÷ host bytes times your DIMMs'. That ratio contains no bandwidth figure at all, which is why it is the form this page states:

The ratio is a property of the file at -ncmoe 17, so it is the same on any 16 GB card running this recipe. What is not the same is where your board sits against it — and because NVIDIA publishes this card's 256-bit width and not its speed, the useful form is to convert the threshold into the memory data rate a 256-bit board would need. A 256-bit bus moves 32 bytes per transfer, so bandwidth = per-pin data rate × 32:

Look your board's memory speed up and compare. Above the number in your row and column, the host half of a decode step is the slow one and faster DIMMs are the upgrade that helps; below it, your own VRAM is the slow half and faster DIMMs buy less than you expect.

Three readings of that table are safe to state without knowing any card's speed:

1. At this recipe's window against dual-channel DDR5, no 256-bit GDDR6X board is host-bound. 39.5 Gbps per pin is not a rate GDDR6X ships at. So on a modern DDR5 desktop, decode on this card at 131,072 tokens of context is VRAM-bound, and the lever that helps is the one already in the run command: -ctk q8_0 -ctv q8_0 halves the largest term in the table.
2. On dual-channel DDR4-3200 at the same window the threshold is 21.1 Gbps per pin, which lands inside the range GeForce 40-series GDDR6X boards actually run at. That is the one row where a reader on this card genuinely has to look their own board up rather than take an answer from this page — and where two 256-bit GDDR6X boards a few hundred MHz apart can land on opposite sides of the line. When a threshold sits that close to the hardware, the honest reading is that the two halves cost about the same — not that this card is host-bound.
3. Depth moves the answer more than the card does. The same board that is comfortably host-bound at an empty cache can be VRAM-bound at 131,072 tokens: the threshold rises 71% from 7.70× to 13.18× purely because the KV cache fills. Derive it at the depth you actually run.

⚠️ The × 32 shortcut is a GDDR6/GDDR6X rule and does not survive a move to Blackwell. NVIDIA's Blackwell architecture whitepaper describes GDDR7 as PAM3 signalling carrying 1.5 bits per cycle against PAM4's 2, so the familiar clock-doubling arithmetic gives the wrong answer on a GDDR7 board. This card is GDDR6X per NVIDIA's own spec page; do not carry the arithmetic to a 50-series card without redoing it.

⚠️ And say which terms a crossover contains before comparing it with another page's. Every ratio above includes the recurrent state, and the three rows add the KV cache at the stated occupancy. The figure most often quoted for this technique is 5.3×, which is a 12 GB card at -ncmoe 26 measured against a loose ≤ 1.744 GiB bound on non-expert VRAM with both the KV cache and the recurrent state left out. It is not this page's 7.70× with a term restored; it is a different calculation on different hardware.

Prefill is the phase the link pays for

At batch sizes of 32 or more the scheduler hands the expert matmuls to the GPU, which means copying host-resident expert tensors across PCIe. llama.cpp #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0", "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26" — finding that "prefill is bottlenecked by PCIe transfers, not GPU compute", with the GPU sitting "idle a large fraction of each prefill pass waiting on expert-weight H2D copies" that "run serially with the matmul that consumes them". ⚠️ Read the number carefully, because it is easy to mis-group: the reporter's "~42%" is the baseline idle he measured before his own proposed overlap change, which took it "to a few percent" — it is not a steady-state property of the path. And carry his caveats: a different model, one box, gen-4-specific, and he dates it to "around the b6xxx era" and says outright that "the exact idle percentage may be stale". It is the strongest evidence this page found that the shape of this path is real, and it is not a measurement of Apodex.

If the whole offloaded set crosses per physical batch — the shape that report describes — then at -ncmoe 17 that is 8.32 GB per -ub batch. PCIe 4.0 carries about 1.97 GB/s per lane after 128b/130b encoding, so the transfer term alone bounds prompt processing at roughly 1,938 tokens/s over sixteen lanes and 969 over eight — theoretical link rates, with real achieved throughput lower.

This page does not assert a lane count. NVIDIA publishes the PCI Express generation for this card — Gen 4 — and stops there. Read your own, since it is a property of your slot as much as your card:

nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.width.max,pcie.link.gen.current,pcie.link.width.current --format=csv

If width.max reads 8, halve the ceiling above; if gen.current reads 3 where gen.max reads 4, you are in a slot that halves it again, silently. Either way it is a time-to-first-token problem and not a tokens-per-second one, because at decode batch 1 those weights do not move. And if the link turns out to be your constraint, --no-op-offload disables the offload rule globally, keeping the expert matmuls on the CPU in both phases and removing the transfers entirely — a real trade, untested here, and one flag to try.

What -fa off would cost here, and why this page does not offer it

-ctk q8_0 -ctv q8_0 requires FlashAttention. With the default -fa auto, src/llama-context.cpp:3675-3678 logs "enabling flash_attn since it is required for quantized V cache" and turns it on; an explicit -fa off is a hard error at :3681 and a second throw at :465 reads "quantized V cache was requested, but this requires Flash Attention". That is fine here — the same file checks that q8_0's 32-element block divides n_embd_head_k and n_embd_head_v, both 256 on this model — but a flag a page tells you not to change still needs its alternative priced, and the price is larger than it looks.

Turning FlashAttention off changes the graph, not just the cache. On the non-FA path src/llama-graph.cpp:2607 builds ggml_tensor * kq = ggml_mul_mat(ctx0, k, q); and :2612 calls ggml_mul_mat_set_prec(kq, GGML_PREC_F32). That tensor is [n_kv, n_tokens, n_head] at F32, this model's head_count is 16, and the worst-case graph is reserved at n_tokens = min(n_ctx, n_ubatch). At -c 131072:

At the default -ub 512 that single tensor is 4.000 GiB, against 0.371 GiB of slack in the budget above — before the mask, the softmax or expert staging. Even -ub 128 consumes the entire 1024 MiB compute reserve, so a 16 GB card at this window would need -ub 64, and 64 is still ≥ 32, so prefill expert staging still happens inside whatever is left. Meanwhile the KV saving is smaller than the usual telling: -fa off rejects only a quantised V, not K — the K check at :3686 is gated behind flash_attn_type != DISABLED and never runs when FA is off — so -ctk q8_0 -ctv f16 is legal and costs 15,680 B/token, i.e. 1.914 GiB at 131,072 rather than the 2.500 GiB an all-f16 cache would.

So the fallback is cheaper than usually stated on the cache and far more expensive overall, and on this card at this window it is not available at any sane -ub. If you need it — see the illegal-memory-access entry under Troubleshooting — the affordable form changes the context, not the ubatch.

head_count = 16 is worth pinning, because getting it wrong doubles every cell above. Three routes agree or fail visibly:

⚠️ The metadata key is a strict prefix of qwen35moe.attention.head_count_kv, so a substring or de-duplicating probe sees one key where there are two and reports the shorter absent. Check the u64 length prefix that precedes a GGUF key, or take the attn_output route.

System RAM, and the prompt cache that will surprise you

At -ncmoe 17 the expert weights hold 7.752 GiB of host RAM for the life of the process. That is the part the ladder tells you about. There is a second consumer that it does not.

llama-server keeps a prompt cache, capped by --cache-ram at a default of 8192 MiB (common/common.h:632). For an attention-only model a cache entry is a few MiB and the default is invisible. For a hybrid recurrent model it is not: an entry carries a full recurrent state, and this model's is 65,863,680 bytes (62.8 MiB) per sequence. A community reporter on another qwen35moe model opened #27894 as a per-request memory leak — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted it after a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs +126 MiB per request until it hits +507 MiB against the bound and then stays flat for ten consecutive requests. Closed as completed by its own author on 2026-08-28.

⚠️ Two things to carry if you cite that. The 126 MiB per entry is the only measured entry size this page found, and it does not equal one recurrent state on this geometry — 62.8 MiB is about half of it, and nothing in that thread instruments the difference, so the reporter's identification of the entry as one recurrent state is an inference that fails the arithmetic while the 126 stands as a measurement. And it was measured on a different qwen35moe build, not on Apodex. What survives is the shape and the fix: the growth is the prompt cache, it is bounded by the flag, and the flag is the fix. The reporter also rebuilt the same source against Vulkan and got +126.0 MiB/request against ROCm's +126.3, so this is not a vendor story.

Hence --cache-ram 2048 in the command above: an explicit couple of gigabytes rather than an invisible eight, on a host already holding 7.752 GiB of experts. --cache-ram 0 disables the cache entirely and is right if you are memory-constrained and your prompts do not share prefixes. With the cache capped, 16 GB of system RAM is arithmetically sufficient; 32 GB is what this recipe asks for, because the gap between the two is exactly the headroom that stops the OS swapping the expert pages you are reading on every token.

Three llama.cpp threads that run this technique on this card

Everything above this heading is arithmetic that any 16 GB page reaches. This section is not. The RTX 4070 Ti Super turns out to have an unusually dense footprint on the llama.cpp tracker for exactly this technique — CPU-side offload of a Qwen3.x hybrid model on a 16 GB board — and three of those threads change what this page recommends. None of them is Apodex and none is a number you may expect; they are evidence about the technique on the silicon.

Method note, because the search is a trap on this card: a text search for the card's name also matches issue #4070 on its number. Every hit below was confirmed by fetching the issue body and its comments and matching the literal string, which cut a 26-result search to 23 real mentions.

1. The auto-fit margin, and why this page pins -ncmoe by hand

#23772 (closed as completed, labelled stale) opens as a cross-version performance drop. A commenter reports on "RTX 4070 TI SUPER 16 GB + Ryzen 9900X + RAM 96 GB", running Qwen3.6-35B-A3B-UD-Q6_K — a 35B-A3B hybrid, at --ctx-size 131072, --flash-attn on, --cache-type-k q8_0, --cache-type-v q8_0, --parallel 1 and --fit on — that TG fell from ~48 t/s at b9190 to ~38 t/s at b9334. That is this card, this context depth, this KV type and this slot count, on a near-sibling of this model's base, under offload. It is the closest configuration match to this recipe found in the space this page searched — the llama.cpp tracker and the 22 model cards enumerated under Results — and it is still a different model at a different quant.

The thread's diagnosis is the part that changes advice. A second contributor bisected it to PR #23485, which adds the draft model's estimated footprint to the auto-fit margin, and demonstrated the effect at simulated 16 GB (an RTX 3090 Ti with an 8 GiB VRAM hog):

A maintainer's reply: "In that case it's just an issue with what value is used by default for --fit-target. The default margin of 1 GiB is chosen conservatively to make an OOM unlikely but you can potentially get better performance by setting a lower value." A third reporter confirmed on their own hardware that setting the flag "to something smaller than 1024 does bring back in my case the performance of old versions"

⚠️ Read it precisely. The simulated-16 GB table is -ngl layer offload with MTP on, not -ncmoe expert offload; it is a different card with a memory hog; and nobody re-tested the RTX 4070 Ti Super run specifically, so that card's 48→38 is consistent with the mechanism rather than proven by it. What the thread does establish is that on a 16 GB board the auto-fitter's margin, not the model, can decide how much lands on the card, and the margin has changed between releases.

That is the strongest argument on this page for the run command's shape. Passing an explicit -ncmoe does not merely add a setting: it takes the fitter out of the loop entirely. common/fit.h:27 states the contract — "only parameters that have the same value as in llama_default_model_params are modified" — and common/fit.cpp:484 makes the buffer-override case an outright bail, throwing "model_params::tensor_buft_overrides already set by user, abort", which the caller catches and logs as a failed fit before loading with your parameters untouched. Auto-fit is on by default (fit_params = true, common/common.h:476), so "not passing -fit" is not the same as "not using it" — but passing -ncmoe is. Adding -fit off as well is harmless and removes one warning line.

A second, independent reason to pin it by hand: #27171, a --fit-target throughput regression filed 2026-08-16 against a Q4_K_M of another 35B-A3B model, is still open and still labelled bug-unconfirmed (re-checked 2026-08-31, 3 comments), with a contributor's fix in flight at PR #27207 and still unmerged. Different card, different model — a reason to pin, not a claim about this pair.

2. A -ncmoe run on this exact board, and what it does and does not bound

#24886 (closed as completed 2026-07-05) is a tensor-parallel bug report, but one comment inside it is the only record of -ncmoe occupancy on this card that this page's tracker sweep found — see the method note above for what that sweep covered. Its author states the environment as "GPU 0: RTX 4070 Ti SUPER (16 GB, sm_89)", Windows 11, prebuilt CUDA binaries, model Qwen 3.5 122B-A10B-APEX, and reports:

"Single-GPU works fine:"

"ngl 48 -ncmoe 43 → GPU0 14.7 GB, 23 t/s"

"ngl 48 -ncmoe 40 → GPU0 >16 GB, spills to CPU, slower than 43"

⚠️ What this bounds and what it does not. It is a much larger model at an IQ quant, so 43 is not comparable to this page's 17, and 23 t/s is not a number for this file. Its "14.7 GB" is also unit-ambiguous: read as decimal GB it is 13.69 GiB, just under this page's 13.917 GiB ceiling; read as GiB it is above it. The report does not say which, so it corroborates the order of magnitude of the ceiling and cannot tighten it.

The half that is unambiguous is more useful anyway: on this card, an offload that is one or two steps too shallow does not crash, it spills and goes quiet. That is Windows behaviour — the WDDM driver will let a process oversubscribe VRAM into system memory — and it means the failure mode you should expect on this board is a configuration that works and is slow, not an out-of-memory error. If your tokens-per-second is a small fraction of what the roofline above allows, check nvidia-smi before you check anything else.

A note on the rest of that thread, because it is easy to read as a live limitation and is not: the reporter's dual-GPU crash — an assertion failure that dumped all MoE tensors onto GPU 0 — was diagnosed by a third party and fixed in PR #25028, merged 2026-07-05, long before this page's build floor. Whether tensor-parallel now distributes a -ncmoe MoE usefully across two cards is a separate question the thread does not answer, and this page does not claim it either way.

3. This card is where the CPU-offload options were developed

The option next to the one this recipe uses was written on this board. PR #26622, llama : add --n-cpu-ffn option — merged 2026-08-27, so it is present at this page's b10666 pin as {"-ncffn", "--n-cpu-ffn"} at common/arg.cpp:2799 — was written and benchmarked on "RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS". Its author reports "a tolerable speed of 15-25 t/s tg using Q4_K_M model with a large context of over 90k" on that machine. ⚠️ The PR body never names the model, and --n-cpu-ffn targets the dense FFN of dense models rather than MoE experts, so that range is not a figure for this recipe — it is a statement that a 16 GB board with 32 GB of DDR5 holds a usable interactive rate at a six-figure context with a large fraction of a Q4_K_M in host RAM.

The follow-up, #27987 (open, filed 2026-08-29), publishes the same author's benchmark tables on the same board — Qwen3.8-27B, Unsloth UD3 quants, build 10280, measured "at context prefill of 50k". For UD-Q4_K_M with the current sequential band selection, roughly 2.7–3.8 GiB of FFN on the CPU gives 14.9 t/s with MTP off and 18.9 t/s with MTP on; the proposed largest-first selection reaches 15.5 and 21.7 at comparable CPU sizes. ⚠️ Again: different model, different flag, a smaller host-resident set than this recipe's 7.752 GiB, and prefill at 50k rather than 131k. It is a shape, not a prediction.

But two lines in it bear directly on flags this page discusses. The first is quoted under Why the steps are uneven — the observation that a uniform expert size makes size-ordered selection pointless, which is true of the model measured there and false of this file. The second is about MTP and is in the MTP entry below.

Results

• Speed: omitted, and not because nobody has measured the model. Two quantisers publish throughput for Apodex 1.1 mini and neither number can travel to this page. ghazni101/Apodex-1.1-mini-MQ4RP reports 226 tok/s — on an RX 7900 XTX, i.e. gfx1100, from a .mq4r file under hipfire, a separate Rust engine, so it is neither this file nor this runtime nor this vendor. kingjones777 publishes a four-rung ladder — 63.30, 45.05, 32.54 and 64.87 tok/s — every rung measured with full GPU offload (-ngl 999) on a Ryzen AI MAX+ 395 unified-memory APU under a ROCmFPX fork: a machine with no expert offload and no PCIe hop, which is the entire subject of this page. The top of that range is also unsettled: 64.87 belongs to the STRIX_LEAN build, and that build's own card has a "Speed — full offload" section that is a one-row table holding a dash, above a note saying the run is still pending on an idle box — while the shared variants table further down prints 64.87 for that same build, under a caption reading "A dash means I haven't measured that one yet". There is no throughput figure for this file, on this runtime, at any -ncmoe setting, on any card. The space searched, re-run 2026-08-31: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces (fetched and scanned for every throughput unit, not only tok/s, because a table that prints its unit in a column header defeats a unit-suffix pattern); the llama.cpp tracker, where the card-name search above returned 23 confirmed hits and none of them mentions Apodex; and this catalogue, where /check/apodex-1-1-mini/rtx-4070-ti-super returns unknown with zero benchmarks. If you run it, please contribute the numbers.
• VRAM usage: 13.546 GiB derived sub-total at -ncmoe 17 and 131,072 tokens with q8_0 KV, plus the ~1,109 MiB driver share and the 1024 MiB compute reserve this page sizes against but cannot derive. 7.752 GiB of expert weights sit in system RAM alongside. Dropping to -ncmoe 14 at 32,768 tokens gives 13.880 GiB and 6.422 GiB respectively. See /check/apodex-1-1-mini/rtx-4070-ti-super for live data as it lands.
• Quality notes: the vendor publishes agentic evaluation scores on the model card, but all of them run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. The lead quant is imatrix-calibrated — the publisher ships both the calibration corpus and the imatrix alongside the ladder — which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier. Note that the routed experts are already narrow (expert_feed_forward_length 512), which is part of why the sub-4-bit rungs give up more than the file sizes suggest.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-4070-ti-super.

Troubleshooting

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE, which arrived in (b7989, b7990] — so this error means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

It loads fine, then runs out of memory when a long prompt lands

Expected if -ncmoe is too low for the context you asked for. Loading allocates the weights and the full KV cache, but the compute buffers grow with the batch, and prompt processing additionally needs room for the expert weights it is copying to the card. Raise -ncmoe by two or three — on this file that frees between 0.844 and 1.459 GiB depending on which blocks the steps land on — or drop -c. If the card is also driving a display, start two higher than the table above and check nvidia-smi after load.

On Windows, expect the other failure mode first: the driver will let the process oversubscribe into system memory rather than fail, so a too-shallow -ncmoe shows up as collapsed throughput and not as an error. The report quoted under A -ncmoe run on this exact board is exactly that, on this card.

An illegal memory access on the second request

llama.cpp #26609 is open, deterministic across five-plus reproductions on two builds, and is a CUDA error: an illegal memory access in ggml_backend_cuda_synchronize on qwen35moe under partial expert offload — our architecture and our technique. It disappears with --flash-attn off.

⚠️ It is not a report about this card, and this page will not pretend it is. Read in full the thread reproduces on an RTX 5070 12 GB (Blackwell, Windows) and on an RTX 3070 Laptop GPU 8 GB (Ampere, Linux): two architectures, two capacities, two operating systems, neither of them Ada and neither of them 16 GB. That makes it an architecture-scoped defect rather than a card-scoped one, which is a reason for any qwen35moe offload page to name it and not a reason to expect it here.

If you hit it, note that the documented workaround costs more on a 16 GB card at this window than the thread's own runs paid: every passing -fa off arm in that matrix is at -c 32768, where the F32 score matrix is 1.000 GiB at -ub 512; no arm of that matrix runs -fa off at 131,072 on a card this size, and the arithmetic under What -fa off would cost here says why. The affordable form drops the context rather than the ubatch — -c 32768 -ub 256 -ctk q8_0 -ctv f16 — and note that -ctv f16 alone is enough, since the K guard never runs when FA is off. The thread's own probe takes both halves to f16 and over-corrects by 0.146 GiB at that depth — 0.293 GiB if you run it at 65,536 instead.

The GPU hangs minutes into a long agent session

llama.cpp #27330 is open and reports CUDA graphs hanging the GPU channel — the NVIDIA kernel driver's Robust Channel watchdog declaring the GPU locked after a 7-second notify timeout, raising Xid 8, with llama-server aborting inside ggml_abort. The trigger is "a real coding agent workload (I used OpenCode) so the slot stays busy with long prompts and long generations", which is the workload this model exists for.

⚠️ Every reported instance is Blackwell, and one comment in that thread is an in-box control that points away from Ada specifically. A reporter running an RTX 4090 (Ada) and an RTX 5080 (Blackwell) in one machine writes that "The failing GPU is always the 5080 (device 1, Blackwell); the 4090 (Ada) never reports an error", and adds that the problem "started after I replaced a RTX 4060 Ti 16G with a RTX 5080". One machine, one variable. This card is Ada, so on the evidence in that thread it is not yours. Four further limits travel with it: every report is qwen35 and not qwen35moe; none is an expert-offload run; the hang is stochastic; and the workaround's evidence is statistical.

Mentioned here only so that a reader who finds the thread while searching their own symptom knows why it is not the explanation — and because of one thing that is relevant to any CUDA card. The workaround is GGML_CUDA_DISABLE_GRAPHS=1, and ggml/src/ggml-cuda/common.cuh reads that switch as getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr — presence, not value. So GGML_CUDA_DISABLE_GRAPHS=0 disables CUDA graphs exactly as thoroughly as =1. If you want graphs on, unset the variable; never zero it.

Prompt processing is slow and the GPU looks idle

This is the expected shape of the offloaded path rather than a fault. Confirm mmap is off (--load-mode none); llama.cpp warns about it, and a community reporter on the profiling thread above measured a gain on both phases from following it. Check what your PCIe link negotiated, since prompt processing is the phase that spends it. Raising -b and -ub is the largest lever that reporter found — 8192/2048 gave him ~1524 t/s of prompt processing against ~705 at 1024/512 — but read that gap carefully: his baseline sits below the 2048 default on -b and exactly at the 512 default on -ub, and he published no row at the actual default pair, so it is not a ratio a reader starting from defaults should expect. His nearest at-or-above-default row (2048/1024, ~1008 t/s) makes it about 1.5×. His 16384 row was slightly worse than 8192, so this is a sweep and not a direction, and it costs compute buffer that this page's 1024 MiB reserve has not been sized for. Different card, different model, different backend.

Measuring it: do not divide tokens by wall time

If you benchmark this configuration, read llama-server's own predicted_per_second rather than dividing generated tokens by total request time. llama.cpp #27623 is an open report headlined as a ~25× decode collapse at context on a hybrid Gated-DeltaNet model, retracted by its own reporter in comment 6: his metric was flawed: "The numbers in my matrix were completion_tokens / total request time, which includes prompt processing (~43s for a 68K fill at ~1500 t/s pp)", and so "it was prefill time, not decode" On an expert-offload path prefill is bus-bound and slow while decode is not, so that arithmetic makes decode look broken when only prefill is loaded. The issue is still open with the alarming headline above the correction; cite the retraction, never the headline.

System thrashing, or resident memory climbing request after request

Two different causes with the same symptom. If it climbs and plateaus, that is the prompt cache filling to --cache-ram, which defaults to 8192 MiB and costs a full recurrent state per entry on this architecture — cap it, as the command above does. If it does not plateau, your system RAM is simply short: at -ncmoe 17 more than a third of the file lives in RAM permanently, on top of the OS.

The server downloads an extra file you did not ask for

If you launch with -hf bartowski/apodex_Apodex-1.1-mini-GGUF instead of a local -m path, llama.cpp resolves a projector sibling as well — find_best_mmproj in common/download.cpp matches any sibling filename containing mmproj, and this repo ships two. Pass --no-mmproj, or use an explicit -m path as above.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and mainline llama.cpp implements it behind --spec-type draft-mtp (the exact spelling matters; mtp alone is rejected). On this card, under offload, the costs are concrete and traceable and the benefit is not.

Three costs, and the third is the one usually left out.

1. Block 40 stops being skipped: +0.444 GiB of weights, which -ncmoe 17 does not touch — the flag only covers blocks 0 to 16.
2. The recurrent state is widened. The chain is four hops at b10666 and worth stating because the multiplier is not obvious: --spec-draft-n-max defaults to 3 (common/common.h:326); need_n_rs_seq() at :393-399 returns draft.n_max when the speculative type is DRAFT_MTP; that lands in cparams.n_rs_seq at src/llama-context.cpp:104; and src/llama-memory-recurrent.cpp:101 allocates mem_size * (1 + n_rs_seq) rows. So at the default depth the recurrent state goes from 0.061 GiB to 0.245 GiB. src/llama-context.cpp:105-108 clamps n_rs_seq to 0 for architectures without recurrent partial rollback, so it is worth checking that the widening actually applies here — it does: llm_arch_supports_rs_rollback at src/llama-arch.cpp:1099 lists LLM_ARCH_QWEN35MOE at :1102, falling through to return true at :1109. The clamp does not fire on this architecture and the 0.245 GiB stands unconditionally.
3. A second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk.

Against 0.371 GiB of slack, items 1 and 2 alone are more than the budget has. And there is a fourth cost that only shows up if you let auto-fit size the run: PR #23485 adds the draft model's estimated footprint to the fit margin, which was measured in #23772 as the margin moving from 1024 to 2336 MiB — llama.cpp's own estimate that the MTP path wants about 1312 MiB more on the device.

The benefit is unknown here, and the two nearest reports disagree in a way that has a plausible mechanism. Unsloth's published guidance for this class is that "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs). Under CPU offload the two sides split further apart: an RTX 3060 owner running --n-cpu-moe 25 with MTP on, on a different model of this architecture, reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" (discussion #18, a community thread with no org member in it) — while on this card, #27987's tables show MTP worth +37% to +47% on a dense-FFN model under --n-cpu-ffn offload. A mechanism consistent with both, offered as an inference and not as a measurement: verifying K drafted tokens is one batch, so a dense FFN is read once for K tokens instead of K times — but this model routes 8 of 256 experts per token, so K tokens activate up to 8K distinct experts and the host read is not amortised. If that is right, the dense result does not transfer to -ncmoe at all.

One thing from that thread does transfer, because it is about the runtime rather than the model. Its author reports that "All MTP runs used GGML_CUDA_DISABLE_GRAPHS=1, because with MTP + CPU offload the CUDA graphs get re-captured every step (no measured speedup) and the recapture can OOM at a tight VRAM margin. Output is identical either way." A tight VRAM margin is what this recipe runs at, so if you try MTP here, try it with graphs disabled — and see the =0 trap above.

Try it if you like; do not budget VRAM for a speed-up you have not seen on your own machine, and if you measure one, tell us.

The 1.183 tok/s figure you will find, and should not use

Searching for this model plus offload turns up 1.183 tok/s, and two siblings of it at 0.292 and 0.163. All three are on kingjones777's model cards — 1.183 on …ROCmFP4-COHERENT-GGUF, 0.292 on …ROCmFPX-Q8_0-GGUF and 0.163 on …ROCmFPX-Q8_0-AGENT-GGUF — all three are retracted by their own author, and all three are the most misleading numbers a reader of this page can meet — because the first was measured at -ngl 17, one character from this recipe's -ncmoe 17, and they are not the same flag at all. -ngl offloads whole layers; -ncmoe offloads only the routed experts and leaves attention, the recurrent state and the shared path on the card. The author's own conditions: a Strix Halo box serving "8 live llama-server seats holding ~107 GiB of unified memory", leaving about 16 GiB, with the rest of the model on CPU mmap. He re-measured on an idle machine at full offload and published the corrected figures. Read those three numbers as a statement about a busy box and a different technique, not about expert offload.

Choosing a different quant publisher

Conversions of this model are still appearing: a Hub enumeration on 2026-08-31 returned 22 repositories matching Apodex-1.1-mini across 11 publishers, one of which had not existed 24 hours earlier. Re-enumerate before you decide, and check the byte count and the file list rather than the tier name — across publishers a file named Q4_K_M spans more than a gigabyte for the same nominal tier. Within bartowski's own ladder the rungs adjacent to the lead are Q4_K_S at 19.619 GiB and IQ4_XS at 17.955 GiB, worth roughly two and six steps of -ncmoe respectively, which is a legitimate way to buy back host RAM rather than VRAM.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, but they are vLLM/SGLang safetensors rather than GGUF and llama.cpp cannot load them at all. They are also too large for this card before anything else is counted — 22.548 GiB and 22.958 GiB of weights for the first two, against 16 GiB — and the -NVFP4 repo is not a 4-bit checkpoint anyway: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4. Their own Quick Starts launch on a single GPU: --tp-size 1 on both -NVFP4 and -GPTQ-Int4, and --tp 1 on -FP8. The --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone. For one RTX 4070 Ti Super the GGUF route above is the path.

A vision request crashes the server

Do not send one — this recipe runs with --no-mmproj and the model is documented as text-only. For completeness: llama.cpp #25717 reported a CUDA illegal memory access on vision requests with this architecture plus a projector, and it was closed as not_planned on 2026-08-30 with the label stale. Anything you read about that issue written before that date will describe it as open, including this site's own earlier pages for this model — that dates the text rather than changing the advice.

No Ollama tag

There is no Ollama library entry for this model. Checked 2026-08-31: ollama.com/library/apodex, /apodex-1.1-mini, /apodex1.1 and /apodex-1-1-mini all return 404, and the corresponding registry.ollama.ai manifest returns 404 while the same request shape returns 200 for a model known to be in the library. Use llama.cpp directly. If a tag appears later, please tell us.