Apodex 1.1 mini on RTX 4070 Ti: a 35B-A3B agent in 12 GB, and what Ada can and cannot buy

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one 12GB RTX 4070 Ti, at a 65,536-token context. The Q4_K_M weights are 19.908 GiB. The card holds 12 GiB. It runs anyway, because 91.2% of those weights are routed experts that llama.cpp can leave in system RAM.

Hardware data: RTX 4070 Ti (12GB VRAM) · 8.891 GiB derived working set at -ncmoe 26, 65,536-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 4070 Ti. /check/apodex-1-1-mini/rtx-4070-ti holds zero benchmarks. Every VRAM figure below is arithmetic over the GGUF's own tensor table and llama.cpp's allocation code, and it is shown so you can check it. There is no throughput number on this page at all — see Results for what exists and why none of it transfers. If you run this, please send the numbers.

🧭 The two things this card changes, and the one it does not. Capacity is settled by the 12 GiB and not by the silicon: the offload depth, the context and the VRAM ladder below are what any 12 GB card gets, and this page says so rather than dressing them up as an Ada result. Speed is where the board is its own case, and it splits in two. Decode never touches this GPU's expert path at all — the offloaded experts are multiplied on your CPU, so your generation rate is set by your DIMMs. Prefill is the phase where Ada's compute has something to do, and there is a profiled report saying that even there roughly two fifths of the time is the GPU idle on the bus. Both are priced out under Where this card's speed actually goes, including the one arithmetic test that tells you which side of the line your particular box sits on — and on a 192-bit board that test is worth running rather than assuming, which is not true of every card in this tier.

ℹ️ This recipe is text-only. The checkpoint has a vision tower and the GGUF repo ships a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card contains no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-30). The word images appears once, in a capability sentence — "Apodex 1.1 works directly with papers, datasets, spreadsheets, images, and code" — which is a claim about what the model can work on, not an instruction for feeding it a picture through llama.cpp. On a card this tight the projector is also 0.838 GiB you cannot spare, so the run command below refuses it explicitly rather than merely omitting the flag.

Requirements

Two version notes, because both have bitten this catalogue. The build tag moves several times a day — b10701 was published 2026-08-30T18:56:31Z, three tags inside one hour — so this page pins b10666 and makes no claim to be current; check the release list rather than trusting a date. And GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, not a bNNNN tag, because every build tag is marked prerelease; use /releases?per_page=N if you are scripting a version check.

Disk size and VRAM residency are different numbers here and they are not interchangeable. The file is 21,864,082,336 bytes — 20.363 GiB — on disk; a normal run loads 21,376,133,632 of them, 19.908 GiB. ⚠️ The 487,948,704-byte difference is two terms, not one, and it is worth spelling out because the obvious reading gets it wrong. The larger part is blk.40, the multi-token-prediction head, 476,956,672 bytes, created with TENSOR_SKIP unless you ask for it (src/models/qwen35moe.cpp). The remaining 10,992,032 bytes (10.5 MiB) never was a tensor: it is the GGUF header, the key-value metadata block, the tensor-info table and the inter-tensor alignment padding — exactly the gap between the file size and the 21,853,090,304-byte sum of all tensor payloads. Small enough to ignore in a budget, large enough that "disk size minus the MTP block" does not reproduce the residency figure. Budget 22 GB of disk and 19.908 GiB of weights.

What the vendor publishes about this board, and what it does not

NVIDIA's own 40-series spec table gives the RTX 4070 Ti as 7,680 CUDA cores, a 2.61 GHz boost clock, 12 GB GDDR6X, a 192-bit memory interface width, and PCI Express Gen 4. It publishes no memory-bandwidth figure and no PCIe lane count, so this page asserts neither. Both matter below, so here is how to get your own rather than borrow someone's:

# what your board actually is, and what it negotiated
nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.max,pcie.link.width.max,clocks.max.memory --format=csv

To turn that memory clock into bandwidth, multiply by 2 (GDDR is double-data-rate) and by 24 bytes (192 bits ÷ 8). If the answer does not land in the hundreds of GB/s, your driver is reporting a different clock convention — sanity-check it against the interface width before using it. Do the same for your DIMMs from their rated transfer rate: dual-channel DDR4-3200 is 3200 MT/s × 8 B × 2 channels = 51.2 GB/s, dual-channel DDR5-6000 is 96 GB/s. The second number matters more on this page than the first, and the section on speed shows exactly how much more.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. Use b10630 or newer — the release the lead quant was produced with. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that makes blk.40 optional landed in the bracket (b10211, b10212], so b10630 is comfortably past it.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

AD104 is compute capability 8.9; the default CUDA build covers it and no architecture flag is needed.

The build floor is a throughput statement here, not only a "does it load" one — but only on one architecture, and yours is not it. llama.cpp issue #25162 records that commit 9e58d4d69 cost 24–42% on Turing (SM75) for the Qwen3.5/Qwen3.6 family, specifically in ssm-scan.cu and ssm-conv.cu — the kernels behind 30 of this model's 40 layers — while "Gemma4 (no SSM layers) is unaffected". It is closed as completed, labelled stale, and b10630 carries the fix. ⚠️ Read the scope precisely: the regression was measured on Turing only. The codegen branch it came from is shared — GGML_CUDA_RESTRICT is emptied only for PDL builds at __CUDA_ARCH__ >= 900, and Ada is 890 — so whether the same effect ever reached this card is unmeasured, and nothing on this page upgrades a Turing measurement into an Ada one. The reason to be current is that on an older build you would lose speed silently on at least one architecture, with no error to tell you.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes, imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z; the file list and size above were re-fetched on 2026-08-30 rather than copied from a sibling page.

Do not start from the publisher's own quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card states plainly that "llama.cpp downloads the mmproj automatically when using" -hf. llama-server is in llama.cpp's mmproj_examples list (common/arg.cpp), mmproj_use_gpu defaults to true and no_mmproj defaults to false (common/common.h:594,596), so that command puts 899,283,488 bytes — 0.838 GiB — of vision projector onto a card this recipe has budgeted to about a gigabyte of slack. Download the file explicitly, as above, and load it with -m.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 26 \
    -np 1 \
    -c 65536 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Eight of those flags are decisions rather than defaults. Each default below was read out of the source at b10666 for this page rather than carried across from a sibling card, and each decision is labelled capacity-driven (it transfers to any 12 GB card) or bandwidth-driven (it does not):

• --no-mmproj refuses the vision projector rather than merely not asking for it — 0.838 GiB, for the reason given under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 26 keeps the routed experts of blocks 0–25 in system RAM. The flag's own help text at this pin is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp:2790, the option token itself at :2789), and the implementation matches the wording exactly — see How the offload actually works. 26 is derived in Choosing your own -ncmoe; it is not a measured optimum. Capacity-driven, and therefore the same number a 12 GB Ampere card gets — the constraint is the 12 GiB, not the architecture.
• -np 1. llama-server sets params.n_parallel = -1; // auto by default (common/arg.cpp:1400) and resolves that to four slots sharing one KV pool. The Gated-DeltaNet recurrent state is allocated per sequence, so four slots quadruple it — 0.245 GiB instead of 0.061 — and split your context four ways. One slot is what a single-user agent loop wants. Capacity-driven.
• -ctk q8_0 -ctv q8_0 halves the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 65,536-token cache from 1.250 GiB to 0.664 GiB. Capacity-driven.
• -fa on makes Flash Attention explicit. The default at this pin is auto — flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO (common/common.h:499) — and pinning it means a future change to what auto decides cannot silently change your memory profile.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader itself warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp:1194). Taking that advice means the 11.742 GiB of expert weights are ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure rather than a cache you can squeeze. Bandwidth-driven — it is a decision about how the host reads those weights, and the host is the thing this recipe is limited by.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632, flag at common/arg.cpp:1713 — and it is more expensive on this architecture than on a plain transformer. See the host-RAM entry under Troubleshooting.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05). The card notes that "Apodex follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF and enables Jinja by default, so tool calls and <think> blocks parse the way the vendor intends. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — the card is explicit about that: "Do not inline tool descriptions in the system prompt" — the emphasis is the vendor's.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It builds one regex per block and hands them to the model loader as buffer-type overrides (common/common.h:1130):

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, …) pushes "blk\.0" + regex … "blk\.(N-1)" + regex

The loader then does a plain std::regex_search on each tensor name and, on a match, allocates that tensor in a CPU buffer instead of the GPU one (src/llama-model-loader.cpp). Nothing in that path consults the architecture, so it applies to this hybrid recurrent model exactly as it does to a plain MoE. 123 tensors in this file match — three per block across all 41 blocks — totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads. That is 91.2% of the 19.908 GiB resident weight.

What does not move is as important. The offload takes each block's ffn_down_exps, ffn_gate_exps and ffn_up_exps, and leaves on the GPU: the attention path, the recurrent state tensors (ssm_conv1d, ssm_a, ssm_alpha, ssm_beta, ssm_norm, ssm_out), the router (ffn_gate_inp), the shared expert (ffn_*_shexp) and every norm. So routing is decided on the GPU, the Gated-DeltaNet state stays on the GPU, and only the selected experts' matmuls happen elsewhere. Offload everything the regex can reach — -ncmoe 40 — and 1.744 GiB is still resident.

That last word — elsewhere — means two different places depending on what you are doing, and on this card the distinction is the whole performance story. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights", per the scheduler's own comment (ggml/src/ggml-backend.cpp) — unless a higher-priority backend wants to offload it. CUDA does, and the test is a single comparison:

// ggml/src/ggml-cuda/ggml-cuda.cu:5341-5344 at b10666
static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
    ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context;

    return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size;
}
// :5515 — const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;

.offload_op is wired to that function in the device interface at :5390, and :5538 is where the 32 lands in the device context. Generating a token is a batch of one, so decode reads the offloaded experts from system RAM and multiplies them on your CPU — they never cross the bus. Prompt processing is a batch of hundreds, so prefill copies those tensors across PCIe into a duplicate the graph allocator sizes for the run. A narrow slot is therefore a time-to-first-token problem, not a tokens-per-second problem, and this page keeps the two apart throughout.

The VRAM budget

Weights, KV cache and recurrent state, at the recipe's settings.

The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 26 moves 12,608,077,824 of that (11.742 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks are loaded, and full_attention_interval 4 makes ten of them full-attention: blocks 3, 7, 11, 15, 19, 23, 27, 31, 35 and 39. Each carries head_count_kv 2 at key_length 256 and value_length 256 — 1,024 cache elements per token per layer, so 10,240 across ten layers. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows with the prompt.
• Recurrent state. Those thirty blocks each allocate two F32 tensors, sized (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements (src/llama-hparams.cpp:204,232). That is 2,195,456 bytes per layer per sequence and 65,863,680 across all thirty. llama-server prints it at startup as RS buffer size.

The two terms that table cannot produce, and what this page reserves for them

The 8.891 GiB is not a peak. Two things sit outside it, and on a 12 GB card they are the difference between a configuration that loads and one that does not.

1. The card's usable pool is not 12 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. Our catalogue holds one measured expert-offload run in this VRAM tier with the numbers written down — gemma4-26b on an RTX 3060 at -ncmoe 12, benchmark id 280, stored as peak_vram_gb: 10.92 — whose submitter recorded "That peak is 11,179 of the card's 12,288 MiB" on a headless box, adding that -ncmoe 8 and below "fail to load outright" rather than merely slowing down. ⚠️ That is a different GPU in the same tier — an Ampere GA106 board, not this Ada one — so read it as the order of the term and not as your board's reserve, which will differ with driver version and with whether a monitor is plugged in. This page sizes against ~11.0 GiB of usable pool, not 12.0. Capacity-driven, tier-level.

2. The compute buffers. llama.cpp allocates working memory for the graph on top of everything above, and under expert offload it also stages the host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 0.4219 or 0.4863 GiB, so that staging term alone is of that order. This page reserves 1024 MiB. That number is llama.cpp's own default margin per device for auto-fit — fit_params_target is initialised to std::vector<size_t>(llama_max_devices(), 1024 * 1024*1024) at common/common.h:481 — so it is the runtime's own opinion of how much room it wants, not a figure this page invented.

For scale on what happens if you leave nothing: llama.cpp issue #27698 records a run where the weights loaded, expert offload was honoured, and then context creation failed on a ~504 MiB compute buffer. ⚠️ That report is not about this card and does not transfer — it is an RX 6700 XT, gfx1031, RDNA2, on the ROCm backend, open with zero comments since 2026-08-25. It is quoted for one thing only: the magnitude of a term every table in this family of recipes excludes, and the shape of the failure it produces — a clean load followed by an OOM the moment context is allocated.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 11.0 GiB − 1.0 GiB reserve = 10.0 GiB.

At -ncmoe 26 and 65,536 tokens the sub-total is 8.891 GiB, which clears that by 1.109 GiB. That slack is the answer to "why 26 and not 24".

Choosing your own -ncmoe

Every row is weights + KV + recurrent state on a 12 GiB card, q8_0 KV, one slot. The last-but-one column applies the rule above — headroom against the 10.0 GiB working ceiling, after the 1024 MiB compute reserve:

Read the last two columns together. The Free of 12 GiB column is what the naive arithmetic says; the Slack column is what is actually left once the driver's share and the compute reserve come out, and the top two rows are the point — -ncmoe 20 at 32,768 tokens looks like it has 0.781 GiB to spare and in practice does not load. The bottom row is the model's entire declared window (context_length 262144) on a 12GB card, and it clears the ceiling; it costs four more steps of offload than the recipe's setting, which the next section prices in milliseconds.

If your board is headless and you want the throughput, -ncmoe 24 at 65,536 is a row in this table rather than a different recipe — but note it clears the ceiling by only 0.201 GiB, so it is the row most likely to be wrong if your driver's reserve is larger than the tier observation above.

This ladder is identical on any 12 GB card, and that is a finding rather than an omission. Nothing in it reads a clock, a bus width or a compute capability — it is the file's tensor table against a capacity. Where this card stops being interchangeable is the next section.

Then stop guessing and measure it. llama.cpp ships a tool that answers this question on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments:
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device line splitting total / free / model / context / compute / unaccounted. That compute column is the number this page reserved 1024 MiB for, and unaccounted is your driver's share — the two terms the table above cannot produce. If you run this, those two figures are the most useful thing you could send us, more useful than a speed number.

Where this card's speed actually goes

This is the section that is not the same on every 12 GB board, and it is why the capacity numbers above are interchangeable while the performance advice is not.

Decode: the card is not in the loop

At batch 1 the CUDA offload test fails, the offloaded expert matmuls stay on the CPU, and their weights never cross the bus. So a decode step splits across two memory systems, and the split is computable. expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token:

The VRAM row is an upper bound, because the token-embedding table is read one row at a time while the output head is read in full, and this page has not separated the two in this file. That the bound points upward matters twice below, so hold on to it.

The host side is therefore at least 15.87% of the bytes — 394,002,432 of 2,482,104,832 — and because the VRAM row is an upper bound, that share is a floor: the real host share can only be higher.

Now the threshold, which needs no bandwidth figure from anybody. The two halves of the decode roofline take equal time exactly when your card's bandwidth is 2,088,102,400 ÷ 394,002,432 = 5.30× your system RAM's. Since the VRAM side is an upper bound, the true crossover ratio is below 5.30 — so a board that computes out at exactly 5.30× is already past it, and the host half is already the larger one.

Here is that threshold as a card bandwidth you can compare against, using the two arithmetic recipes under Requirements:

Why this table is worth actually running on this board rather than waving at. NVIDIA publishes this card's memory interface as 192-bit and, in the same table, the RTX 4070 Ti SUPER's as 256-bit; elsewhere in the catalogue there are 12 GB boards on a 384-bit interface. Two boards at the same memory clock and twice the width differ by a factor of two in bandwidth, which is a factor of two in where they sit against the column above. On a 384-bit board the comparison is a foregone conclusion against anything you can put in two DIMM slots. On a 192-bit board with fast DDR5 it is not a foregone conclusion — it is a calculation, and the answer depends on which DIMMs you bought. Run the two lines under Requirements and put your own numbers in.

⚠️ What does not change either way, and this is the part to act on. Even at the threshold the host half is half the decode time while carrying under 16% of the bytes, and past it the host half is the majority — so on this recipe a faster GPU cannot fix decode, because decode was never on the GPU. What moves the number is DIMM bandwidth and channel count. Bandwidth-driven: none of this transfers to another card, and the table above is the tool for redoing it rather than a result to copy.

That also gives a ceiling that does not depend on the GPU at all — divide the 0.367 GiB of host traffic by your memory bandwidth:

⚠️ Those are ceilings from one term, not predictions. They exclude the CPU time to actually multiply those experts, kernel launch latency, the VRAM side, and everything else. The one measured 12 GB-tier expert-offload run this catalogue holds with a stated memory bandwidth — gemma4-26b on an RTX 3060, -ncmoe 12, DDR4-2133 dual-channel — achieved 37.2 tok/s, comfortably below what a host-bandwidth roofline would allow it. Read the table as "you cannot beat this", never as "you will get this". A single-channel configuration halves every row.

The same record is also the cleanest available demonstration that the mechanism is real, on this exact VRAM tier: that submitter's ladder is -ncmoe 12 = 37.23 tok/s against -ncmoe 30 = 21.20 tok/s. Eighteen more blocks on the host path, on one unchanged card, cost 43% of the generation rate. Different model, so the numbers are not yours; the direction and the size of the effect are the point.

Which makes offload depth the lever, priced per step. Moving one block from card to host frees its whole expert triple from VRAM — 0.4219 or 0.4863 GiB — and adds one thirty-second of it to every token's host read: 13.5 MiB for a cheap block, 15.6 MiB for an expensive one. On dual-channel DDR4-3200 that is about 0.28–0.32 ms per token each; on dual-channel DDR5-6000, about 0.15–0.17 ms. Going from the recipe's -ncmoe 26 to 30 for the full 262,144-token window adds four blocks — roughly 1.2 ms per token on DDR4-3200, 0.6 ms on DDR5-6000. Going the other way, to 24, is two blocks — 975,175,680 bytes of expert weight, so 29.1 MiB less host traffic per token, about 0.6 ms on DDR4-3200 and 0.3 ms on DDR5-6000 — and it costs 0.908 GiB of the headroom the previous section spent so carefully.

Prefill: the one phase where Ada's compute has a job

Here the bus binds instead. llama.cpp issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04." running "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26, prefill at pp2048 / ub2048" — and it finds that "prefill is bottlenecked by PCIe transfers, not GPU compute", with GPU idle around 42% of each prefill pass waiting on serial expert-weight H2D copies that, in its words, "run serially with the matmul that consumes them".

Take that at face value and it bounds what any amount of GPU compute can buy on this phase. If about 42% of a prefill pass is the GPU stalled on copies that faster arithmetic does not accelerate, then even an infinitely fast GPU leaves that 42% untouched, and the whole pass can shrink by at most 1 ÷ 0.42 = 2.38×. NVIDIA gives this board 7,680 CUDA cores at 2.61 GHz, which is a large budget by 12 GB standards — and 2.38× is the ceiling on what that budget can be worth here, before any of it is spent.

⚠️ Every caveat that report carries, carries here. It is a different model, a single box, PCIe 4.0 specifically, a custom fork used for the A/B, and its author dates the measurement to "around the b6xxx era", so the idle fraction may be stale against current master. Treat the 2.38× as the shape of the answer rather than its value. Bandwidth-driven, and specifically link-bandwidth-driven: it is the one number on this page that a wider or faster slot changes.

Two things you can act on, both of which point away from the GPU: put your DIMMs in dual channel and buy memory bandwidth before you buy silicon; and if time-to-first-token specifically is bad while generation is fine, look at the slot, because prefill is the only phase where the link is on the critical path.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true at common/common.h:476 — and moves MoE tensors to system memory until it can leave its 1024 MiB margin. That is a reasonable path, with two caveats. It records its choice nowhere except the log, so you cannot reproduce a run from the command line alone; and there is an open regression report against the margin flag on this same architecture family — llama.cpp issue #27171, filed 2026-08-16 against Qwen3.6-35B-A3B Q4_K_M with --fit-target, still open, labelled bug-unconfirmed, three comments, re-checked 2026-08-30, with a proposed fix (PR #27207) still unmerged.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp:484 throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line in the log means your offload is being honoured, not that something broke.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys you X GB" is wrong here. bartowski's imatrix recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28 and 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates, so the first few steps of -ncmoe and the last few buy more room than the ones in between. Read the cumulative column in the table above rather than multiplying.

That map is checkable against the publisher's own card, which says the MTP layers "are stored at Q4_0 in the imatrix quants" — and blk.40 in this file is indeed Q4_0 across all three expert tensors, the only block in the file that is.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 19,278,556,064 bytes — 17.955 GiB on disk, re-fetched 2026-08-30 — and its per-block quant mix is its own. If you would rather spend quality than throughput, that is the trade; derive its table from its own tensor list rather than scaling this one, because the unevenness above is a property of how one publisher built one file and nothing guarantees the next file is uneven in the same places. This page has not read that file's tensor table and states no ladder for it.

Does the arithmetic survive contact with a real card?

No one has run Apodex on an RTX 4070 Ti, and /check/apodex-1-1-mini/rtx-4070-ti returns unknown with zero benchmarks. The nearest thing to a test of the method is a different model on a 12 GB card, and our catalogue has one with a figure at four offload depths: Qwen3.6-35B-A3B at UD-Q4_K_M on an RTX 3060. That model is the same architecture shape as this one — 41 blocks, ten full-attention, identical ssm parameters and expert dimensions — so a weight ladder computed from its own tensor table is directly comparable:

⚠️ Read that right-hand column as GiB of weights, and do not read it as a peak. It tracks the derived weight ladder to within 0.16 GiB across a 9 GiB range. Read as decimal GB, every point falls below the weights that setting must load, which is impossible. And a figure that tracks the weights cannot also contain the KV cache and the compute buffers: at the top rung, 11.7 means the interval [11.65, 11.75] GiB against 11.542 GiB of derived weights, leaving at most 0.208 GiB for everything else — while that run's f16 KV cache alone was 0.168 GiB at its bench depth, plus 0.061 of recurrent state. (That depth is 8,832 tokens, read off the benchmark record's own command line — llama-bench … -p 512 -n 128 -d 0,4096,8192, benchmark id 278 — not off any recipe.) The excluded terms do not fit in what remains, so those numbers are floors, not peaks. As published on 2026-08-30, our own RTX 3060 recipe for that model built its context table on the 9.8 figure read as a measured peak; that reading does not survive this arithmetic, and no number from it appears anywhere on this page.

What the comparison does establish is the shape of the ladder, at four settings spanning 9 GiB. It is not four independent measurements — one submitter, one rig, one model, one tool — so every systematic error the method could have is shared across all four points.

Two other pages worth reading first

gpt-oss 20B on RTX 4070 Ti is this exact card and this exact technique on a much smaller model, and it is the cheap way to find out whether your box's CPU and DIMMs are up to an offloaded MoE at all — build llama.cpp, run that, then come back. If instead you have 24 GB and landed here by accident, Apodex 1.1 mini on RTX 3090 fits the whole model on the card with no offload, and none of this page's arithmetic applies there.

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this file, under llama.cpp, on any NVIDIA card, at any offload depth exists in the space searched on 2026-08-30, and the space was: the model cards and file listings of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces (re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100; it was 21 repos across 10 namespaces two days ago, so re-run it rather than trusting this sentence); the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 271 for qwen35moe and 727 for n-cpu-moe on the same query shape; and our own /check. Figures for the model do exist and every one of them is a different artifact on a different engine:

  • kingjones777 publishes a table covering four of his own repacks — 64.87 / 63.30 / 45.05 / 32.54 tok/s at 17.46 / 18.48 / 33.36 / 33.90 GiB — in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (Strix Halo, gfx1151), which is not a catalogue card. His own caption is worth keeping attached: "A dash means I haven't measured that one yet", and the STRIX_LEAN build that carries 64.87 in the shared table has a hand-written speed section on its own card that still shows a dash.
  • ghazni101 reports "generation measured at 226 tok/s end-to-end for MQ4R". ⚠️ Three disqualifications, all of them about the run rather than the file. It is a .mq4r container under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF, so nothing this page installs is what was timed; and it was measured on an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim. What is not a disqualification, though the repo's name invites the mistake: the 226 is for MQ4R and the repo is called -MQ4RP, but it ships both builds — apodex-1.1-mini-mq4r.mq4r (18,700,048,128 B) and apodex-1.1-mini-mq4rp.mq4r (18,778,601,728 B) — under a card headed "Apodex-1.1-mini — MQ4R & MQ4RP quants". So the figure does measure a file that repo hosts. The same card's "~235 tok/s (MQ4R) vs ~212 tok/s (MQ4RP)" pair is a different model — its Ornith reference builds — and belongs to neither this page nor that one.

  One figure on this pair would be worth more than all of the above. If you run this, contribute it.

• VRAM usage: 8.891 GiB derived sub-total at -ncmoe 26 and a 65,536-token context, plus the compute buffers and driver share this page reserves 1024 MiB and ~1 GiB for respectively but cannot derive. See /check/apodex-1-1-mini/rtx-4070-ti for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: with 26 of 40 blocks' experts on the host, decode is paced by your CPU and system-memory bandwidth and not by this card — see the ceiling table above, and treat it as a ceiling. Prefill is where this board's 7,680 cores have something to do, and even there a profiled run of this shape leaves at most a 2.38× improvement available to compute.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated against "a calibration corpus rendered through this model's own chat template" that "pairs plain prose with tool-calling and reasoning conversations", which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-4070-ti.

Troubleshooting

First, check you have the 12 GB card and not the 16 GB one wearing its name

NVIDIA's 40-series spec table lists the RTX 4070 Ti SUPER immediately beside the RTX 4070 Ti, and it is a different part: 8,448 CUDA cores, 16 GB GDDR6X on a 256-bit interface, against this card's 7,680 and 12 GB on 192-bit. Every row of the ladder above is keyed to a capacity, and the speed section is keyed to an interface width, so both change if you have the SUPER. Our catalogue has already had to draw this distinction on this card once — MiniMax H3 on RTX 4070 Ti is named for it.

nvidia-smi --query-gpu=name,memory.total --format=csv

If that reports roughly 16384 MiB, you have 4 GiB more to play with, and the ladder shifts rather than scales: at -ncmoe 20 and 65,536 tokens the sub-total is 10.826 + 0.664 + 0.061 = 11.551 GiB, which the 12 GB ceiling rule refuses and a 16 GB one does not. Recompute from the Weights on GPU column, which is capacity-independent, and re-apply the ceiling rule against your own usable pool rather than against this page's 10.0 GiB.

"Decode collapsed at long context" — measure it again before you believe it

This is the misdiagnosis this recipe makes most likely, and there is a fully worked public example of a careful person making it.

On an offloaded run prefill is slow, for the reasons above. So if you compute tokens per second as completion tokens ÷ total request time, a long prompt dominates the answer and decode looks broken. A 68,000-token fill at ~1,500 t/s of prompt processing is about 45 seconds by that arithmetic — the reporter quoted below records ~43 s for exactly that shape — and putting a 100-token generation behind it makes the division report roughly 2 tok/s no matter how fast decode actually was.

llama.cpp issue #27623 is that mistake, in public, over three days. It was filed 2026-08-23 against a qwen35-family hybrid Gated-DeltaNet model on an RTX 4080 SUPER, sm_89 — the same compute capability as this board — claiming decode collapsed about 25× past roughly 80K KV position while prompt processing stayed fast. Two people failed to reproduce it, on 2×RTX 3090 (sm_86) and on an RTX 5090 (sm_120). The reporter then built a driver-regression theory, tested three driver branches against it, and on 2026-08-26 retracted the whole thing:

"The numbers in my matrix were completion_tokens / total request time, which includes prompt processing (~43s for a 68K fill at ~1500 t/s pp)." — and so, in the same breath, "it was prefill time, not decode".

With server-side eval-only timings the same runs were healthy everywhere — 32.4 tok/s at a 91K position — and the reporter closes with "So the decode cliff does not reproduce on any combination available to me now, including the exact original commit."

⚠️ Read the scope before you take anything from this. It is a different model (Qwen3.8-27B), on a 16 GB card, fully GPU-resident with zero CPU tensor overrides — the opposite of what this recipe does — so none of its numbers is a number for this page, including the 32.4. It is here for two reasons. First, the metric: read llama-server's own eval time line, or use llama-bench, and never divide completion tokens by wall-clock on an offloaded run. Second, because the issue is still open as of 2026-08-30 with the retraction as its sixth and last comment — so a reader searching for "qwen35moe slow long context sm_89" meets the alarming headline and the wrong driver theory several screens before they meet the correction.

Out of memory once a long prompt lands, but not at load

Two different causes, and they need opposite fixes.

The KV cache grows with the prompt at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the table above and either raise -ncmoe or lower -c.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means copying those tensors into a duplicate the graph allocator sizes for the run. Each of this file's expert triples is 0.4219–0.4863 GiB, so prefill peaks materially above the generation footprint the table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those operations on the CPU where their weights already are; expect slower prefill in exchange, and note that this is the one place a wider slot was helping you.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it, not one.

The first is the offload itself: at -ncmoe 26 this recipe puts 11.742 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second is the one that catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache and it is on by default at 8192 MiB — cache_ram_mib = 8192 in common/common.h:632, exposed as -cram / --cache-ram with -1 for no limit and 0 to disable (added by PR #16391, "server : host-memory prompt caching", whose body says the cache "is stored in regular RAM"). So a default llama-server may hold up to 8 GiB of RAM beyond the offload — with this recipe's 11.742 GiB that is close to 20 GiB of host memory before the operating system gets any, which is why the command above pins it lower.

And an entry here has a floor. Saving a prompt calls llama_state_seq_get_data_ext on the whole sequence state (tools/server/server-context.cpp), and on a hybrid model llama_memory_hybrid::state_write writes both children — the attention cells at :192 and the recurrent memory at :194, the latter unconditionally. So every cached prompt carries its tokens' KV at your -ctk/-ctv type plus a full copy of the 62.8 MiB recurrent state, whatever its length; the fixed copy is the larger half of an entry until roughly 6,000 tokens, where 10,880 B/token overtakes it.

There is one direct measurement of this growth and it is worth reading before you size your box. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. Closed as completed on 2026-08-28. ⚠️ Two cautions if you cite it yourself. The 126 MiB is the only measured per-entry figure anywhere and it is roughly twice one recurrent state (62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support — the 126 stands as a measurement, its composition does not. And it was measured on a different model of the same architecture, and on ROCm; the reporter rebuilt the same source with -DGGML_VULKAN=ON and got +126.0 MiB per request against ROCm's +126.3, so this is a model-and-request-layer behaviour and not a vendor story.

So: 32GB is the practical floor for this recipe as written, and 16GB is not enough. On a 32 GB box keep -cram at 2048 as above, or pass -cram 0 to turn caching off entirely and trade prompt-reuse speed for headroom. Lowering -ncmoe moves weight back onto the card and off the host; the one lever that helps both sides at once is a smaller quant.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE. Check llama-server --version against the release list and move to b10630 or newer.

The model emits !!!! or uniform-probability gibberish

Searching the llama.cpp tracker for this architecture string turns up several gibberish-shaped reports, and the first thing to do with any of them is check which backend it is about. The nearest one to this recipe is issue #25857, filed 2026-07-18 on a Windows 12GB NVIDIA box: every token at an identical log-probability of −12.42 — the uniform distribution over the vocabulary — at any offload setting including pure CPU. Read its scope before you inherit the symptom. It is against a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins; it remains open with the stale label and zero comments as of 2026-08-30; and a separate publisher reports coherent output from an Apodex Q4_K_M on an older build still (vcruz305 smoke-verified -ngl 80 on b9835, on an RTX 6000 24GB). The other reports in that neighbourhood are on other backends — #24168 is SYCL on an Intel Arc card, #23321 is Vulkan with --no-kv-offload and is closed — so neither describes a CUDA build. Rebuild at a current release first.

For completeness on the one issue this catalogue's earlier Apodex pages pointed at when they were written: llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label. It was open when our RTX 3090 and RTX 5090 pages were written and it is not now. It also never applied to a text-only run — which is what --no-mmproj makes this one.

Throughput far below what your CPU should manage

First rule out the measurement itself, using the entry above — on this recipe a wall-clock rate is dominated by prefill and will look like a decode failure.

Then check three things, and note that none of them is the GPU. -ncmoe too high pushes more blocks onto the host path than you need — each step is 13.5–15.6 MiB more read from RAM per token. Single-channel system memory halves the bandwidth those reads get, which halves the ceiling table above. And if you are using auto-fit rather than an explicit -ncmoe, pin the offload yourself rather than passing --fit-target, because of issue #27171.

If it is specifically the wait before the first token that is bad while generation itself is fine, look at the slot rather than the model: prefill is the only phase where the expert weights cross the bus, so a card in a chipset-fed ×4 slot or on a riser pays there and nowhere else.

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp will drive it with --spec-type draft-mtp — the exact spelling matters. On a 24 GB card that is a reasonable experiment. On this one, price it first, because the costs land on the scarcest resource you have:

• blk.40 stops being skipped: +0.444 GiB, and note that -ncmoe 26 does not touch it. The flag installs overrides for blk.0 through blk.25; only -ncmoe 41 would reach block 40's experts.
• The recurrent state gains one slot per unit of draft depth, on top of the one you already had. need_n_rs_seq() at :394 returns draft.n_max, and the allocator then sizes the cache at mem_size * (1 + n_rs_seq) (src/llama-memory-recurrent.cpp:101) — note the 1 +, which is the difference between a right answer and a wrong one here. So --spec-draft-n-max 3 allocates four sequences' worth, not three: 0.061 GiB becomes 0.245, an increase of 0.184.
• A second, single-layer KV cache is allocated for the draft context. It is f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k / -ctkd (common/arg.cpp:4090) — so at 65,536 tokens on one layer it is 2,048 bytes per token, 0.125 GiB.

That is 0.753 GiB out of the 1.109 GiB of slack this page reserved, before any throughput arrives — which on its own is close to disqualifying. And the return is unknown here: nobody has measured MTP on this model on any card, and the mechanism argues against it. Speculative decoding wins when the GPU is waiting on itself; under expert offload it is waiting on your RAM, and the draft pass walks the same host-resident experts the target pass does. The nearest evidence agrees and is on a 12 GB card, though on a different model — an owner who describes their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" and runs Qwen3.6-35B-A3B at --n-cpu-moe 25 with --spec-type draft-mtp --spec-draft-n-max 2 reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" in discussion #18 on that GGUF repo, and a second community member in the same thread offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model, so read it as a warning rather than a result. Measure both ways on your own prompts before keeping it, and please tell us what you get.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps publishes a hardware-guidance table whose 12 GB row names "IQ1_M or IQ2_M" with the note "IQ1_M full offload; IQ2_M may require hybrid offload". Weigh it with the rest of that card, which is candid about the cost — "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." — and which ends the same table with "Prefer Q4_K_M or higher when system memory permits."

That closing sentence is this recipe. A 1.75-bit quantisation of a model whose routed experts are 512 wide, used for tool-calling where a malformed argument is a failed task rather than a clumsy sentence, is a large quality bet to avoid a memory purchase. And check what the smaller quant actually buys before taking it. IQ2_M is 12,241,706,816 bytes (11.401 GiB) in that publisher's own ladder and 12,543,404,960 bytes (11.682 GiB) in the lead repo — either way, on a card whose usable pool this page sizes at ~11.0 GiB, that leaves nothing for the KV cache or the compute buffers, which is what their own table means by "may require hybrid offload". Below Q4 the trade is quality for a shallower -ncmoe, not quality for no -ncmoe at all. Only IQ1_M — 8,821,679,936 bytes, 8.216 GiB — escapes offload entirely, and it is the one its own card tells you to avoid when you can.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well (checked 2026-08-30, with ollama.com/library/qwen3.5 and the registry.ollama.ai manifest for qwen3 both returning 200 as controls). Use llama.cpp directly. If a tag appears later, please tell us.