Apodex 1.1 mini on RTX 3060: a 35B-A3B agent model on 12GB via expert offload

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one 12GB RTX 3060, at a 65,536-token context. The Q4_K_M weights are 19.908 GiB. The card holds 12 GiB. It runs anyway, because 91.2% of those weights are routed experts that llama.cpp can leave in system RAM.

Hardware data: RTX 3060 (12GB VRAM) · 8.891 GiB derived working set at -ncmoe 26, 65,536-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 3060. /check/apodex-1-1-mini/rtx-3060 holds zero benchmarks. Every VRAM figure below is arithmetic over the GGUF's own tensor table and llama.cpp's allocation code, and it is shown so you can check it. There is no throughput number on this page at all — see Results for what exists and why none of it transfers. If you run this, please send the numbers.

ℹ️ This recipe is text-only. The checkpoint has a vision tower and the GGUF repo ships a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card contains no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-30). On a card this tight the projector is also 0.838 GiB you cannot spare, so the run command below refuses it explicitly rather than merely omitting the flag.

Requirements

Disk size and VRAM residency are different numbers here and they are not interchangeable. The file is 20.363 GiB on disk; a normal run loads 19.908 GiB of it, because blk.40 — the multi-token-prediction head — is created with TENSOR_SKIP unless you ask for it (src/models/qwen35moe.cpp). Budget 22 GB of disk and 19.908 GiB of weights.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. Use b10630 or newer — the release the lead quant was produced with.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes, imatrix-calibrated, produced on llama.cpp b10630.

Do not start from the publisher's own quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card states plainly that "llama.cpp downloads the mmproj automatically when using" -hf. llama-server is in llama.cpp's mmproj_examples list, no_mmproj defaults to false and mmproj_use_gpu defaults to true (common/common.h), so that command puts 0.838 GiB of vision projector on a card this recipe has already budgeted to within a few hundred megabytes. Download the file explicitly, as above, and load it with -m.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 26 \
    -np 1 \
    -c 65536 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Eight of those flags are decisions rather than defaults, and on a card this size each one is worth its line:

• --no-mmproj refuses the vision projector rather than merely not asking for it — 0.838 GiB, for the reason given under Installation.
• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 26 keeps the routed experts of blocks 0–25 in system RAM. llama.cpp's own help text for the flag is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp), and the implementation matches the wording exactly — see How the offload actually works below. 26 is derived in Choosing your own -ncmoe; it is not a measured optimum.
• -np 1. llama-server sets its slot count to auto and resolves that to 4 slots with a unified KV pool when you do not say otherwise (common/arg.cpp sets n_parallel = -1 for the server). The Gated-DeltaNet recurrent state is allocated per sequence, so four slots quadruple it — 0.245 GiB instead of 0.061 — and split your context four ways. One slot is what a single-user agent loop wants.
• -ctk q8_0 -ctv q8_0 halves the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 65,536-token cache from 1.250 GiB to 0.664 GiB.
• -fa on makes Flash Attention explicit. The default is auto at this build (common/arg.cpp); the one measured expert-offload run this page compares against on the same card had it on.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader itself warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp). Taking that advice means the 11.742 GiB of expert weights are ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure rather than a cache you can squeeze.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB and it is more expensive on this architecture than on a plain transformer — see the host-RAM entry under Troubleshooting. The RTX 3060 owner quoted further down passes --cache-ram 4096 in their own config, without saying why.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05). The card notes that Apodex "follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF and enables Jinja by default, so tool calls and <think> blocks parse the way the vendor intends. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It builds one regex per block and hands them to the model loader as buffer-type overrides (common/common.h):

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, …) pushes "blk\.0" + regex … "blk\.(N-1)" + regex

The loader then does a plain std::regex_search on each tensor name and, on a match, allocates that tensor in a CPU buffer instead of the GPU one (src/llama-model-loader.cpp). Nothing in that path consults the architecture, so it applies to this hybrid recurrent model exactly as it does to a plain MoE. 123 tensors in this file match — three per block across all 41 blocks — totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads. That is 91.2% of the 19.908 GiB resident weight.

What does not move is as important. Taking blk.5 as an example, the offload takes ffn_down_exps, ffn_gate_exps and ffn_up_exps, and leaves on the GPU: the attention path (attn_qkv, attn_gate), the recurrent state tensors (ssm_conv1d, ssm_a, ssm_alpha, ssm_beta, ssm_norm, ssm_out), the router (ffn_gate_inp), the shared expert (ffn_*_shexp) and every norm. So routing is decided on the GPU, the Gated-DeltaNet state stays on the GPU, and only the selected experts' matmuls happen elsewhere. Offload everything it can reach — -ncmoe 40 — and 1.744 GiB is still resident, made of 0.695 GiB of attention, 0.655 GiB of embeddings and output head, 0.205 GiB of recurrent-state tensors, 0.111 GiB of shared experts and 0.078 GiB of routers.

"Elsewhere" is two different places depending on what you are doing, and this is the fact that governs how the recipe performs. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights", per the scheduler's own comment (ggml/src/ggml-backend.cpp) — unless a higher-priority backend wants to offload it, which CUDA does once the op's batch size reaches 32 (ggml/src/ggml-cuda/ggml-cuda.cu, GGML_OP_OFFLOAD_MIN_BATCH, default 32). Generating a token is a batch of one, so decode reads the offloaded experts from system RAM at host memory bandwidth. Prompt processing is a batch of hundreds, so prefill copies those tensors across PCIe to the GPU, into a duplicate the graph allocator sizes for the run. Two consequences you can act on: your decode rate is set by your CPU and your RAM, not by this card; and your peak VRAM during a long prompt is higher than during generation, which is the subject of the last Troubleshooting entry.

The VRAM budget

Weights, KV cache and recurrent state, at the recipe's settings. Nothing here was measured on hardware, and the table excludes llama.cpp's compute buffers and the CUDA context — see below for how to get those on your own machine.

Against 12 GiB that leaves 3.109 GiB. The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 26 moves 12,608,077,824 of that (11.742 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks are loaded, and full_attention_interval 4 makes ten of them full-attention: blocks 3, 7, 11, 15, 19, 23, 27, 31, 35 and 39. Each carries head_count_kv 2 at key_length 256 and value_length 256 — 1,024 cache elements per token per layer, so 10,240 across ten layers. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows.
• Recurrent state. Those thirty blocks each allocate two F32 tensors, sized (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements (src/llama-hparams.cpp). That is 2,195,456 bytes per layer per sequence and 65,863,680 across all thirty. llama-server prints it at startup as RS buffer size.

Choosing your own -ncmoe

The whole point of the flag is that it trades speed for headroom, so here is the trade priced out. Every row is weights + KV + recurrent state on a 12 GiB card, q8_0 KV, one slot:

The bottom row is the model's entire declared window — context_length 262144 — on a 12GB card. It costs four more steps of offload than the recipe's setting, and four more steps means four more blocks' experts read from RAM on every token.

Leave real headroom. The free column is not spare capacity; it is what the compute buffers, the CUDA context and your desktop have to fit into, and none of those is in the table. Two data points for scale: llama.cpp's own auto-fit tries to leave a 1024 MiB margin per device by default (common/common.h), and the worked example in tools/fit-params/README.md shows a 30B-class MoE on an RTX 4090 reporting 898 MiB of compute buffer beside 384 MiB of context at a 4,096-token window — a different model on a different card, quoted only to say that this term is hundreds of megabytes, not tens. And this card's usable ceiling is not 12 GiB. Our catalogue holds a second expert-offload run on an RTX 3060 — gemma4-26b at -ncmoe 12, a community submission from the same rig as the sweep discussed below — and it peaked at 11,179 of the card's 12,288 MiB on a headless box, with the submitter recording that -ncmoe 8 and below "fail to load outright" rather than merely slowing down. So 1,109 MiB — 1.083 GiB — of an RTX 3060 is spoken for before your model starts, with nothing driving a display. Treat ~10.9 GiB as the ceiling a working configuration has actually reached on this silicon, and the -ncmoe 20 row's 0.781 GiB of free space as out of reach rather than as a tight fit.

Then stop guessing and measure it. llama.cpp ships a tool that answers this question on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments:
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device line splitting total / free / model / context / compute / unaccounted. That last column is everything on the card that is not this process — your display, other applications — and on a 12GB card it is the difference between the arithmetic above and what you actually have.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default (fit_params = true) and moves MoE tensors to system memory until it can leave its margin. That is a reasonable path, with two caveats. It records its choice nowhere except the log, so you cannot reproduce a run from the command line alone; and there is an open regression report against the margin flag on this same architecture family — llama.cpp issue #27171, filed 2026-08-16 against Qwen3.6-35B-A3B Q4_K_M with --fit-target, still open and unconfirmed at the time of writing, with a proposed fix (PR #27207) still unmerged.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line in the log means your offload is being honoured, not that something broke.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys you X GB" is wrong here. bartowski's imatrix recipe upcasts ffn_down_exps to Q6_K in exactly half the loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28 and 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates, so the first few steps of -ncmoe and the last few buy more room than the ones in between. Read the cumulative column in the table above rather than multiplying.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 17.955 GiB on disk and 17.500 GiB resident, and its ladder is perfectly linear at 0.3984 GiB per step: -ncmoe 20 leaves 9.531 GiB on the card, roughly what Q4_K_M leaves at 22, with two to four fewer blocks on the host path. If you would rather spend quality than throughput, that is the trade — but derive its table from its own tensor list, do not scale this one.

Does the arithmetic survive contact with a real card?

No one has run Apodex on an RTX 3060. But the same arithmetic, applied to a different model on this card, can be checked against a measurement we hold, and it comes out well.

The RTX 3060 carries three expert-offload benchmarks in our catalogue, and one of them reports a VRAM figure at more than one offload depth: Qwen3.6-35B-A3B at UD-Q4_K_M, a four-rung -ncmoe sweep with a number at every rung. (The gemma4-26b record quoted above gives one; the third, a --n-cpu-moe 2 figure for gpt-oss-20b, stores no peak at all.) That model is the same architecture shape as this one — 41 blocks, ten full-attention, identical ssm parameters and expert dimensions — so its own tensor table yields a ladder computed exactly as the one above. Setting the two side by side:

Read the reported column as GiB and it tracks the derived weights within 0.16 GiB across a 9 GiB range. Read it as decimal GB and every single point falls below the weights that setting must load, which is impossible. And the unit does not have to stay an inference: the same submitter, on the same rig, filed the gemma4-26b run cited above, and that record states its peak in MiB — 11,179 of 12,288 — which the catalogue stores as 10.92. Their "GB" is GiB. So two things follow, and the second one matters more than the first. The ladder predicts what a card in this class actually shows — four settings spanning 9 GiB, which is a real test of the ladder's shape. It is not four independent measurements: one submitter, one rig, one model, one tool, so every systematic error the method could have is shared across all four points.

And a figure of that kind is a floor, not a peak. Take the top rung, where the margin is widest relative to the rounding: the reported 11.7 means the interval [11.65, 11.75] GiB, against 11.542 GiB of derived weights — so at most 0.208 GiB is left for everything else, while that run's f16 KV cache alone is 0.168 GiB — the sweep was benched at 8,832 tokens, 512 + 128 + 8,192 — and its recurrent state another 0.061. The excluded terms do not fit in what remains, at either end of the interval. (Do not run the same argument at -ncmoe 32, where the derived 6.104 sits 4 MiB under a reported 6.1: one decimal place is a ±51 MiB interval, twelve times the gap, and the comparison says nothing.) Do not read that page's numbers, or this page's table, as a peak you can fit a card to.

The same arithmetic also explains the failure in that sweep. Its -ncmoe 20 point loads 11.542 GiB of weights, leaving 0.458 GiB on a 12 GiB card — and its f16 KV cache at a 32,768-token context needs 0.625 GiB. The submitter's note reads exactly that way: the setting benchmarks fastest and then dies once real context arrives.

Why this page says -ncmoe 26 and the Qwen3.6 page says 24

Our RTX 3060 recipe for Qwen3.6-35B-A3B leads with -ncmoe 24 on the same card with the same flag, so the difference is worth stating rather than leaving for a reader to trip over. The two pages also disagree about one number, and it is worth knowing which before you compare them: that recipe builds its context table on the 9.8 figure read as a measured peak, and the section above shows it is a weights-only floor. The method is shared; the base is not. Beyond that, three reasons:

1. That page has a measurement at 24 and this one does not. Its setting is the one a submitter actually ran; changing it would throw away the only thing that page has. This page has arithmetic only, so it can afford to be more conservative, and should be.
2. Different files. Its weights at -ncmoe 24 are 9.729 GiB; ours are 9.074 GiB. Ours at 26 are 8.166.
3. A bigger context and an unmeasured compute term. That page documents 32,768 tokens with an f16 cache; this one documents 65,536 with q8_0, on a card where the compute buffers are the one term nobody has. -ncmoe 26 buys 0.908 GiB against 24 for two blocks' worth of host traffic, and on a card that may also be driving a display that is the right way round.

If your 3060 is headless and you would rather have the throughput, -ncmoe 24 at 65,536 tokens is 9.799 GiB and 2.201 GiB free — a row in the table above, not a different recipe.

Results

• Speed: omitted, and the reason is a predicate rather than a count: no throughput figure for this file, under llama.cpp, on any NVIDIA card, at any offload depth exists in the space searched — the model cards and discussion tabs of all 21 HuggingFace repositories matching Apodex-1.1-mini (enumerated 2026-08-30; the discussion tabs hold one thread between them, an announcement posted by a community user — not the vendor, and not a measurement), the llama.cpp issue tracker, which returns nothing for "apodex" while the same query shape returns hundreds for the architecture string, and our own /check. Figures for the model do exist, and every one of them is a different artifact on a different engine. kingjones777 publishes a table covering four of his own repacks — STRIX_LEAN 17.46 GiB at 64.87 tok/s, COHERENT 18.48 GiB at 63.30, Q8_0 33.36 GiB at 45.05 and Q8_0-AGENT 33.90 GiB at 32.54 — all in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (gfx1151), at -ngl 999 with 128-token greedy generation. That shared table states no context length and no idle-box condition — but each build's own card does, in a Corrected speed section giving the same idle machine at 32K for 63.30, 45.05 and 32.54. 64.87 is the only one of the four without that paragraph — the same build whose card says the measurement has not been made. It still carries the shared table's conditions; what it lacks is a card of its own saying when and on what it was run. ⚠️ And 64.87 should be read with its own card's warning attached: the STRIX_LEAN repo's hand-written speed section shows a dash and says "Full-offload speed being measured on an idle box, card will be updated", while the shared variants table thirty lines below prints 64.87 for the same build under a caption promising "A dash means I haven't measured that one yet". The file asserts both; the likeliest reading is a synced table outpacing a hand-edited section, but the card does not say so. ghazni101 separately reports a "generation measured at 226 tok/s end-to-end for MQ4R" on an RX 7900 XTX — a catalogue card, but from a .mq4r file served by hipfire, a Rust engine that is not llama.cpp. None of these was measured on the Q4_K_M this page installs, so none of them is this page's number. kingjones777 also posted 1.183, 0.292 and 0.163 t/s for three of those builds earlier and retracted them in place as an artifact of a box serving eight other models — a useful reminder that a partial-offload timing measures your host, not the model. One figure on this pair would be worth more than all five above: if you run this, contribute it.
• VRAM usage: 8.891 GiB derived working set at -ncmoe 26 and a 65,536-token context, plus compute buffers this page cannot derive. See /check/apodex-1-1-mini/rtx-3060 for live data as it lands.
• What to expect anyway, stated as a mechanism rather than a number: with 26 of 40 blocks' experts on the host, decode is paced by CPU and system-memory bandwidth, not by this card. Dual-channel memory matters more than the GPU does; a single-channel configuration reads those experts at half the bandwidth. That is the same mechanism the RTX 3060 submitter on the sibling model attributed their own flat-through-8K curve to.
• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated against a corpus rendered through this model's own chat template with tool-calling conversations at 63.8% of the chunks, which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-3060.

Troubleshooting

Out of memory once a long prompt lands, but not at load

Two different causes on this card, and they need opposite fixes.

The KV cache grows with the prompt at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the table above and either raise -ncmoe or lower -c.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means copying those tensors into a duplicate that the graph allocator sizes for the run. Each of this file's expert triples is 0.42–0.49 GiB, so prefill peaks materially above the generation footprint the table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those operations on the CPU where their weights already are; expect slower prefill in exchange.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it, not one.

The first is the offload itself: at -ncmoe 26 this recipe puts 11.742 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second is the one that catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache and it is on by default at 8192 MiB — cache_ram_mib = 8192 in common/common.h, exposed as -cram / --cache-ram with -1 for no limit and 0 to disable (added by PR #16391, "server : host-memory prompt caching", whose body says the cache "is stored in regular RAM"). So a default llama-server may hold up to 8 GiB of RAM beyond the offload — with this recipe's 11.742 GiB that is close to 20 GiB of host memory before the operating system gets any, which is why the command above pins it lower.

And an entry here has a floor. Saving a prompt calls llama_state_seq_get_data_ext on the whole sequence state (tools/server/server-context.cpp), and on a hybrid model llama_memory_hybrid::state_write writes both children — the attention cells and the recurrent memory, the latter unconditionally. So every cached prompt carries its tokens' KV at your -ctk/-ctv type plus a full copy of the 62.8 MiB recurrent state, whatever its length: the fixed copy is the larger half of an entry until roughly 6,000 tokens, where 10,880 B/token finally overtakes it. That floor is the thing to budget for. It is not a claim that caching is dearer here than on an ordinary transformer — the opposite is true at the lengths this page documents, because only 10 of 40 blocks cache anything that grows: against a same-shaped model that cached all forty, this architecture's 10,880 B/token repays the fixed 62.8 MiB after about 2,000 tokens, and this recipe's context is thirty times that. The server prints the real per-save figure (total state size = … MiB) only with verbosity raised — it is a trace-level line — if you want to watch it.

So: 32GB is the practical floor for this recipe as written, and 16GB is not enough. On a 32 GB box keep -cram at 2048 as above, or pass -cram 0 to turn caching off entirely and trade prompt-reuse speed for headroom. The RTX 3060 owner in discussion #18 runs --cache-ram 4096, but read their first line before copying it — rtx 3060 12G+96G RAM. Same card, same offload approach, three times this recipe's system memory, which is exactly why they can afford the larger cache. Lowering -ncmoe moves weight back onto the card and off the host, and the one lever that helps both sides at once is a smaller quant.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE. Check llama-server --version against the release list and move to b10630 or newer.

The model emits !!!! or uniform-probability gibberish

Searching the llama.cpp tracker for this architecture string turns up several gibberish-shaped reports, and the first thing to do with any of them is check which backend it is about. The nearest one to this recipe is issue #25857, filed 2026-07-18 on a Windows 12GB NVIDIA box: every token at an identical log-probability of −12.42 — the uniform distribution over the vocabulary — at any offload setting including pure CPU. Read its scope before you inherit the symptom. It is against a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins; it is open with stale and no comments; and a separate publisher reports coherent output from an Apodex Q4_K_M on an older build still (vcruz305 smoke-verified -ngl 80 on b9835). The other reports in that neighbourhood are on other backends — #24168 is SYCL on an Intel Arc card, #23321 is Vulkan with --no-kv-offload and is closed — so neither describes a CUDA build. Rebuild at a current release first.

Throughput far below what your CPU should manage

Check three things before suspecting the model. -ncmoe too high pushes more blocks onto the host path than you need — every step is one more block read from RAM per token. Single-channel system memory halves the bandwidth those reads get. And if you are using auto-fit rather than an explicit -ncmoe, drop --fit-target and pin the offload yourself, because of issue #27171.

If it is specifically the wait before the first token that is bad while generation itself is fine, look at the slot rather than the model: prefill is where the expert weights cross the bus, so a card in a chipset-fed ×4 slot or on a riser pays there and nowhere else. nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv tells you what you actually negotiated.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp will drive it with --spec-type draft-mtp — the exact spelling matters. On a 24 GB card that is a reasonable experiment. On this one, price it first, because the costs land on the scarcest resource you have:

• blk.40 stops being skipped: +0.444 GiB, and note that -ncmoe 26 does not touch it. The flag installs overrides for blk.0 through blk.25; only -ncmoe 41 would reach block 40's experts.
• The recurrent state is multiplied by the draft depth — need_n_rs_seq() returns draft.n_max — so --spec-draft-n-max 3 takes 0.061 GiB to 0.245.
• A second, single-layer KV cache is allocated for the draft context. It is f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k / -ctkd — so at 65,536 tokens on one layer it is 2,048 bytes per token, 0.125 GiB.

That is 0.753 GiB out of 3.109 before any throughput arrives, and the return is unknown here: nobody has measured MTP on this model on any card. The nearest evidence is discouraging and it is on this exact card, though on a different model — an RTX 3060 12GB owner running Qwen3.6-35B-A3B at --n-cpu-moe 25 with --spec-type draft-mtp --spec-draft-n-max 2 reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" in discussion #18 on that GGUF repo, and a second community member in the same thread offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model, so read it as a warning rather than a result — but the mechanism is the one this page has already described. Speculative decoding wins when the GPU is waiting on itself; under expert offload it is waiting on your RAM. Measure both ways on your own prompts before keeping it, and please tell us what you get.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps publishes a hardware-guidance table whose 12 GB row points at IQ1_M or IQ2_M rather than at this recipe's path. Weigh it with the rest of that card, which is candid about the cost — "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." — and which ends the same table with "Prefer Q4_K_M or higher when system memory permits."

That closing sentence is this recipe. A 1.75-bit quantisation of a model whose routed experts are 512 wide, used for tool-calling where a malformed argument is a failed task rather than a clumsy sentence, is a large quality bet to avoid a memory purchase. And check what the smaller quant actually buys before taking it. IQ2_M is 11.401 GiB on disk in that publisher's ladder and 11.682 GiB in the lead repo — either way, on a 12 GiB card that leaves nothing for the KV cache or the compute buffers, which is what their own table means by marking the 12 GB row "IQ2_M may require hybrid offload". Below Q4 the trade is quality for a shallower -ncmoe, not quality for no -ncmoe at all. Only IQ1_M — 8,821,679,936 bytes, 8.216 GiB, the smallest rung in that publisher's ladder — escapes offload entirely, and it is the one its own card tells you to avoid when you can.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well (checked 2026-08-30, with ollama.com/library/qwen3.5 returning 200 as a control). Use llama.cpp directly. If a tag appears later, please tell us.