Apodex 1.1 mini on RTX 4080: a 36B agent in 16 GB via llama.cpp expert offload

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single RTX 4080, at a 131,072-token working context, from a Q4_K_M GGUF that is larger than the card.

That last part is the page. The Q4_K_M file loads 19.908 GiB of weights and the RTX 4080 holds 16 GiB, so nothing here is a fit calculation in the usual sense. It runs because the model is a sparse Mixture-of-Experts and llama.cpp will leave the routed expert tensors of the first N blocks in system RAM: 123 tensors carry 18.586 GiB of the file, and across the forty loaded blocks that is 18.164 GiB of the 19.908 GiB resident — 91.2 %. Everything the card actually needs to keep — attention, the Gated-DeltaNet mixers, the shared experts, the router, the embeddings and the output head — is 1.744 GiB. The recipe is a choice of how much of that 91.2 % to push across, and what the choice buys you in context.

Hardware data: RTX 4080 (16 GB VRAM) · 13.546 GiB derived working set at -ncmoe 17, 131,072-token context · See benchmark data

⚠️ This model does not fit 16 GB on its own, and system RAM is a prerequisite rather than a footnote. At the setting below, 7.752 GiB of expert weights are resident in host RAM for as long as the server runs, and llama-server's prompt cache can commit several GiB more on top (see System RAM). bartowski's own sizing rule for this file is to "add both your system RAM and your GPU's VRAM together" and pick a quant a couple of GB below the sum (model card); with 20.363 GiB of weights that is satisfied on paper by 16 GB of RAM, and in practice you want 32.

ℹ️ This recipe is text-only, deliberately. The checkpoint contains a vision tower and the GGUF repos ship a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions at all — the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. Every number below is for text generation, and the run command refuses the projector explicitly rather than merely omitting --mmproj. Our RTX 3090 page sets out the same reasoning at length.

⚠️ Two version floors, about 2,200 builds apart. They are quoted here identically to the RTX 3090 page, because they were bisected once and they are properties of the model rather than of the card.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990, so it arrived in (b7989, b7990].
• The qwen35moe-specific MTP wiring — the load_mtp / TENSOR_SKIP handling in src/models/qwen35moe.cpp that keeps block 40 off the card until you ask for it — absent at b10211, present at b10212, so it arrived in (b10211, b10212].

Both are written as intervals on purpose: each is bracketed by adjacent published tags, which is as fine as a tag-granularity probe can resolve. Use b10630 or newer — the build the lead quant was produced with. That is a sufficient floor, not a bisected minimum: every flag on this page is older than it, and this page does not claim to know which release first carried them all. Source references below were read at release b10666 (4e97ac86ebe2c4cb8212d98d2641ad6768810896); each behaviour this page depends on was re-checked at b10690, the current build at the time of writing, and none of them had changed.

Requirements

Installation

1. Get a llama.cpp build that knows the architecture

The architecture string baked into the GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. Download a release binary at b10630 or newer, or build from source:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

That is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf, 21,864,082,336 bytes (20.363 GiB), imatrix-calibrated and produced on llama.cpp b10630.

Q4_K_M on a 16 GB card is a deliberate choice, and the obvious alternative is narrower than it looks. Ten of bartowski's rungs are under 16 GiB on disk, so "small enough to load" is not the constraint. The constraint is what has to sit beside the weights. Hold this page's own terms — 32,768 tokens of q8_0 KV (0.332 GiB), the recurrent state (0.061 GiB) and a 2.0 GiB reserve, marginally tighter than the 2.454 GiB the lead rung happens to leave — and a fully resident configuration needs weights at or below 13.607 GiB. That is Q2_K_L (12.650) and everything smaller; the whole 3-bit tier misses it, IQ3_XXS at 14.287 GiB leaving 1.320 GiB and Q3_K_S at 14.886 leaving 0.720. So the escape from offload is not "drop to 3-bit", it is "drop to 2-bit" — and that is exactly where the publisher of a second GGUF set draws the line: abenzerps suggests IQ2_M or IQ3_M as a 16 GB starting point and then closes the section with "Prefer Q4_K_M or higher when system memory permits." This recipe follows that advice, because the model is an agent that runs tool calls and multi-step plans, and 2-bit weights are the wrong place to save on a workload where a single malformed argument ends the run. A 3-bit rung with a shallow -ncmoe is a reasonable middle and this page does not measure it either.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 17 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --load-mode none \
    --cache-ram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05), whose text notes that "Apodex follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF by default.

Eight of those flags are load-bearing and none of them is a default.

• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 17 is this recipe's subject; the next three sections derive the number.
• --no-mmproj — llama-server honours the projector auto-download, and both no_mmproj (false) and mmproj_use_gpu (true) default that way in common/common.h, so a bare -hf …-GGUF fetches a sibling projector and puts it on the GPU. The lead repo ships two, at 899,283,488 and 902,822,432 bytes (0.838 and 0.841 GiB). On this card that is just under the 0.844 GiB that the two cheapest steps of -ncmoe would free, spent on a capability the recipe does not use.
• -np 1 — with no slot count, llama-server resolves n_parallel to 4. The Gated-DeltaNet recurrent state is per sequence, so four slots quadruple it. One slot is right for a single-user agent loop.
• -c 131072 — the GGUF declares context_length 262144. Half of it is what this recipe budgets; the ladder below prices the other half.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 is a 2-byte scale plus 32 one-byte quants, i.e. 34 bytes per 32 elements — 8.5 bits per element rather than 16. It takes the 131,072-token cache from 2.500 GiB to 1.328 GiB, which on this card is a saving of 1.172 GiB — worth between two and three steps of -ncmoe you do not have to spend, since a step costs 0.4219 or 0.4863 GiB so two come to 0.844–0.973 and three to 1.266–1.459. It also decides a flag you are not passing: a quantised V cache requires FlashAttention, so with the default -fa auto src/llama-context.cpp logs "enabling flash_attn since it is required for quantized V cache" and turns it on, while an explicit -fa off is a hard error and the context is never created. That is fine here — the same file checks that q8_0's 32-element block divides n_embd_head_k and n_embd_head_v, both 256 on this model — but it means this configuration runs with FlashAttention enabled, and a crash in that area should not be blamed on -ctk.
• --load-mode none — llama.cpp asks for this itself. When a CPU tensor override meets memory mapping, src/llama-model-loader.cpp prints "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance". The mechanism is worth knowing. An overridden tensor is given the first workable buffer type from a list that src/llama-model.cpp builds in the order "ACCEL -> GPU host -> CPU extra -> CPU", and whose GPU-host entry exists precisely because "storing the tensors in a host buffer is useful when the processing of large batches" is offloaded to a GPU — pinned memory the card can DMA from. The loader then downgrades that choice to ordinary CPU memory whenever mmap is on. So mmap silently costs you the pinned buffer this whole path was built around.
• --cache-ram 2048 — see System RAM. The default is 8192 MiB and on this architecture that is not a small number.

What -ncmoe actually moves

-ncmoe N / --n-cpu-moe N is documented as "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp), and the implementation is worth reading once, because on a hybrid recurrent model like this one it is easy to assume that more moves than actually does.

It matches tensor names, not architectures. The flag appends one buffer-type override per block index i in 0 … N-1, each a regex built from LLM_FFN_EXPS_REGEX in common/common.h:

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";

In this GGUF exactly 123 tensors match — ffn_down_exps, ffn_gate_exps and ffn_up_exps, three per block across all 41 blocks. The override is applied at tensor-creation time, in the loader, before any buffer exists: a matching tensor is given a CPU-side buffer type and is then allocated and read into it. Those bytes never enter VRAM at all. Nothing about the architecture is consulted, which is why the flag works identically on a hybrid model and on a plain transformer MoE.

Three things it does not move, all of which stay on the card:

• The Gated-DeltaNet mixers and the attention weights. A GDN block's ssm_* tensors, its attn_qkv and attn_gate, and a full-attention block's attn_q/k/v/output are untouched by the regex. Per block that is 21.1–32.0 MiB of tensors that stay, against 432.0–498.0 MiB of experts that go.
• The shared expert and the router. ffn_down_shexp, ffn_gate_shexp, ffn_up_shexp, ffn_gate_inp and ffn_gate_inp_shexp do not match _exps, so the dense path every token takes stays resident.
• The KV cache and the recurrent state. Both pick their buffer type from the layer's device, which is decided by -ngl alone; -ncmoe rewrites weight tensors and never touches the layer assignment. So a block whose experts are in RAM still keeps its recurrent state in VRAM — which is what makes the budget below decomposable at all.

Add it up and -ncmoe 40 leaves 1.744 GiB on the card: 0.655 GiB of token_embd and output, and 1.089 GiB of everything else across forty blocks.

The VRAM budget

Every figure here is derived from the artifact's own tensor table and from llama.cpp's allocation code at the pinned build. Nothing on this page was measured on an RTX 4080.

Every GiB figure is rounded from its own byte count and the total is rounded once from the byte sum, which is why the GiB column adds to 13.545 while the total reads 13.546. The bytes are the arithmetic; the GiB are a convenience.

Against the card's 16 GiB that leaves 2.454 GiB, and on this recipe that reserve is doing more work than it does on a card the model fits:

• llama.cpp's compute buffers and the CUDA context, as on any recipe.
• The display driver's reservation, which is larger on Windows with a monitor attached than on a headless Linux box, and cudaDeviceProp::totalGlobalMem sitting slightly under the nominal 16,384 MiB.
• A GPU-side staging copy of the expert weights during prompt processing, which only exists on the offloaded path. The next section explains why. The scheduler's copy slots are keyed per tensor, and the largest single expert tensor here is ffn_down_exps at 220,200,960 B (0.205 GiB); one block's three expert tensors together come to 0.486 GiB at the largest, which is the bound to reserve against rather than a figure the allocator is known to hold at once.

None of those three is derivable from a tensor table, so 2.454 GiB is a reservation, not a measurement. Read the real ones off the server's own startup log: llama.cpp prints model buffer size once per buffer, so with -ncmoe you get a CUDA line and a host line and the split above is directly checkable, followed by KV buffer size, RS buffer size and compute buffer size. Check the sum against nvidia-smi --query-gpu=memory.total,memory.used --format=csv before you rely on the configuration. If you do, please send the numbers; this pair has no benchmark in our catalogue.

The offload ladder

The ladder is not linear, and this is the part that does not transfer from any other recipe. bartowski upcasts ffn_down_exps to Q6_K in 20 of the 40 loaded blocks — 0, 1, 2, 3, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, and then 34 through 39 — so one step of -ncmoe costs either 0.4219 GiB or 0.4863 GiB depending on which block it lands on. Both ends of the range are the expensive kind. Any sentence of the form "a step of -ncmoe costs X GB" is wrong on this file; read the cumulative column instead.

Weights only — no KV cache, no recurrent state, no compute buffers. The left column is a system-RAM requirement, and it is the number to size your host against.

Put the KV cache and the recurrent state back on top, hold the 2.454 GiB reserve roughly constant, and the whole recipe collapses to one trade. The KV cache is linear in context, so the price of a doubling is the whole cache you already had: at the top of this range each doubling costs about three steps of -ncmoe, or about 1.330 GiB more in system RAM — 131,072 to 262,144 adds 1.328 GiB of KV, which blocks 17, 18 and 19 exactly cover. Lower down it is cheaper: 32,768 to 65,536 adds only 0.332 GiB, under a single step.

All three rows use -ctk q8_0 -ctv q8_0 and -np 1. All three are arithmetic. The middle row is the one the command above runs, and it is chosen so that this page and the RTX 3090 page document the same context — the entire difference between a 24 GB card and this one is that 7.752 GiB of experts have moved off the card and into RAM.

Going the other way is also legitimate and this page will not pretend otherwise: -ncmoe 14 at 32,768 tokens leaves less in RAM and less to stream, and if your agent loop never fills a six-figure window it is the better setting.

Where the work happens: prompt processing versus generation

The single most useful thing to understand about this configuration is that the offloaded experts are not treated the same way in the two phases, and llama.cpp's scheduler says so in one rule.

An operation whose weights live in a host buffer is normally run on the CPU — ggml/src/ggml-backend.cpp puts it plainly: "operations with weights are preferably run on the same backend as the weights". But immediately after that the scheduler will "check if a backend with higher prio wants to offload the op", and asks the GPU. The CUDA backend answers with a single test in ggml/src/ggml-cuda/ggml-cuda.cu: the op's batch size — for the MoE expert matmul, the number of tokens in the batch — must be at least 32, a threshold whose only override is the GGML_OP_OFFLOAD_MIN_BATCH environment variable.

So the same weights take two completely different paths:

• Prompt processing runs a batch of hundreds of tokens (-ub defaults to 512), so the expert matmuls are offloaded to the GPU and the expert weights are copied host → device over PCIe, layer by layer, for every physical batch. This phase is bus-bound. A community reporter profiling exactly this path with Nsight — on a different card and a different model, at -ncmoe 26 — found that "prefill is bottlenecked by PCIe transfers, not GPU compute", with the GPU idle a large fraction of each pass waiting on those copies (ggml-org/llama.cpp#25859, open, no maintainer verdict on the finding itself).
• Token generation runs one token at a time. One is less than 32, so the offloaded expert matmuls stay on the CPU backend and the expert weights never cross the bus at all. This phase is bound by your system RAM bandwidth and your CPU, and the only thing moving over PCIe is a small activation vector per layer.

Two practical consequences, and they are the reason this page separates the two phases at all:

1. Dual-channel memory and memory clock matter more than the PCIe link for the tokens-per-second you feel while the model is answering. Single-channel RAM roughly halves the bandwidth that the decode path reads those experts at.
2. The PCIe link matters for time-to-first-token on long prompts. Check what yours negotiated with nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.width.max --format=csv — this is a property of your card and your slot, not of the model, and it is the one term in this page's arithmetic that varies between machines that are otherwise identical. If the link turns out to be the constraint, --no-op-offload disables that offload rule globally, moving prompt-processing expert matmuls onto the CPU as well and removing the transfers entirely. It is a real trade, it is untested here, and it is one flag to try.

Throughput at any of these settings is unmeasured on this pair. What the ladder gives you is capacity, not speed, and the two do not have the same shape: an owner of a 16 GB card tuning a different model of the same qwen35moe architecture on stock llama.cpp b10644 found -ncmoe 20 to be an isolated peak rather than a plateau edge — 22 and 21 both sit near 36.7 tok/s, 20 jumps to 42.18, and 19 falls back to 35.33 — and was careful to add "I am not claiming a cause for this boundary; I did not profile PCIe transfers or GPU idle time." (the same thread — an RX 9070 XT under Vulkan on Windows, so nothing there is a number for this card). Treat every rung above as a capacity statement and sweep the two or three either side of it on your own machine.

System RAM, and the prompt cache that will surprise you

At -ncmoe 17 the expert weights hold 7.752 GiB of host RAM for the life of the process. That is the part the ladder tells you about. There is a second consumer that it does not.

llama-server keeps a prompt cache, capped by --cache-ram at a default of 8192 MiB (common/common.h). For an attention-only model a cache entry is a few MiB and the default is invisible. For a hybrid recurrent model it is not: an entry has to carry a full recurrent state, and this model's is 65,863,680 bytes per sequence — the same figure as the recurrent row in the budget table. A community reporter who took a per-request RSS growth on another qwen35moe model for a memory leak was shown the cause in the same thread and retracted it, concluding that "a cache entry for a hybrid model is one full recurrent state, a constant ~126 MiB" (ggml-org/llama.cpp#27894, closed) — his measurement, his model, and enough to show the scale: the default ceiling on that machine would have committed 8 GiB of host RAM to prompt caching alone.

Hence --cache-ram 2048 in the command above: an explicit couple of gigabytes rather than an invisible eight, on a host that is already holding 7.752 GiB of experts. --cache-ram 0 disables the cache entirely and is the right setting if you are memory-constrained and your prompts do not share prefixes. With the cache capped, 16 GB of system RAM is arithmetically sufficient; 32 GB is what this recipe asks for, because the gap between the two is exactly the headroom that stops the OS swapping the expert pages you are reading on every token.

-fit turns itself off, and that is what you want here

llama.cpp can size the offload for you: auto-fit is on by default (fit_params = true in common/common.h, with a per-device margin defaulting to 1024 MiB), so "not passing -fit" is not the same as "not using it".

It does not compete with an explicit -ncmoe. The fitter's contract in common/fit.h is that "only parameters that have the same value as in llama_default_model_params are modified", and common/fit.cpp makes the buffer-override case an outright bail: it throws "model_params::tensor_buft_overrides already set by user, abort", which the caller catches and logs as a failed fit before loading the model with your parameters untouched. So passing -ncmoe gives you the explicit offload and disables the fitter, at the cost of one warning line in the log. Passing -fit off as well is harmless and removes the warning.

Prefer the explicit number anyway, for a second reason: there is an open regression report against --fit-target on a Q4_K_M of another 35B-A3B model — llama.cpp issue #27171, filed 2026-08-16 by a community reporter, still open and labelled bug-unconfirmed, with a contributor's fix in flight at PR #27207 and still unmerged as of this writing. Different card, different model — it is a reason to pin the offload by hand until the fix lands, not a claim about this pair.

Results

• Speed: omitted, and not because nobody has measured the model. Two quantisers publish throughput for Apodex 1.1 mini and neither number can travel to this page. ghazni101/Apodex-1.1-mini-MQ4RP reports 226 tok/s on an RX 7900 XTX — but from a .mq4r file under hipfire, a separate Rust engine, so it is neither this file nor this runtime. kingjones777 publishes a four-rung ladder: 63.30 tok/s for the build that repo holds, 45.05 and 32.54 for the two 8-bit tiers, and 64.87 at the top — for a fourth build whose own card says it has not been measured. That card's Speed — full offload section is a one-row table holding a dash, with a note that the run is still pending on an idle box, some thirty lines above the shared variants table that prints 64.87 for the same build, and that table's own caption reads "A dash means I haven't measured that one yet — I won't put a number in a card I didn't measure." So read the top of that range as unsettled rather than as a figure. It changes nothing here, because every rung of the ladder is measured with full GPU offload (-ngl 999) on a Ryzen AI MAX+ 395 unified-memory APU under the ROCmFPX fork — a machine with no expert offload and no PCIe hop, which is the entire subject of this page. Nor is there one anywhere else I looked. The space: the model cards of all 21 HuggingFace repositories matching Apodex-1.1-mini (re-enumerated 2026-08-30 — the surface both figures above came from, and the only surface carrying any), every issue in the vendor's own FrontierAgent tracker, enumerated rather than searched — no title or body in it mentions any hardware term at all — and the llama.cpp tracker, whose search returns nothing for "apodex" while the identical query returns hundreds of hits for the architecture string, so the zero is about the term and not about the search. There is no throughput figure for this file, on this runtime, at any -ncmoe setting, on any card. If you run it, please contribute the numbers.
• VRAM usage: 13.546 GiB derived working set at -ncmoe 17 and 131,072 tokens with q8_0 KV, leaving 2.454 GiB of the card, with 7.752 GiB of expert weights in system RAM. Dropping to -ncmoe 14 at 32,768 tokens gives 13.880 GiB and 6.422 GiB respectively. See /check/apodex-1-1-mini/rtx-4080 for live data as it lands.
• Quality notes: the vendor publishes agentic evaluation scores on the model card, but all of them are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. The lead quant is imatrix-calibrated — the publisher ships both the calibration corpus and the imatrix alongside the ladder — which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier. Note that the routed experts are already narrow (expert_feed_forward_length 512), which is part of why the sub-4-bit rungs give up more than the file sizes suggest.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-4080.

Troubleshooting

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE, which arrived in (b7989, b7990] — so this error means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

It loads fine, then runs out of memory when a long prompt lands

Expected if -ncmoe is too low for the context you asked for. Loading allocates the weights and the full KV cache, but the compute buffers grow with the batch, and prompt processing additionally needs room for the expert weights it is copying to the card. Raise -ncmoe by two or three — on this file that frees between 0.844 and 1.459 GiB depending on which blocks the steps land on — or drop -c. If the card is also driving a display, start two higher than the table above and check nvidia-smi after load.

Prompt processing is slow and the GPU looks idle

This is the expected shape of the offloaded path rather than a fault, and there are four things to try before suspecting the model. Confirm mmap is off (--load-mode none); llama.cpp prints a warning about it, and the RX 9070 XT owner cited above reported a gain on both phases from following it, larger on prompt processing (the same comment — a different card, a different model and a different backend, so read it as a direction to test rather than a number to expect). Check what your PCIe link negotiated with nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.width.max --format=csv, since prompt processing is the phase that spends it. Raising -b and -ub is the largest lever that same reporter found: 8192/2048 gave him ~1524 t/s of prompt processing against ~705 at 1024/512. Read that gap carefully before expecting it — his baseline is below the 2048 default on -b and exactly at the 512 default on -ub, and he published no row at the actual default pair, so the ratio is not a number a reader starting from defaults should expect — a reader on the defaults already starts with more logical batch than his 705 t/s row had. His nearest at-or-above-default row (2048/1024, ~1008 t/s) makes it about 1.5x. Note also that his 16384 row was slightly worse than 8192 and that an oversized -ub was actively harmful, so this is a sweep and not a direction. It costs compute buffer, which the 2.454 GiB reserve above has not been sized for, so raise them and re-read nvidia-smi rather than assuming. And --no-op-offload keeps the expert matmuls on the CPU in both phases, which removes the transfers entirely and is worth an A/B if your link is narrow.

System thrashing, or resident memory climbing request after request

Two different causes with the same symptom. If it climbs and plateaus, that is the prompt cache filling to --cache-ram, which defaults to 8192 MiB and costs one full recurrent state per entry on this architecture — cap it, as the command above does. If it does not plateau, your system RAM is simply short: at -ncmoe 17 more than a third of the file lives in RAM permanently, on top of the OS.

The server downloads an extra file you did not ask for

If you launch with -hf bartowski/apodex_Apodex-1.1-mini-GGUF instead of a local -m path, llama.cpp resolves a projector sibling as well — find_best_mmproj matches any sibling filename containing mmproj, and this repo ships two. Pass --no-mmproj, or use an explicit -m path as above.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and mainline llama.cpp implements it behind --spec-type draft-mtp (the exact spelling matters; mtp alone is rejected). On this card, under offload, the honest answer is that nobody has measured it and the costs are all on the wrong side of the ledger.

The costs are concrete: block 40's tensors stop being skipped (+0.444 GiB of weights, which -ncmoe 17 does not touch — the flag only covers blocks 0 to 16); the recurrent state is multiplied by the draft depth; and a second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk. Against a 2.454 GiB reserve the weights alone are one full step of -ncmoe, spent before any of it has been shown to help.

The benefit is unknown and the nearest evidence is discouraging. unsloth's published guidance for this class of model is that "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs) — so 1.15–1.25× at best, before offload is taken into account. And under offload even that goes. An RTX 3060 owner running --n-cpu-moe 25 on a different model of this architecture, with the MTP flags on, reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" (discussion #18 on that model's GGUF repo — a community thread, no org member in it); a second commenter there offers the explanation that CPU-MoE offload neuters the MTP gain, which nobody in the thread measured either way. That explanation is at least consistent with the mechanism above: if the pace is set by the CPU reading expert weights out of system RAM, drafting more tokens per step does not help, because the bottleneck is not the GPU waiting on itself. Try it if you like; do not budget VRAM for a speed-up you have not seen on your own machine.

Choosing a different quant publisher

Conversions of this model are still appearing: a Hub enumeration on 2026-08-30 returned 21 repositories matching Apodex-1.1-mini across ten publishers, several created within the preceding week. Re-enumerate before you decide, and check the byte count and the file list rather than the tier name — across publishers a file named Q4_K_M spans more than a gigabyte for the same nominal tier. Within bartowski's own ladder the rungs adjacent to the lead are Q4_K_S at 19.619 GiB and IQ4_XS at 17.955 GiB; each is worth roughly two and six steps of -ncmoe respectively, which is a legitimate way to buy back host RAM rather than VRAM.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, but they are vLLM/SGLang safetensors rather than GGUF and llama.cpp cannot load them at all. They are also much too large for this card before anything else is counted — 22.548 GiB and 22.958 GiB of weights for the first two, against 16 GiB — and the -NVFP4 repo is not a 4-bit checkpoint anyway: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4. Their own Quick Starts launch on a single GPU — --tp-size 1 on both -NVFP4 and -GPTQ-Int4, --tp 1 on -FP8; the --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone. For one RTX 4080 the GGUF route above is the path.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex, /apodex-1.1-mini and /apodex1.1 all return 404, and the corresponding registry.ollama.ai manifest returns 404 while the same request shape succeeds for a model known to be in the library. Use llama.cpp directly. If a tag appears later, please tell us.