Apodex 1.1 mini on RTX 3080 Ti: a 35B-A3B agent in 12 GB, where the card is not the bottleneck

What You'll Build

A local, OpenAI-compatible agent endpoint serving Apodex 1.1 mini — the Apache-2.0 agent model Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on one 12GB RTX 3080 Ti, at a 65,536-token context. The Q4_K_M weights are 19.908 GiB. The card holds 12 GiB. It runs anyway, because 91.2% of those weights are routed experts that llama.cpp can leave in system RAM.

Hardware data: RTX 3080 Ti (12GB VRAM) · 8.891 GiB derived working set at -ncmoe 26, 65,536-token context · See benchmark data

⚠️ Nothing on this page was measured on an RTX 3080 Ti. /check/apodex-1-1-mini/rtx-3080-ti holds zero benchmarks. Every VRAM figure below is arithmetic over the GGUF's own tensor table and llama.cpp's allocation code, and it is shown so you can check it. There is no throughput number on this page at all — see Results for what exists and why none of it transfers. If you run this, please send the numbers.

🧭 The headline you should take away before the arithmetic. This is a fast card, and this recipe deliberately moves most of the work off it. At -ncmoe 26 a decode step reads at most 1.945 GiB from VRAM and exactly 0.367 GiB from system RAM — and the two halves take equal time at the point where your card's bandwidth is 5.3× your DIMMs'. Past that ratio the host half is the bigger one, and a 384-bit GDDR6X board against dual-channel system memory is well past it. So the smaller number takes the larger share of the time. Your generation rate is set by your DIMMs, not by this GPU. The section Where this card's speed actually goes prices that out. Everything about capacity on this page is the same answer any other 12 GB card gets; everything about speed is not.

ℹ️ This recipe is text-only. The checkpoint has a vision tower and the GGUF repo ships a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card contains no image-input instructions — the words vision, visual, multimodal and mmproj appear zero times in it (counted 2026-08-30). On a card this tight the projector is also 0.838 GiB you cannot spare, so the run command below refuses it explicitly rather than merely omitting the flag.

Requirements

Two version notes, because both have bitten this catalogue. The tag moves several times a day — b10701 was published 2026-08-30T18:56:31Z, three tags inside one hour — so this page pins b10666 and does not claim to be current; check the release list rather than trusting a date. And GET /repos/ggml-org/llama.cpp/releases/latest answers v0.3.0, not a bNNNN tag, because every build tag is marked prerelease; use /releases?per_page=N if you are scripting a version check.

Disk size and VRAM residency are different numbers here and they are not interchangeable. The file is 20.363 GiB on disk; a normal run loads 19.908 GiB of it, because blk.40 — the multi-token-prediction head — is created with TENSOR_SKIP unless you ask for it (src/models/qwen35moe.cpp). Budget 22 GB of disk and 19.908 GiB of weights.

What the vendor publishes about this card, and what it does not. NVIDIA's spec table gives the RTX 3080 Ti as 10,240 CUDA cores, 1.67 GHz boost, 12 GB GDDR6X on a 384-bit interface, PCI Express Gen 4 (nvidia.com). It publishes no memory-speed figure, no memory-bandwidth figure and no PCIe lane count, so this page states none of the three. Where a bandwidth number would be needed, the arithmetic below is written so that you supply your own:

# what your board actually negotiated, and what its memory clock is
nvidia-smi --query-gpu=name,memory.total,pcie.link.gen.max,pcie.link.width.max,clocks.max.memory --format=csv

Multiply the reported memory clock by 2 (GDDR is double-data-rate) and by 48 bytes (384 bits ÷ 8) to get bytes per second. Do the same for your DIMMs from their rated transfer rate: dual-channel DDR4-3200 is 3200 MT/s × 8 B × 2 channels = 51.2 GB/s, dual-channel DDR5-6000 is 96 GB/s. Both of those numbers matter on this page and the second one matters more.

Installation

1. Build llama.cpp with CUDA

The architecture string in this GGUF is qwen35moe, registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE. Use b10630 or newer — the release the lead quant was produced with. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that makes blk.40 optional landed in the bracket (b10211, b10212], so b10630 is comfortably past it.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

GA102 is compute capability 8.6; the default CUDA build covers it and no architecture flag is needed.

2. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf is 21,864,082,336 bytes, imatrix-calibrated, produced on llama.cpp b10630. Repo lastModified 2026-08-26T09:14:35Z; the file list and size above were re-fetched on 2026-08-30 rather than copied from a sibling page.

Do not start from the publisher's own quickstart on this card. That line is llama-server -hf bartowski/apodex_Apodex-1.1-mini-GGUF:Q4_K_M, and the card states plainly that "llama.cpp downloads the mmproj automatically when using" -hf. llama-server is in llama.cpp's mmproj_examples list (common/arg.cpp), mmproj_use_gpu defaults to true and no_mmproj defaults to false (common/common.h:594,596), so that command puts 0.838 GiB of vision projector on a card this recipe has already budgeted to within about a gigabyte. Download the file explicitly, as above, and load it with -m.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 26 \
    -np 1 \
    -c 65536 \
    -ctk q8_0 -ctv q8_0 \
    -fa on \
    --load-mode none \
    -cram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Eight of those flags are decisions rather than defaults. Each was re-derived at b10666 for this page rather than carried across from another card, and each one's default is quoted from the source at that pin:

• --no-mmproj refuses the vision projector rather than merely not asking for it — 0.838 GiB, for the reason given under Installation. Capacity-driven.
• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 26 keeps the routed experts of blocks 0–25 in system RAM. The flag's own help text at this pin is "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp:2790), and the implementation matches the wording exactly — see How the offload actually works. 26 is derived in Choosing your own -ncmoe; it is not a measured optimum. Capacity-driven — and identical to what any 12 GB card gets, because the constraint is the 12 GiB, not the silicon.
• -np 1. llama-server sets params.n_parallel = -1; // auto by default (common/arg.cpp:1400) and resolves that to four slots sharing one KV pool. The Gated-DeltaNet recurrent state is allocated per sequence, so four slots quadruple it — 0.245 GiB instead of 0.061 — and split your context four ways. One slot is what a single-user agent loop wants. Capacity-driven.
• -ctk q8_0 -ctv q8_0 halves the KV cache. block_q8_0 is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element, not 8. It takes the 65,536-token cache from 1.250 GiB to 0.664 GiB. ⚠️ That 1.250 is the all-f16 baseline with Flash Attention on, and it is not the baseline if you turn Flash Attention off — the two halves are not governed by the same rule. Quantised V requires FA (llama-context.cpp:3675, and a second hard throw at :464 — "quantized V cache was requested, but this requires Flash Attention"), while the matching check on K is gated flash_attn_type != DISABLED at :3686 and therefore never runs when FA is off. So -ctk q8_0 -ctv f16 is legal without FA: 15,680 B/token, 0.957 GiB at 65,536 — cheaper than the 1.250 an all-f16 reading would predict. Capacity-driven.
• -fa on is not cosmetic on this card, and it is not optional. The default at this pin is auto (flash_attn_type = LLAMA_FLASH_ATTN_TYPE_AUTO, common/common.h:499), and with a quantised V cache auto does not get a choice: src/llama-context.cpp:3675-3684 enables Flash Attention itself, logging "enabling flash_attn since it is required for quantized V cache", and hard-errors with "quantized V cache requires flash_attn to be enabled" if you disabled it explicitly. So the flag documents what already happens rather than changing it. The reason it earns a line anyway is the reverse case: turning Flash Attention off is not an available lever at this context on a 12 GB card, and the price is not the one people expect. See Something told you to turn Flash Attention off under Troubleshooting before you try it. Capacity-driven, and it is the tightest constraint on the page.
• --load-mode none. When you override tensors to CPU while memory-mapping the file, the loader itself warns: "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance" (src/llama-model-loader.cpp). Taking that advice means the 11.742 GiB of expert weights are ordinary allocations rather than file-backed pages, which is also why the RAM line in Requirements is a hard figure rather than a cache you can squeeze.
• -cram 2048 caps llama.cpp's host-memory prompt cache at 2 GiB. The default is 8192 MiB — cache_ram_mib = 8192 at common/common.h:632 — and it is more expensive on this architecture than on a plain transformer. See the host-RAM entry under Troubleshooting.

Sampling values are the vendor's own from the model card (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05). The card notes that Apodex "follows the Qwen3.5 chat template"; llama-server uses the template embedded in the GGUF and enables Jinja by default, so tool calls and <think> blocks parse the way the vendor intends. Pass tool schemas through the API's tools= field rather than inlining them in the system prompt — that is the vendor's explicit instruction.

How the offload actually works

-ncmoe N is not an architecture feature and does not know what a Mixture of Experts is. It builds one regex per block and hands them to the model loader as buffer-type overrides (common/common.h:1130):

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";
// llm_add_n_cpu_ffn_overrides(N, …) pushes "blk\.0" + regex … "blk\.(N-1)" + regex

The loader then does a plain std::regex_search on each tensor name and, on a match, allocates that tensor in a CPU buffer instead of the GPU one (src/llama-model-loader.cpp). Nothing in that path consults the architecture, so it applies to this hybrid recurrent model exactly as it does to a plain MoE. 123 tensors in this file match — three per block across all 41 blocks — totalling 18.586 GiB, of which 18.164 GiB sit in the forty blocks a normal run loads. That is 91.2% of the 19.908 GiB resident weight.

What does not move is as important. The offload takes each block's ffn_down_exps, ffn_gate_exps and ffn_up_exps, and leaves on the GPU: the attention path, the recurrent state tensors (ssm_conv1d, ssm_a, ssm_alpha, ssm_beta, ssm_norm, ssm_out), the router (ffn_gate_inp), the shared expert (ffn_*_shexp) and every norm. So routing is decided on the GPU, the Gated-DeltaNet state stays on the GPU, and only the selected experts' matmuls happen elsewhere. Offload everything the regex can reach — -ncmoe 40 — and 1.744 GiB is still resident.

That last word — elsewhere — means two different places depending on what you are doing, and on this card the distinction is the whole story. An op whose weights live in a host buffer runs on the CPU — "operations with weights are preferably run on the same backend as the weights", per the scheduler's own comment (ggml/src/ggml-backend.cpp) — unless a higher-priority backend wants to offload it. CUDA does, and the test is one comparison:

// ggml/src/ggml-cuda/ggml-cuda.cu:5341-5344 at b10666
static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) {
    ...
    return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size;
}
// :5515 — const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv(...)) : 32;

Generating a token is a batch of one, so decode reads the offloaded experts from system RAM and multiplies them on your CPU — they never cross the bus. Prompt processing is a batch of hundreds, so prefill copies those tensors across PCIe into a duplicate the graph allocator sizes for the run. A narrow slot is therefore a time-to-first-token problem, not a tokens-per-second problem, and this page treats the two separately throughout.

The VRAM budget

Weights, KV cache and recurrent state, at the recipe's settings.

The three rows come from:

• Weights. The file's tensor payloads sum to 21,853,090,304 bytes, of which 476,956,672 are blk.40, leaving 21,376,133,632 (19.908 GiB) resident. -ncmoe 26 moves 12,608,077,824 of that (11.742 GiB) to the host.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so forty blocks are loaded, and full_attention_interval 4 makes ten of them full-attention: blocks 3, 7, 11, 15, 19, 23, 27, 31, 35 and 39. Each carries head_count_kv 2 at key_length 256 and value_length 256 — 1,024 cache elements per token per layer, so 10,240 across ten layers. At q8_0 that is 10,880 bytes per token; at f16 it is 20,480. The other thirty blocks are Gated-DeltaNet and cache nothing that grows.
• Recurrent state. Those thirty blocks each allocate two F32 tensors, sized (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements (src/llama-hparams.cpp). That is 2,195,456 bytes per layer per sequence and 65,863,680 across all thirty. llama-server prints it at startup as RS buffer size.

The two terms that table cannot produce, and what this page reserves for them

The 8.891 GiB is not a peak. Two things sit outside it, and on a 12 GB card they are the difference between a configuration that loads and one that does not. Rather than leave them as a warning, this recipe budgets for them explicitly:

1. The card's usable pool is not 12 GiB. The CUDA context, the driver and anything driving a display take a share the model budget never sees. Our catalogue holds one measured expert-offload run in this VRAM tier with the numbers written down — gemma4-26b on an RTX 3060 at -ncmoe 12 — whose submitter recorded "That peak is 11,179 of the card's 12,288 MiB — roughly 1.1 GB spare, which is why the lower rungs fail rather than merely slow down", on a headless box. That is a different GPU in the same tier, so read it as the order of the term and not as your board's exact reserve. This page sizes against ~11.0 GiB of usable pool, not 12.0.

2. The compute buffers. llama.cpp allocates working memory for the graph on top of everything above, and under expert offload it also stages the host-resident expert tensors on the GPU during prefill — each of this file's expert triples is 0.4219 or 0.4863 GiB, so that staging term alone is of that order. This page reserves 1024 MiB. That number is llama.cpp's own default margin per device for auto-fit — fit_params_target is initialised to 1024 * 1024*1024 at common/common.h:481 — so it is the runtime's own opinion of how much room it wants, not a figure this page invented.

For scale on what happens if you leave nothing: llama.cpp issue #27698 records a run where the weights loaded, expert offload was honoured, and then context creation failed on a ~504 MiB compute buffer. ⚠️ That report is not about this card and does not transfer — it is an RX 6700 XT, gfx1031, RDNA2, on the ROCm backend, open with zero comments since 2026-08-25. It is quoted for one thing only: the magnitude of a term every table in this family of recipes excludes, and the shape of the failure it produces — a clean load followed by an OOM the moment context is allocated.

So the rule this page sizes to is:

weights + KV + recurrent state ≤ 11.0 GiB − 1.0 GiB reserve = 10.0 GiB.

At -ncmoe 26 and 65,536 tokens the sub-total is 8.891 GiB, which clears that by 1.109 GiB. That slack is the answer to "why 26 and not 24", and it is the reason this page does not simply take the fastest row that fits inside 12.

Choosing your own -ncmoe

Every row is weights + KV + recurrent state on a 12 GiB card, q8_0 KV, one slot. The last column applies the rule above — headroom against the 10.0 GiB working ceiling, after the 1024 MiB compute reserve:

Read the last two columns together. The Free of 12 GiB column is what the naive arithmetic says; the Slack column is what is actually left once the driver's share and the compute reserve are taken out, and the two top rows are the point — -ncmoe 20 at 32,768 tokens looks like it has 0.781 GiB to spare and in practice does not load. The bottom row is the model's entire declared window (context_length 262144) on a 12GB card, and it clears the ceiling; it costs four more steps of offload than the recipe's setting, which the next section prices in milliseconds.

If your board is headless and you want the throughput, -ncmoe 24 at 65,536 is a row in this table rather than a different recipe — but note it clears the ceiling by 0.201 GiB, so it is the row most likely to be wrong if your driver's reserve is larger than the tier observation above.

Then stop guessing and measure it. llama.cpp ships a tool that answers this question on your hardware without running the model:

# what the fitter would choose on your box, as CLI arguments:
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536

# estimated MiB per device: model, context, compute
./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf -c 65536 -fitp on

And when llama-server exits it prints llama_memory_breakdown_print, a per-device line splitting total / free / model / context / compute / unaccounted. That compute column is the number this page reserved 1024 MiB for, and unaccounted is your driver's share — the two terms the table above cannot produce. If you run this, those two figures are the most useful thing you could send us, more useful than a speed number.

Where this card's speed actually goes

This is the section that is not the same on an RTX 3060, and it is the reason the capacity numbers above are identical while the performance advice is not.

Decode. At batch 1 the CUDA offload test fails, the offloaded expert matmuls stay on the CPU, and their weights never cross the bus. So a decode step splits across two buses, and the split is computable. expert_used_count is 8 of expert_count 256, so exactly one thirty-second of each block's expert weight is read per token:

The VRAM row is an upper bound, because the token-embedding table is read one row at a time while the output head is read in full, and this page has not separated the two in this file. That the bound points upward is convenient: the real VRAM figure is lower, which makes the conclusion below stronger rather than weaker.

Now put the two on their buses. The host side is at least 15.9% of the bytes — 394,002,432 of 2,482,104,832 — and note the direction: because the VRAM row is an upper bound, the share it yields is a floor, so the real host share can only be higher than that. What this page will not tell you is how much more than that share of the time it is, because that needs a bandwidth figure this page is not entitled to state. So here is the threshold instead, which needs none. The two halves of the decode roofline take equal time exactly when your card's bandwidth is 1.945 ÷ 0.367 = 5.3× your system RAM's, and because the VRAM figure is an upper bound the true crossover is lower still. Compute both sides with the two arithmetic recipes under Requirements and compare: a 384-bit GDDR6X board against dual-channel DDR4 or DDR5 is not close to that threshold, and neither is any other discrete GPU in this catalogue. Past it, the host term dominates — and it dominates more the faster the card. On this recipe, a faster card cannot fix decode, because decode was never on the card.

That gives a ceiling that does not depend on which GPU you own at all — divide the 0.367 GiB of host traffic by your memory bandwidth:

⚠️ Those are ceilings from one term, not predictions. They exclude the CPU time to actually multiply those experts, kernel launch latency, the VRAM side, and everything else. The one measured 12 GB-tier expert-offload run this catalogue holds with a stated memory bandwidth — gemma4-26b on an RTX 3060, -ncmoe 12, DDR4-2133 dual-channel — achieved 37.2 tok/s, comfortably below what a host-bandwidth roofline would allow for it. Read the table as "you cannot beat this", never as "you will get this". A single-channel configuration halves every row.

The same record is also the cleanest available demonstration that the mechanism is real, on this exact VRAM tier: the submitter's ladder is -ncmoe 12 = 37.23 tok/s against -ncmoe 30 = 21.20 tok/s. Eighteen more blocks on the host path, on one unchanged card, cost 43% of the generation rate. That is a different model, so the numbers are not yours; the direction and the size of the effect are the point.

Which makes the offload depth the lever, priced per step. Moving one block from card to host frees its whole expert triple from VRAM — 0.4219 or 0.4863 GiB — and adds one thirty-second of it to every token's host read: 13.5 MiB for a cheap block, 15.6 MiB for an expensive one, or about 0.28–0.32 ms per token each on dual-channel DDR4-3200. Going from the recipe's -ncmoe 26 to 30 for the full 262,144-token window adds four blocks and roughly 1.2 ms per token. Going the other way, to 24, buys back about 0.6 ms — and costs 0.908 GiB of the headroom the previous section spent so carefully.

Prefill is the opposite case, and it is where the bus binds. llama.cpp issue #25859 is an open profiling report against exactly this shape of run — "My box: RTX 3060 12GB, Ryzen 5600X, 32GB DDR4, PCIe 4.0, Ubuntu 24.04." running "Qwen3.6-35B-A3B (Q4_K_M), -ncmoe 26, prefill at pp2048 / ub2048" — and it finds that "prefill is bottlenecked by PCIe transfers, not GPU compute", with GPU idle around 42% of each prefill pass waiting on serial expert-weight H2D copies that do not overlap the matmuls consuming them.

Take that seriously and it bounds what this card's extra compute can buy. NVIDIA's own spec tables make the RTX 3080 Ti 10,240 CUDA cores at 1.67 GHz against the RTX 3060's 3,584 at 1.78 GHz (30-series 3080 table, 30-series 3060 table) — a large gap in arithmetic throughput — but if roughly two fifths of prefill is the GPU waiting on copies that a faster GPU does not accelerate, then the compute advantage applies to the other three fifths only. The report's own caveats limit this further and all of them matter: it is a different model, a single box, a custom fork used for A/B, and the author says the measurement dates to "around the b6xxx era" and may be stale against current master. Treat it as the shape of the answer, not its value — the shape being that a fast card in this recipe is fast at the part that was already not the bottleneck.

Two things you can act on, both of which point away from the GPU: put your DIMMs in dual channel and buy memory bandwidth before you buy silicon; and if time-to-first-token specifically is bad while generation is fine, look at the slot, because prefill is the only phase where the link is on the critical path.

If you leave -ncmoe off entirely

llama.cpp will pick an offload for you. Auto-fit is on by default — fit_params = true at common/common.h:476 — and moves MoE tensors to system memory until it can leave its 1024 MiB margin. That is a reasonable path, with two caveats. It records its choice nowhere except the log, so you cannot reproduce a run from the command line alone; and there is an open regression report against the margin flag on this same architecture family — llama.cpp issue #27171, filed 2026-08-16 against Qwen3.6-35B-A3B Q4_K_M with --fit-target, still open and labelled bug-unconfirmed when this page was written (2026-08-30), with a proposed fix (PR #27207) still open and unmerged.

You do not need -fit off alongside an explicit -ncmoe. The fitter refuses to run when you have set your own overrides — common/fit.cpp throws model_params::tensor_buft_overrides already set by user, abort, which is caught and logged as a warning while the run continues with your settings. Seeing that line in the log means your offload is being honoured, not that something broke.

Why the steps are uneven

A step of -ncmoe does not cost a fixed amount of VRAM in this file, and any advice of the form "one step buys you X GB" is wrong here. bartowski's imatrix recipe upcasts ffn_down_exps to Q6_K in exactly twenty of the forty loaded blocks, so a block's three expert tensors are either 452,984,832 bytes (0.4219 GiB, all Q4_K) or 522,190,848 (0.4863 GiB, Q6_K down-projection). The expensive twenty are blocks 0–4, then 7, 10, 13, 16, 19, 22, 25, 28 and 31, then 34–39 — the top and bottom of the stack are all expensive and the middle alternates, so the first few steps of -ncmoe and the last few buy more room than the ones in between. Read the cumulative column in the table above rather than multiplying.

This is a property of the file, not of the model. IQ4_XS from the same publisher is 17.955 GiB on disk and its ladder is linear at 0.3984 GiB per step. If you would rather spend quality than throughput, that is the trade — but derive its table from its own tensor list, do not scale this one.

Does the arithmetic survive contact with a real card?

No one has run Apodex on an RTX 3080 Ti, and /check/apodex-1-1-mini/rtx-3080-ti returns unknown with zero benchmarks. The nearest thing to a test of the method is a different model on a 12 GB card, and there is one in our catalogue with a figure at four offload depths: Qwen3.6-35B-A3B at UD-Q4_K_M on an RTX 3060. That model is the same architecture shape as this one — 41 blocks, ten full-attention, identical ssm parameters and expert dimensions — so a weight ladder computed from its own tensor table is directly comparable:

⚠️ Read that right-hand column as GiB of weights, and do not read it as a peak. It tracks the derived weight ladder to within 0.16 GiB across a 9 GiB range. Read as decimal GB, every point falls below the weights that setting must load, which is impossible. And a figure that tracks the weights cannot also contain the KV cache and the compute buffers: at the top rung, 11.7 means the interval [11.65, 11.75] GiB against 11.542 GiB of derived weights, leaving at most 0.208 GiB for everything else — while that run's f16 KV cache alone was 0.168 GiB at its 8,832-token bench depth, plus 0.061 of recurrent state. (That 8,832 is read off the benchmark record's own command line — llama-bench … -p 512 -n 128 -d 0,4096,8192, benchmark id 278 on that /check page — not off the recipe, so the claim below stands as written.) The excluded terms do not fit in what remains, so those numbers are floors, not peaks. Our own RTX 3060 recipe for that model builds its context table on the 9.8 figure read as a measured peak; that reading does not survive this arithmetic, and no number from it appears anywhere on this page.

What the comparison does establish is the shape of the ladder, at four settings spanning 9 GiB. It is not four independent measurements — one submitter, one rig, one model, one tool — so every systematic error the method could have is shared across all four points.

Two other pages on this exact card worth reading first

gpt-oss 20B on RTX 3080 Ti is the same card and the same technique on a much smaller model, and it is the cheaper way to find out whether your box's CPU and RAM are up to an offloaded MoE at all — build it, run that, then come back. If instead you have 24 GB and landed here by accident, Apodex 1.1 mini on RTX 3090 fits the whole model on the card with no offload and none of this page's arithmetic applies.

Results

• Speed: omitted, and the reason is a predicate rather than a count. No throughput figure for this file, under llama.cpp, on any NVIDIA card, at any offload depth exists in the space I searched on 2026-08-30, and the space was: the model cards and file listings of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces (re-enumerated today via GET /api/models?search=Apodex-1.1-mini&limit=100; the count was 21/10 two days ago, so re-run it rather than trusting this sentence); the llama.cpp issue tracker, where repo:ggml-org/llama.cpp apodex returns 0 against 271 for qwen35moe and 726 for n-cpu-moe on the same query shape; and our own /check. Figures for the model do exist and every one of them is a different artifact on a different engine:

  • kingjones777 publishes a table covering four of his own repacks — 64.87 / 63.30 / 45.05 / 32.54 tok/s — in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp opens, on a Ryzen AI MAX+ 395 (gfx1151), which is not a catalogue card.
  • ghazni101 reports "generation measured at 226 tok/s end-to-end for MQ4R". ⚠️ Three disqualifications, and every one of them is about the run and not about the file. It is a .mq4r container under hipfire, a Rust engine that is not llama.cpp; .mq4r is not a GGUF, so nothing this page installs is what was timed; and it was measured on an RX 7900 XTX, i.e. gfx1100 — name the target, because one gfx id away is a different claim. What is not a disqualification, though the repo's name invites the mistake: the 226 is for MQ4R and the repo is called -MQ4RP, but it ships both builds — apodex-1.1-mini-mq4r.mq4r (18,700,048,128 B) and apodex-1.1-mini-mq4rp.mq4r (18,778,601,728 B), per the tree API — under a card headed "MQ4R & MQ4RP quants". So the figure does measure a file that repo hosts. The card's "expect a similar ~10% RP decode penalty here" is a prediction about the other one.

  One figure on this pair would be worth more than all of the above. If you run this, contribute it.

• VRAM usage: 8.891 GiB derived sub-total at -ncmoe 26 and a 65,536-token context, plus the compute buffers and driver share this page reserves 1024 MiB and ~1 GiB for respectively but cannot derive. See /check/apodex-1-1-mini/rtx-3080-ti for live data as it lands.

• What to expect anyway, as a mechanism rather than a number: with 26 of 40 blocks' experts on the host, decode is paced by your CPU and system-memory bandwidth and not by this card — see the ceiling table above, and treat it as a ceiling. Prefill is where the RTX 3080 Ti's compute has something to do, and even there a profiled run of this shape spent about two fifths of the time waiting on the bus.

• Quality notes: Q4_K_M on a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated, which is the reason to prefer it over an uncalibrated conversion at the same nominal tier. The vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness, so they are a vendor claim and are not reproduced here.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-3080-ti.

Troubleshooting

First, check that your card is the desktop 12 GB board

NVIDIA has shipped two different products under the RTX 3080 Ti name. This page budgets for the desktop board on NVIDIA's 30-series spec table: 12 GB GDDR6X, 384-bit. There is also a laptop GPU sold under the same name and it is a different part with a different memory configuration — llama.cpp issue #27187 is filed by an owner of one running --n-cpu-moe 34, and its hardware block reads "NVIDIA GeForce RTX™ 3080 Ti Laptop GPU". Every row of the ladder above is keyed to a capacity, so use the capacity you actually have:

nvidia-smi --query-gpu=name,memory.total --format=csv

If that reports something other than roughly 12288 MiB, the ladder's Weights on GPU column still applies but every Free and Slack figure does not — recompute them against your own number.

Out of memory once a long prompt lands, but not at load

Two different causes, and they need opposite fixes.

The KV cache grows with the prompt at 10,880 bytes per token, so a configuration that loads cleanly can still die deep into a context. Check your -c against the table above and either raise -ncmoe or lower -c.

The other cause is specific to expert offload. During prompt processing — any batch of 32 or more tokens on CUDA — the scheduler moves the host-resident expert matmuls onto the GPU, which means copying those tensors into a duplicate the graph allocator sizes for the run. Each of this file's expert triples is 0.4219–0.4863 GiB, so prefill peaks materially above the generation footprint the table describes. If you OOM while a long prompt is being processed but not while generating, try --no-op-offload, which keeps those operations on the CPU where their weights already are; expect slower prefill in exchange, and note that this is the one place a wider slot was helping you.

System thrashing, swapping, or an OOM kill with VRAM to spare

Your system RAM is the limit, and there are two terms in it, not one.

The first is the offload itself: at -ncmoe 26 this recipe puts 11.742 GiB of expert weights on the host, and with --load-mode none those are real allocations rather than file-backed pages.

The second is the one that catches people, because nothing in the command asks for it. llama-server keeps a host-memory prompt cache and it is on by default at 8192 MiB — cache_ram_mib = 8192 in common/common.h:632, exposed as -cram / --cache-ram with -1 for no limit and 0 to disable (added by PR #16391, "server : host-memory prompt caching", whose body says the cache "is stored in regular RAM"). So a default llama-server may hold up to 8 GiB of RAM beyond the offload — with this recipe's 11.742 GiB that is close to 20 GiB of host memory before the operating system gets any, which is why the command above pins it lower.

And an entry here has a floor. Saving a prompt calls llama_state_seq_get_data_ext on the whole sequence state (tools/server/server-context.cpp), and on a hybrid model llama_memory_hybrid::state_write writes both children — the attention cells and the recurrent memory, the latter unconditionally. So every cached prompt carries its tokens' KV at your -ctk/-ctv type plus a full copy of the 62.8 MiB recurrent state, whatever its length; the fixed copy is the larger half of an entry until roughly 6,000 tokens, where 10,880 B/token overtakes it.

There is one direct measurement of this growth and it is worth reading before you size your box. llama.cpp #27894 was opened as "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — 40 requests taking RSS from 1340 to 6379 MiB, perfectly linear — and then retracted by its own author once a contributor pointed at --cache-ram: the decisive run is --cache-ram 512 with nothing else changed, where RSS climbs by the same ~126 MiB per request until it reaches +507 MiB against the bound and then stays flat for ten consecutive requests. The growth is the prompt cache, it is bounded by the flag, and the flag is the fix. It is closed as completed, 2026-08-28. Two cautions if you cite it yourself: the 126 MiB is the only measured per-entry figure anywhere and it is roughly twice one recurrent state (62.8 MiB), so the reporter's identification of an entry as one state is an inference the arithmetic does not support even though the 126 stands as a measurement; and it was measured on a different model of the same architecture, and on ROCm — the reporter rebuilt with -DGGML_VULKAN=ON and got +126.0 MiB per request against ROCm's +126.3, so this is a model-and-request-layer behaviour and not a vendor story.

So: 32GB is the practical floor for this recipe as written, and 16GB is not enough. On a 32 GB box keep -cram at 2048 as above, or pass -cram 0 to turn caching off entirely and trade prompt-reuse speed for headroom. Lowering -ncmoe moves weight back onto the card and off the host, and the one lever that helps both sides at once is a smaller quant.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE. Check llama-server --version against the release list and move to b10630 or newer.

The model emits !!!! or uniform-probability gibberish

Searching the llama.cpp tracker for this architecture string turns up several gibberish-shaped reports, and the first thing to do with any of them is check which backend it is about. The nearest one to this recipe is issue #25857, filed 2026-07-18 on a Windows 12GB NVIDIA box: every token at an identical log-probability of −12.42 — the uniform distribution over the vocabulary — at any offload setting including pure CPU. Read its scope before you inherit the symptom. It is against a different model (a 64-block Qwen3.6 variant, where this one has 41), at builds b9994 and b10066, both older than the b10630 this page pins; it remains open with the stale label and zero comments as of 2026-08-30; and a separate publisher reports coherent output from an Apodex Q4_K_M on an older build still (vcruz305 smoke-verified -ngl 80 on b9835). The other reports in that neighbourhood are on other backends — #24168 is SYCL on an Intel Arc card, #23321 is Vulkan with --no-kv-offload and is closed — so neither describes a CUDA build. Rebuild at a current release first.

For completeness on the one issue this catalogue's older Apodex pages point at: llama.cpp #25717, the CUDA illegal-memory-access on vision requests with Qwen3.5-35B-A3B plus an mmproj, was closed as not_planned on 2026-08-30T01:13:13Z with the stale label. It was open when our RTX 3090 and RTX 5090 pages were written and it is not now. It also never applied to a text-only run — which is what --no-mmproj makes this one.

Something told you to turn Flash Attention off

Short answer: you cannot, at this page's context, on this card — and the reason is a term that appears nowhere else in this budget.

-fa off does not merely change how the cache is stored. It selects a different attention graph. On the non-Flash path llama.cpp materialises the full score matrix, src/llama-graph.cpp:2607-2612:

ggml_tensor * kq = ggml_mul_mat(ctx0, k, q);
cb(kq, "kq", il);
// note: this op tends to require high floating point range
//       while for some models F16 is enough, for others it is not, so we default to F32 here
ggml_mul_mat_set_prec(kq, GGML_PREC_F32);

That tensor is [n_kv, n_tokens, n_head] at F32. This model's head count is 16, and it is worth saying how you can check that two independent ways, because the obvious third way is wrong:

⚠️ Do not take the head count from attn_q. It is [2048, 8192] because the q projection carries a concatenated gate — 8192 is twice the 4096 that attn_output consumes — so that route silently doubles every cell of the table below. One further trap if you are grepping the header rather than parsing it: qwen35moe.attention.head_count_kv has qwen35moe.attention.head_count as a prefix, so a substring probe sees the two keys as one and can report the wrong value, or none. And the graph is reserved for the worst case, n_tokens = std::min(cparams.n_ctx, cparams.n_ubatch) (src/llama-context.cpp:595). So ctx × ubatch × 16 × 4 bytes, before anything else in the compute buffer:

At this recipe's -c 65536 with the default -ub 512 that single tensor is 2.000 GiB — twice the whole 1024 MiB compute reserve, and nearly double the 1.109 GiB of slack the budget has. It will not allocate. Dropping to -ub 128 gets it to 0.500 GiB, and then the cache change pushes the other way: you must give up -ctv q8_0, so KV goes 0.664 → 0.957 GiB. Working set becomes 9.184 GiB, slack 0.816, of which the score matrix alone takes 0.500 — leaving 0.316 GiB for the rest of the graph. That is not a configuration this page will recommend.

If you genuinely must run without Flash Attention, change the context, not the ubatch. At -c 32768, -ub 256, -ctk q8_0 -ctv f16: KV is 0.479 GiB, working set 8.706, score matrix 0.500, and 0.794 GiB is left over — which fits. Raising -ncmoe to 28 buys another 0.844 GiB and lets you keep -ub 512. Both cost throughput, and on this card that lands in the worst place: -ub is prefill's batch size, and prefill is the one phase where this GPU's compute has anything to do.

Why you are probably reading this, and it is a real report. llama.cpp #26609 is open (4 comments, filed 2026-08-05): a deterministic CUDA error: an illegal memory access was encountered in ggml_backend_cuda_synchronize, on qwen35moe with partial MoE expert offload, which its seven-probe isolation table pins to the CUDA flash-attention path — probe G, "--flash-attn off", is the only one that stops it. Read its scope before inheriting it: it is a different model (Qwen3.6-35B-A3B UD-Q4_K_M), on Windows 11, at builds b10107 and b10243 — both older than the b10630 this page floors and the b10666 it pins, and nobody has reproduced it on Apodex. But it is the same architecture string and the same technique, and the reporter's card is an RTX 5070 12 GB — the same VRAM tier as yours — so it is the closest thing to a reason a reader of this page would reach for -fa off at all.

Three things about that report are worth carrying. Its frozen args are --ctx-size 32768 --ubatch-size 512, and probe G passes at exactly those settings — where the score matrix is 1.000 GiB, the value in the table above. A later comment extends the matrix to nine rows across builds b10107, b10243 and b10488 (all three still older than the b10630 this page floors), and in every one of them the two off rows sit at n_ctx 32768 while every 131072 row ran with Flash Attention on. ⚠️ Read that for what it is: those 131,072 rows all crashed, with FA on, which is the bug being reported — so nobody has tried -fa off at 131,072 on a 12 GB card. The matrix is consistent with the ceiling computed above; it does not test it, and this page is not going to claim a measurement out of an untested cell. And probe G reads "--flash-attn off (+ KV f16, required)": the reporter took both halves of the cache to f16, which is the over-correction this page's -ctk/-ctv note is about. Only V is required to be unquantized. -ctk q8_0 -ctv f16 would have been legal and cheaper, and on a 12 GB card that difference is not small. Nobody has run any of this on an RTX 3080 Ti — if you do, tell us.

Throughput far below what your CPU should manage

Check three things before suspecting the model, and note that none of them is the GPU. -ncmoe too high pushes more blocks onto the host path than you need — each step is 13.5–15.6 MiB more read from RAM per token, about 0.3 ms on dual-channel DDR4-3200. Single-channel system memory halves the bandwidth those reads get, which halves the ceiling table above. And if you are using auto-fit rather than an explicit -ncmoe, pin the offload yourself rather than passing --fit-target, because of issue #27171.

If it is specifically the wait before the first token that is bad while generation itself is fine, look at the slot rather than the model: prefill is the only phase where the expert weights cross the bus, so a card in a chipset-fed ×4 slot or on a riser pays there and nowhere else.

nvidia-smi --query-gpu=pcie.link.gen.current,pcie.link.width.current --format=csv

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and llama.cpp will drive it with --spec-type draft-mtp — the exact spelling matters. On a 24 GB card that is a reasonable experiment. On this one, price it first, because the costs land on the scarcest resource you have:

• blk.40 stops being skipped: +0.444 GiB, and note that -ncmoe 26 does not touch it. The flag installs overrides for blk.0 through blk.25; only -ncmoe 41 would reach block 40's experts.
• The recurrent state is multiplied by the draft depth — need_n_rs_seq() at :394 returns draft.n_max — so --spec-draft-n-max 3 takes 0.061 GiB to 0.245.
• A second, single-layer KV cache is allocated for the draft context. It is f16 by default and untouched by your -ctk — it has its own flag, --spec-draft-type-k / -ctkd — so at 65,536 tokens on one layer it is 2,048 bytes per token, 0.125 GiB.

That is 0.753 GiB out of the 1.109 GiB of slack this page reserved, before any throughput arrives — which on its own is close to disqualifying. And the return is unknown here: nobody has measured MTP on this model on any card, and the mechanism argues against it. Speculative decoding wins when the GPU is waiting on itself; under expert offload it is waiting on your RAM, and the draft pass has to walk the same host-resident experts the target pass does. The nearest evidence agrees and is on a 12 GB card, though on a different model — an owner who describes their box as "rtx 3060 12G+96G RAM, llama.cpp b9213" and runs Qwen3.6-35B-A3B at --n-cpu-moe 25 with --spec-type draft-mtp --spec-draft-n-max 2 reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" — that is K-Tul-Khu, opening discussion #18 on that GGUF repo — and a second community member in the same thread, Diablo-D3, offers the reason: "CPU MoE seems to neuter any performance improvements that MTP adds." Neither is an org member and neither measured this model, so read it as a warning rather than a result. Measure both ways on your own prompts before keeping it, and please tell us what you get.

You would rather load a smaller quant than offload at all

That is a real alternative and one publisher recommends it: abenzerps publishes a hardware-guidance table whose 12 GB row names "IQ1_M or IQ2_M" with the note "IQ1_M full offload; IQ2_M may require hybrid offload". Weigh it with the rest of that card, which is candid about the cost — "IQ1_M is a 1.75-bit-per-weight format intended for severe memory constraints; use IQ2_M or higher when possible." — and which ends the same table with "Prefer Q4_K_M or higher when system memory permits."

That closing sentence is this recipe. A 1.75-bit quantisation of a model whose routed experts are 512 wide, used for tool-calling where a malformed argument is a failed task rather than a clumsy sentence, is a large quality bet to avoid a memory purchase. And check what the smaller quant actually buys before taking it. IQ2_M is 11.682 GiB in the lead repo — on a card whose usable pool this page sizes at ~11.0 GiB, that leaves nothing for the KV cache or the compute buffers, which is what their own table means by "may require hybrid offload". Below Q4 the trade is quality for a shallower -ncmoe, not quality for no -ncmoe at all. Only IQ1_M — 8.216 GiB in that publisher's ladder — escapes offload entirely, and it is the one its own card tells you to avoid when you can.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the registry.ollama.ai manifest for apodex returns 404 as well (checked 2026-08-30, with ollama.com/library/qwen3.5 and the registry.ollama.ai manifest for qwen3 both returning 200 as controls). Use llama.cpp directly. If a tag appears later, please tell us.