Apodex 1.1 mini on RTX 5090: Q5_K_M Agentic LLM at the Full 262K Context

What You'll Build

A local, OpenAI-compatible agent endpoint serving apodex/Apodex-1.1-mini — an Apache-2.0 agentic MoE fine-tuned from Qwen/Qwen3.5-35B-A3B — through llama-server, at the Q5_K_M quant tier and the model's full 262,144-token context, both of which the RTX 5090's 32 GB holds and a 24 GB card does not.

That is the whole point of this page. This model is not faster on a 5090 because of anything the recipe does; it is bigger and longer-sighted. The 8 GB over a 24 GB card buys one quant tier up and a context window six to eight times what the same card class can hold at the same tier — which for a long-horizon agent that reads files, runs tools and keeps its own <think> trace is the axis that matters.

Hardware data: RTX 5090 (32 GB VRAM) · derived working set 28.532 GiB at Q5_K_M / 262,144 tokens, leaving 3.468 GiB · See benchmark data

⚠️ Nothing here is a measured run. No community throughput or peak-VRAM measurement for this model exists on any surface searched (see Results). Every figure below is arithmetic over bytes read out of the artifact itself — the GGUF tensor table and KV header — and the derivation is shown so you can check it. If you run this, please contribute the numbers.

ℹ️ Text-only, deliberately. The vendor's card declares pipeline_tag: text-generation and documents no image input; the vision tower is inherited from the base checkpoint. The quant repo you are sent to for the weights disagrees: its card declares pipeline_tag: image-text-to-text, lists image among the supported inputs, ships two mmproj projectors and notes that llama.cpp fetches one automatically under -hf. That disagreement is why the run command below disables the projector explicitly rather than merely omitting it — and see Troubleshooting for the open upstream crash that makes it more than housekeeping.

Requirements

b10630 is advice, not an architecture floor, and it is worth being precise about which. It is the build the lead quant was produced with — bartowski's card states "These quants were made with llama.cpp release b10630 - if this model's architecture is newly supported, you'll need that release or newer to run them." — and it is the build family every source claim on this page was verified against, so it is the version to reach for if you have no reason to prefer another.

But that sentence is conditional, and the condition does not hold: qwen35moe is not newly supported. Fetching src/llama-arch.cpp at successive release tags and narrowing to adjacent published ones, LLM_ARCH_QWEN35MOE is absent at b7989 and present at b7990 — so it landed in the interval (b7989, b7990], roughly 2,600 builds before the one the quant was made with. Both of those tags return the file with the control symbol LLM_ARCH_LLAMA in it, so the absence is a real absence; the 404s scattered through that range are unpublished tags and were discarded rather than read as evidence. Loading this GGUF is therefore not what constrains your build: the header declares general.architecture = qwen35moe, and any build from b7990 onward recognises it. The one thing on this page that does carry a real floor is the optional MTP speculative-decoding path, and its floor is named where it is used.

Installation

1. Get a CUDA llama.cpp build

Take a recent CUDA release binary from ggml-org/llama.cpp releases — b10630 or newer is the safe choice, for the provenance reason in Requirements rather than because anything below it fails to load the model — or build from source with CUDA enabled. Nothing model-specific is required: no pip install flash-attn, no separate kernel package. llama.cpp's FlashAttention lives inside its own CUDA backend.

2. Download the Q5_K_M quant, and nothing else

hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q5_K_M.gguf \
    --local-dir ./apodex-1.1-mini

Download the single file by name on purpose. The repo also holds two mmproj projectors of 0.90 GB each, and llama-server's -hf shortcut picks one up automatically and places it in VRAM — the flag that turns that off is in the run command below. This recipe is text-only, so those bytes buy nothing.

Running

llama-server \
  -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q5_K_M.gguf \
  --no-mmproj \
  -ngl 99 \
  -c 262144 \
  --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
  --host 0.0.0.0 --port 8080

-c 262144 pins the model's full trained context, which is what qwen35moe.context_length in the GGUF header declares. The sampling values are the vendor's own recommendation for agentic work (temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05 in the card's Best Practices block). --no-mmproj keeps the vision projector off the card.

Three behaviours of the default server are worth knowing before you touch anything else:

• The chat template is already in the file and already in use. llama.cpp's use_jinja defaults to true, so the model's own template renders tool schemas passed via the API's tools= parameter. The card is explicit about the format: "Apodex follows the Qwen3.5 chat template", with tool calls emitted as <tool_call><function=…> and reasoning inside <think>…</think>.
• If you send no system message, the model is given one. The template embedded in the GGUF sets identity_mode to fallback and injects an "Apodex" role-definition block whenever the first message is not a system message. Send your own system prompt and that block is replaced by yours; that is usually what an agent harness wants.
• Four slots share one KV pool. llama-server resolves its auto --parallel to 4 slots with a unified KV cache, so a single request can still use all 262,144 tokens, while the per-sequence recurrent state is allocated four times over. The budget below accounts for that.

Point any OpenAI-compatible client at http://localhost:8080/v1. The vendor's own agent harness, ApodexAI/FrontierAgent (Apache-2.0), speaks that protocol, as do the generic agent clients.

Results

• Speed: omitted, and worth saying exactly why rather than just that nothing was found. Throughput measurements for this model do exist — they are all on AMD hardware, under runtimes this recipe does not use. The author of the ROCm builds mentioned below publishes self-measured figures of 32.54 to 64.87 tok/s on a Ryzen AI MAX+ 395 at full offload with a 128-token greedy generation, in a tensor format that only loads in a third-party ROCm fork of llama.cpp; and ghazni101/Apodex-1.1-mini-MQ4RP reports 226 tok/s on an RX 7900 XTX under hipfire, a Rust engine for AMD RDNA cards, from a .mq4r container that is not a GGUF at all. Neither transfers here: different vendor, different runtime, different artifact — quoting either against this card would be inventing a number. What is genuinely absent, across the spaces I searched, is any figure on NVIDIA hardware: the HF discussion tabs of all seven vendor and GGUF repositories (one thread, a vendor announcement), the model cards of the four GGUF ladders, the 21 enumerated issues and pull requests of ApodexAI/FrontierAgent, a GitHub issue search across ggml-org/llama.cpp, and two web searches naming the model and the card. Reddit search and LocalScore both failed their positive controls, so neither is inside that space. The six benchmark scores on the vendor's own card are their own evaluation runs, driven by their own Agent Team harness, and not one of them names hardware — so there is nothing there to quote either. If you measure generation or prefill on this card, please contribute it via /contribute and it will land on /check/apodex-1-1-mini/rtx-5090.

• VRAM usage: derived, not measured. The three addends, all read from the artifact:

  Component	Bytes	GiB
  Weights actually loaded (Q5_K_M, MTP block skipped)	25,004,239,360	23.287
  Attention KV cache at 262,144 tokens, f16	5,368,709,120	5.000
  Gated-DeltaNet recurrent state, 30 layers × 4 slots	263,454,720	0.245
  Total	30,636,403,200	28.532

  Against 32 GiB that leaves 3.468 GiB for the CUDA context and llama.cpp's compute buffers. Treat that as a ceiling on headroom rather than a floor: the allocator sees slightly less than the nameplate capacity, and a card also driving a display gives up more again. The conclusion survives either way, because it is still comfortably above the 1024 MiB margin that llama.cpp's own auto-fit targets by default (fit_params_target, common/common.h).

  Where each number comes from. Weights: the file is 25.49 GB on disk, but the tensor table sums to 25,481,196,032 B and the blk.40 MTP block inside it — 476,956,672 B — is not loaded unless you ask for it, because llama_model_default_params() sets load_mtp = false, src/models/qwen35moe.cpp then marks that block TENSOR_SKIP, and the loader logs model has unused tensor … -- ignoring and subtracts its bytes. KV: the header declares block_count 41, nextn_predict_layers 1, full_attention_interval 4, head_count_kv 2, key_length 256 and value_length 256; llama.cpp's n_layer() is n_layer_all − n_layer_nextn = 40, of which the layers at indices 3, 7, … 39 are full-attention and the other 30 carry constant-size recurrent state instead — 10 × 2 × 256 × 2 × 2 B = 20 KiB per token, and only those ten layers grow with context. Recurrent state: ssm.conv_kernel 4, ssm.inner_size 4096, ssm.state_size 128, ssm.group_count 16 give 3 × (4096 + 2 × 16 × 128) + 128 × 4096 elements at fp32 = 2,195,456 B per layer per sequence slot, which is why the slot count matters and the context length does not.

• Quality notes: Q5_K_M is the tier the 32 GB card exists for here. Its 23.287 GiB of weights alone would leave 0.713 GiB on a 24 GB card — not enough for a KV cache, let alone this one — so a 24 GB card is a Q4 card for this model, and a short-context one at that. The 20 KiB-per-token cache is small because only a quarter of the layers are attention layers; that is a property of the hybrid Gated-DeltaNet architecture, not of the card, and it is the reason a six-figure context is discussable at all. As an agentic model the output is reasoning-heavy: <think> blocks consume the same KV budget as anything else, and the vendor caps its own recommendation at max_tokens: 32768.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-5090.

Speculative decoding with the built-in MTP head

The quant carries the model's multi-token-prediction block, and mainline llama.cpp can drive it: docs/speculative.md lists draft-mtp as "Use Multi Token Prediction (MTP) heads from the main model", and bartowski's card says "MTP layers act as a built-in draft model, letting llama.cpp run speculative decoding for faster generation." Add --spec-type draft-mtp to the command above and llama.cpp builds a draft context against the same loaded model rather than loading a second one.

This is the one part of the page with a genuine version floor, so here it is with the probe that found it. The qwen35moe-specific wiring — the load_mtp flag and the TENSOR_SKIP that un-skips blk.40 — lives in src/models/qwen35moe.cpp. Fetching that one file at successive release tags and narrowing to adjacent published ones, it is absent at b10211 and present at b10212, so the wiring landed in the interval (b10211, b10212]. Both tags return the file with its load_arch_hparams control symbol present, so the absence is real rather than a missing tag, and the file grows from 32,858 to 33,068 bytes across the boundary. Read that as an interval, not a point: a tag-granularity probe can say the flag does nothing for this model on a build below b10212 and works from b10212 onward, and it cannot say anything finer.

Two costs, both real and both worth budgeting before you enable it. The MTP block stops being skipped, so its 0.444 GiB comes back into VRAM (Q4_0 in this quant — bartowski quantises it low deliberately, since imatrix calibration never exercises it). And the draft context is sized to the same -c as the target, so it brings its own cache. At 262,144 tokens the 3.468 GiB of headroom above is not obviously enough for both; drop -c to 131,072 first, where the same table gives 5.968 GiB of slack, and raise it once you have watched nvidia-smi. There is no published measurement of the acceptance rate or the speed-up on this model, so treat the whole feature as an experiment rather than as part of the happy path.

Troubleshooting

The server starts, then dies on an image request

Do not pass --mmproj with this architecture yet. llama.cpp issue #25717 — open, labelled stale, last updated 2026-08-15, reported against builds b10002 and b10016, both older than the b10630 these quants were made with — documents a CUDA illegal-memory-access crash on vision requests for Qwen3.5-35B-A3B plus its mmproj, traced in the thread to M-RoPE 3D position ids and the find_slot: non-consecutive token position that the image token batch produces. Every row of the reporter's crash matrix is a request carrying an image, and nothing in the thread reproduces it on a text-only request. Two of the seven rows are reported working, and both have flash attention off — but two of the crashing rows are FA-off as well, so FA-off is necessary in that matrix and not sufficient. Apodex 1.1 mini is the same architecture, so the risk transfers even though the report is not about this checkpoint. The recipe passes --no-mmproj and stays on the text path, which the report does not implicate. If you turn flash attention on and hit an unrelated instability, -fa off is the first thing to try.

It fits on paper but llama.cpp quietly gives me less context

llama.cpp's auto-fit is on by default (fit_params = true) and will reduce unset parameters to leave a 1024 MiB margin per device, down to a floor of 4096 tokens. Because the command above sets -c explicitly, auto-fit will not shrink your context — it will move layers to the CPU instead, which shows up as a collapse in generation speed rather than an error. If that happens, check the load log for offloaded layers, then either drop to a shorter context or turn the behaviour off with -fit off and let the allocation fail loudly instead.

I want maximum quality instead of maximum context

Q6_K is the tier above, and it is a straight trade. Its tensor table sums to 30,515,319,296 B, of which the same MTP block is skipped, leaving 27.975 GiB loaded. Add the 0.245 GiB of recurrent state and the card has 3.780 GiB left for the cache, so the full 262,144 tokens (5.000 GiB of KV) does not fit. At 65,536 tokens the total is 29.471 GiB with 2.529 GiB spare; at 131,072 it is 30.721 GiB and the 1.279 GiB remainder is thinner than the margin llama.cpp's own auto-fit aims for. Pick Q6_K only if you know your prompts are short.

Why not the first-party NVFP4 build?

The vendor publishes apodex/Apodex-1.1-mini-NVFP4, and on a Blackwell card that looks like the obvious answer. Three things to know before you reach for it.

It is not an NVFP4 checkpoint. Its hf_quant_config.json declares quant_algo: MIXED_PRECISION, and of its 290 quantised layer entries exactly 40 are NVFP4 — the routed-expert stacks — while the other 250 are FP8: every shared expert, every linear-attention projection, every self-attention projection. The KV cache quant is FP8 as well, and the MTP block is excluded from quantisation entirely. Both formats are native on Blackwell, so this is a sound design; it is simply not what the repo name says, and a budget that assumes 4-bit throughout will be wrong.

It is safetensors, not GGUF, so llama.cpp cannot load it at all. The runtime that can is vLLM, which registers this exact architecture (Qwen3_5MoeForConditionalGeneration, in vllm/model_executor/models/registry.py) and this exact quantisation scheme (modelopt_mixed, in quantization/modelopt.py) in mainline. Its weights are 24.21 GB (22.548 GiB) across three shards, so it does fit a 32 GB card with room for a cache — and, contrary to the usual assumption, it would also fit the weights on a 24 GB card, with 1.45 GiB left over, which is not enough to serve anything.

It is heavier to install and there is nothing to compare it against. No throughput measurement exists for either path on this card, so choosing vLLM over llama.cpp here would be choosing on faith. The vendor's own launch commands are no help: the card's Quick Start runs SGLang and vLLM at --tensor-parallel-size 8 on the 71.90 GB bf16 weights, which is an eight-GPU server configuration, not a single-card one. If you build the vLLM path and measure it, that is exactly the datapoint this page is missing — /contribute.

Which GGUF publisher should I use?

Searching the Hub for Apodex-1.1-mini on 2026-08-28 returns 21 repositories, of which nine carry at least one .gguf. Four of those nine are single-file ROCm builds, leaving five multi-tier ladders from four publishers; the rest of the 21 are MLX conversions, safetensors, a non-GGUF container and one repo with no weights in it at all.

bartowski's 28-tier ladder is the one this recipe pins: imatrix-calibrated with the calibration file published alongside the quants, the MTP block retained at Q4_0, and both projectors shipped. Two equally serious alternatives come from mradermacher, whose plain ladder carries 11 tiers and whose imatrix ladder carries 14; both put Q5_K_M at 25.35 GB, marginally smaller than the file this recipe downloads, so the budget above gains a little slack rather than losing it if you take one. abenzerps/Apodex-1.1-mini-GGUF is the fourth publisher, 12 tiers, Q5_K_M also at 25.35 GB.

Two groups are excluded, and for different reasons. The four single-file builds under kingjones777/ are quantised into Q4_0_ROCMFP4 / ROCMFPX tensor formats which that author's own card says exist only in a third-party ROCm fork of llama.cpp — so a stock CUDA build cannot load them at all, whatever the card is. And vcruz305/Apodex-1.1-mini-GGUF is worth avoiding until it is explained: its Q3_K_S, Q3_K_M and Q3_K_L files are all exactly 19,095,585,248 bytes, which three genuinely different quant tiers cannot be. Its other tiers are also consistently the largest of the four ladders — 8 of its 9 tiers run 0.74 GB to 3.78 GB above bartowski's same tier, though the ninth, its Q6_K, is 1.29 GB below it, so read that as a pattern rather than a rule.

There is no Ollama tag for this model

Correct, as of this writing. ollama.com/library/apodex returns 404, the registry manifest endpoint returns 404 for apodex, apodex-1.1-mini, apodex1.1 and apodex-mini on both :latest and :35b, and the library search returns no matching model. Use llama-server directly, or point LM Studio at the same GGUF — it runs llama.cpp underneath, so the artifact and the fit arithmetic are identical.