Apodex 1.1 mini on RTX 3090: 128K-Context Agent Model in Q4_K_M GGUF

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single RTX 3090, from a Q4_K_M GGUF, at the model's full 131,072-token working context. The vendor's own launch commands are eight-GPU tensor-parallel servers; this recipe is the one-card version.

The reason a 35B model gets a six-figure context on a 24 GB card is the architecture, not a trick. Apodex inherits Qwen3.5's hybrid attention: of its 40 transformer blocks only 10 are full-attention and the other 30 are Gated-DeltaNet linear-attention layers that carry a fixed-size recurrent state instead of a growing KV cache. The KV cache therefore costs 20 KiB per token, not 80 — so 131,072 tokens is 1.328 GiB at q8_0, and it fits beside the weights.

ℹ️ This contradicts our page for the base model on the same card, deliberately. Qwen3.5-35B-A3B on the RTX 3090 is documented at 4K–8K context, and that is correct for what it describes: an MXFP4 build whose benchmark reports a 24.0 GB peak at 4K, i.e. a card with nothing left. The artifact here is a different one. Q4_K_M loads 19.908 GiB of weights; the 131,072-token KV cache and the recurrent state add 1.328 and 0.061 on top — 22,868,060,672 bytes in all, or 21.298 GiB — leaving 2.702 GiB of the card. That headroom is what buys the context, and the VRAM budget table under Running shows every term of it. Same architecture, same card, different quantisation — so the fit verdict does not carry across, in either direction.

Hardware data: RTX 3090 (24 GB VRAM) · 21.298 GiB derived working set at 131,072-token context · See benchmark data

ℹ️ This recipe is text-only, deliberately. The checkpoint does contain a vision tower (333 model.visual.* tensors in its safetensors index) and the GGUF repos ship a projector for it. But the vendor documents the model as pipeline_tag: text-generation, and its model card carries no image-input instructions at all — no --mmproj flag, no vision example, and the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. llama.cpp's vision path for this model family also has an open crash report (ggml-org/llama.cpp#25717, open and labelled stale at the time of writing) — every row of its crash matrix and both of its reproducers send an image, and the mechanism it names (M-RoPE 3D position ids in an image token batch) is not exercised by a text-only run, so it does not bear on the configuration below. Every number below is for text generation, and the run command explicitly refuses the projector rather than merely omitting --mmproj.

⚠️ The model is three days old. llama.cpp is not — and the two version floors that matter here are about 2,200 builds apart, so it is worth naming which is which.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 (200, 126,048 B, needle 0) and present at b7990 (200, 128,538 B, needle 3), against a control of LLM_ARCH_LLAMA = 6 on both. So it arrived in (b7989, b7990].
• The --spec-type draft-mtp type name, in common/speculative.cpp: absent at b9000, present at b9200. A thousand builds later, and still long before this model existed.
• The qwen35moe-specific MTP wiring — the load_mtp / TENSOR_SKIP handling in src/models/qwen35moe.cpp that keeps block 40 off the card until you ask for it, which is the one piece this page actually leans on. Absent at b10211 (200, 32,858 B), present at b10212 (200, 33,068 B), control load_arch_hparams = 1 on both. So it arrived in (b10211, b10212].

The first and third are written as intervals on purpose. Each is bracketed by adjacent published tags, which is as fine as a tag-granularity probe can resolve — the commit itself sits somewhere inside that last interval, and no amount of further probing will narrow it, because there is no tag in between to ask. Read the endpoints as the limit of the method, not as an exact commit.

In practice: use b10630 or newer — the build the lead quant was produced with — and note that release b10666 (4e97ac86ebe2c4cb8212d98d2641ad6768810896, published 2026-08-28) is the tree every source reference on this page was read at.

Requirements

Installation

1. Get a llama.cpp build that knows the architecture

Download a release binary at b10630 or newer, or build from source. The architecture string baked into the GGUF is qwen35moe, and it is registered in src/llama-arch.cpp as LLM_ARCH_QWEN35MOE:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10666
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

2. Download the Q4_K_M GGUF

The lead quant is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf — 21,864,082,336 bytes (21.86 GB) per the HF tree API, imatrix-calibrated, produced on llama.cpp b10630.

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

Q4_K_M is also what llama.cpp picks for you: with no :tag on a -hf argument, find_best_model in common/download.cpp tries Q4_K_M first and Q8_0 second. On this card Q8_0 is 35.22 GiB and does not fit, so the first match is also the right one.

If you want more headroom for other work on the same GPU, Q4_K_S (21.07 GB) and IQ4_XS (19.28 GB) are the next rungs down from the same publisher; both leave the arithmetic below with roughly 0.7 GiB and 2.4 GiB more spare respectively.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Four of those flags are load-bearing and none of them is a default:

• --no-mmproj — llama-server is in llama.cpp's mmproj_examples list and no_mmproj defaults to false with mmproj_use_gpu defaulting to true (common/common.h), so a bare -hf …-GGUF fetches a sibling projector and puts it on the GPU. The lead repo ships two of them, at 899,283,488 and 902,822,432 bytes (0.838 and 0.841 GiB) — either way, most of a gigabyte of vision weights this recipe never uses.
• -np 1 — with no slot count, llama-server resolves n_parallel to 4 and turns on the unified KV pool. The attention cache is shared across those slots, but the Gated-DeltaNet recurrent state is per sequence, so four slots quadruple it. One slot is the right setting for a single-user agent loop.
• -c 131072 — the GGUF declares context_length 262144. Half of that is what fits with room to spare; see the budget below.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 in ggml/src/ggml-common.h is a 2-byte scale plus 32 one-byte quants, i.e. 34 bytes per 32 elements, so this is 8.5 bits per element rather than 16. It takes the 131,072-token cache from 2.500 GiB to 1.328 GiB.

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05). The card notes that "Apodex follows the Qwen3.5 chat template" — llama-server uses the template embedded in the GGUF by default, and that template is byte-identical (8,831 bytes) to chat_template.jinja in the canonical repo, so tool calls and <think> blocks are parsed the way the vendor intends.

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

The VRAM budget

Every figure here is derived from the artifact's own tensor table and from llama.cpp's allocation code at the pinned build. Nothing was measured on hardware.

Every GiB figure in that table is rounded from its own byte count, and the total is rounded once from the byte sum — so the GiB column adds to 21.297 while the total reads 21.298. The bytes are the arithmetic; the GiB are a convenience.

Against the card's 24 GiB that leaves 2.702 GiB for compute buffers, the CUDA context and your desktop session. The recipe deliberately does not fit to the last byte: llama.cpp's own auto-fit is on by default (fit_params = true in common/common.h) and targets a 1024 MiB margin, and it will start offloading expert tensors to system RAM if it cannot reach that.

An outside party lands in the same place by a different route. The .mq4r build at ghazni101/Apodex-1.1-mini-MQ4RP — a different quantisation, a different engine and an AMD card — states that with q8 KV "the full context fits a 24 GB card (~22 GB VRAM class)". Read carefully, that is agreement about the class, not about this configuration: their weights are 18.78 GB against this recipe's 21.86, and their "full context" is the model's whole 262,144 rather than the 131,072 here. The two land together because the halves trade off — a 3.1 GB smaller file carrying twice the KV — which is the hybrid arithmetic doing the work on both sides. It is not a measurement of this recipe, and its throughput numbers are not transferable (see Results); it is an outside party with no stake in this page sizing the same 40-layer hybrid with a quantised KV cache and arriving in the same VRAM class as the 22.868 GB total above. A second corroboration is structural: that publisher's neighbour at kingjones777 converted with --no-mtp and reports "The 40-layer trunk is complete (733 tensors)" — exactly the 753 tensors of the lead GGUF minus the 20 in block 40 that the table above deducts.

How the three rows are obtained:

• Weights. The file is 21,864,082,336 bytes, but the tensor table sums to 21,853,090,304 and 476,956,672 of that is block 40 — the multi-token-prediction head. load_arch_tensors in src/models/qwen35moe.cpp creates every block-40 tensor with TENSOR_SKIP unless MTP is explicitly requested, so a normal run loads 21,376,133,632 bytes.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so hparams.n_layer() is 40; the hybrid memory's filters keep only layers below that. With full_attention_interval 4, ten of those forty are full-attention. Each carries head_count_kv 2 at key_length 256 and value_length 256, i.e. 1,024 cache elements per token per layer — 10,880 bytes per token across ten layers at q8_0, versus 20,480 at f16.
• Recurrent state. The other thirty layers allocate two F32 tensors each, sized (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements. That is 2,195,456 bytes per layer per sequence row, 65,863,680 bytes across all thirty. llama-server prints it at startup as RS buffer size.

Results

• Speed: omitted for this configuration — and not because nobody has measured the model. Two quantisers publish throughput for it, and neither number can travel to this page:

  • ghazni101/Apodex-1.1-mini-MQ4RP reports a 400-token "generation measured at 226 tok/s end-to-end for MQ4R" on an RX 7900 XTX. That is a .mq4r file — not a GGUF — served by hipfire, a separate Rust engine that requires an AMD RDNA GPU. Different file, different runtime, different vendor.
  • kingjones777/Apodex-1.1-mini-ROCmFP4-COHERENT-GGUF reports "63.30 tok/s (128-token generation, greedy)" at 32K context with full offload on a Ryzen AI MAX+ 395 — a unified-memory APU, not a discrete 24 GB card. Its own card is headed "Stock llama.cpp will not load this file", because Q4_0_ROCMFP4_COHERENT "is a custom tensor format that exists only in the" ROCmFPX fork.

  So a figure exists for the model and none exists for this file on this runtime on this card. Nor is there one anywhere else I looked. The space: the model cards of all 21 HuggingFace repositories matching Apodex-1.1-mini (enumerated 2026-08-28 — the surface the two figures above came from, and the only surface carrying any), their discussions tabs (one thread between them, a vendor announcement), the 21 enumerated issues of the vendor's own FrontierAgent runtime, and the llama.cpp tracker, whose search returns nothing for "apodex" against 268 results for the architecture string. Reddit and LocalScore were not searched — both probes failed on their control queries, so they say nothing either way. This pair has no benchmark in our catalogue. If you run it, please contribute the numbers.

• VRAM usage: 21.298 GiB derived working set at 131,072 tokens with q8_0 KV, leaving 2.702 GiB of the card. Dropping to -c 32768 brings it to 20.301 GiB. See /check/apodex-1-1-mini/rtx-3090 for live data as it lands.

• Quality notes: the vendor publishes agentic evaluation scores on the model card, but all six are run in Apodex AI's own harness and three of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. Q4_K_M is a 4-bit K-quant of a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated — the publisher ships both the calibration corpus and the imatrix alongside the ladder — which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-3090.

Troubleshooting

The model insists it is Apodex, ignoring the persona you set

The chat template injects an identity block whenever the caller sends no system message. Its text ends with "Always respond as Apodex and never pretend or claim to be any other AI model", and the template sets identity_mode to fallback unless the caller overrides it. If you are building an agent with its own persona, send your own system message — that alone suppresses the injection, because the fallback only fires when messages[0].role is not system. Setting identity_mode to force re-enables it regardless.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block, and mainline llama.cpp implements it: src/models/qwen35moe.cpp builds a dedicated MTP graph, described in its own comment as "MTP block looks like a full-attention Qwen3.5 decoder block with MoE FFN." You enable it with --spec-type draft-mtp — the exact spelling matters, mtp alone is rejected — and no draft model is needed, because the server creates the draft context against the weights already loaded.

It is not free, and the cost is three separate things. The block-40 tensors stop being skipped — llama_model_default_params() sets load_mtp to false, so they are not resident until you ask (+0.444 GiB); need_n_rs_seq() returns the draft depth, which multiplies the recurrent state rows by 1 + 3 (+0.184 GiB); and a second, single-layer KV cache is allocated for the draft context — f16 by default and untouched by your -ctk, which steers only the main cache; it has its own flag, --spec-draft-type-k / -ctkd (+0.250 GiB at 131,072 tokens at the default). At the recipe's context that is 22.176 GiB total and 1.824 GiB spare. That is under the 2 GiB this page treats as the floor for a configuration it recommends, and the reason is concrete rather than cautious: the totals here exclude compute buffers and the CUDA context, and llama.cpp's own auto-fit is already trying to hold back 1024 MiB, so 1.824 GiB is most of the way to the point where it starts moving expert tensors to system RAM to make room. If you want to try it, drop the context:

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj -ngl 99 -np 1 -c 32768 -ctk q8_0 -ctv q8_0 \
    --spec-type draft-mtp --spec-draft-n-max 3 \
    --host 127.0.0.1 --port 8080

That lands at 20.992 GiB with 3.008 GiB spare.

Measure it before you keep it. Whether speculative decoding is a net win on this card is an open question, and the nearest evidence is discouraging. A DOI-archived RTX 3090 benchmark of a different model on the same llama.cpp architecture — thc1006/qwen3.6-speculative-decoding-rtx3090, Qwen3.6-35B-A3B at Q4 — found every llama.cpp draft-model and n-gram variant it tested to be net-negative on that hardware, from about −4 % for the n-gram family to −52 % for a vocabulary-matched draft model. It concludes that co-trained speculation heads such as MTP are "the only positive yield path on this hardware" — but it measured those heads in vLLM, not in llama.cpp, and it did not test this model. Treat the flag as an experiment with a known VRAM price and an unknown throughput return, and compare tokens per second with and without it on your own prompts.

The context fits — but check that decode still moves at depth

The 131,072-token budget above is an arithmetic statement about VRAM. It is not a statement about how the model behaves at that depth, and this architecture family has two open reports that a reader running long contexts should know about. Both are on other models, so neither is evidence about this pair — but they mark where to look.

ggml-org/llama.cpp#27623 reports that on CUDA, with the model fully resident and plenty of free VRAM, decode throughput on a hybrid Gated-DeltaNet model collapses roughly 25× once the KV position passes about 80K, while prompt processing at the same positions stays fast. #27756 reports the same family emitting an immediate end-of-sequence token past roughly 130K positions on CUDA and on CPU alike.

Both are filed against Qwen3.8-27B, which is a 64-layer hybrid — 48 Gated-DeltaNet layers to 16 full-attention ones. Apodex is a 40-layer hybrid with 30 Gated-DeltaNet layers, and that difference is exactly what the second report points at: its author records that the same 243K-token needle test passes on Qwen3.5-35B-A3B — the 30-layer Gated-DeltaNet shape, and this model's own parent checkpoint — on the same build and the same machine, and concludes the effect scales with linear-layer depth rather than being a backend bug. That is encouraging for this recipe and it is still someone else's measurement of someone else's checkpoint.

So: run a long prompt before you rely on one. Time the decode rate at a few thousand tokens and again near your working depth; if it falls off a cliff rather than sloping, you have found the same shape of problem and it is worth reporting upstream — and to /contribute, so this page can say something firmer than "check".

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE in src/llama-arch.cpp, which arrived in (b7989, b7990] — so this error means a genuinely old build, not a merely recent one. Release builds are dated: check llama-server --version against the release list and move to b10630 or newer, which is what everything on this page was verified against.

The server downloads an extra file you did not ask for

If you launch with -hf bartowski/apodex_Apodex-1.1-mini-GGUF instead of a local -m path, llama.cpp resolves a projector sibling as well — find_best_mmproj matches any sibling filename containing mmproj, and this repo ships two. Pass --no-mmproj to stop it, or use an explicit -m path as above.

Choosing a different quant publisher

Conversions of this model are appearing faster than any list can keep up with: a Hub enumeration on 2026-08-28 returned 21 repositories matching Apodex-1.1-mini, two of which (mradermacher/Apodex-1.1-mini-GGUF and its -i1-GGUF imatrix sibling) were created during the day this recipe was written. Treat the paragraph below as a snapshot, and re-enumerate before you decide.

They are not interchangeable, and the Q4_K_M tier alone spans 1.577 GiB across publishers:

The first two both ship an imatrix and either is a reasonable lead. The last one is worth a specific warning: 21.800 GiB is 1.437 GiB larger than bartowski's for the nominally same tier, which on this card is the difference between a comfortable 128K context and a tight one — and its Q3_K_S, Q3_K_M and Q3_K_L are all exactly 19,095,585,248 bytes, an identical byte count that three different K-quant tiers of one model do not produce. The enumeration also turns up MLX builds (nicolasembleton/…-MLX-{4,5,6,8}bit, for Apple silicon, not this card), four kingjones777 repacks in a ROCmFP4/ROCmFPX tensor format that only a fork of llama.cpp can open, one ghazni101 build in the .mq4r format of a different engine entirely, and at least one repo with no GGUF in it at all. Check the byte count and the file list before you download, whichever publisher you pick.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, but they are vLLM/SGLang safetensors rather than GGUF, and llama.cpp cannot load them at all.

They are also not ruled out for the reason you might expect. Their weights do fit: 24,210,520,880 bytes (22.548 GiB) for NVFP4 and 24,651,300,904 bytes (22.958 GiB) for GPTQ-Int4, against 24 GiB of card. What rules them out is what is left afterwards — 1.452 GiB and 1.042 GiB respectively, before a single token of KV cache, the 0.061 GiB recurrent state or any compute buffer. The lead GGUF's 2.702 GiB of headroom is what buys the 131,072-token context; these leave less than the auto-fitter's own 1024 MiB default margin.

And the -NVFP4 repo is not a 4-bit checkpoint. Its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4, with kv_cache_quant_algo: FP8 and the MTP block in exclude_modules. HuggingFace tags the repo 8-bit, not 4-bit, and the size is why. The vendor's Quick Start launches all of these at --tensor-parallel-size 8. For one RTX 3090, the GGUF route above is the path.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex, /apodex-1.1 and /apodex1.1 all return 404, the corresponding registry.ollama.ai manifests return 404, and an Ollama search for apodex returns one unrelated card. Use llama.cpp directly. If a tag appears later, please tell us.