Apodex 1.1 mini on RTX 4090: 128K Agent Context in Q4_K_M, and the 24 GB Ceiling

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single RTX 4090, from a Q4_K_M GGUF, at a 131,072-token working context. The vendor's own launch commands are eight-GPU tensor-parallel servers; this is the one-card version.

A 35B model gets a six-figure context on a 24 GB card because of the architecture, not a trick. Apodex inherits Qwen3.5's hybrid attention: of its 40 transformer blocks only 10 are full-attention, and the other 30 are Gated-DeltaNet linear-attention layers carrying a fixed-size recurrent state instead of a growing KV cache. The KV cache therefore costs 20 KiB per token rather than 80 — so 131,072 tokens is 1.328 GiB at q8_0, and it fits beside the weights.

Hardware data: RTX 4090 (24 GB VRAM) · 21.298 GiB derived working set at a 131,072-token context · See benchmark data

⚠️ This page's whole budget is capacity, and capacity is the one thing the 4090 does not add.

Every number below is byte-for-byte identical to what our RTX 3090 page derives, and that is the finding rather than an oversight: both cards hold 24 GB, the working set is decided by the artifact and the context length, and neither is a function of how fast the silicon is. So the honest framing of this page is not "the 4090 runs it better". It is: here is the entire 24 GB budget, here is exactly what the leftover headroom can and cannot buy, and the axis on which a 4090 might actually differ from a 3090 — throughput — has no measurement for this model on either card, from us or from anyone else (see Results).

Throughout, conclusions are labelled. A capacity conclusion is arithmetic over bytes and transfers to any 24 GB card. A bandwidth conclusion does not transfer, and this page makes none it cannot support.

ℹ️ This recipe is text-only, deliberately. The checkpoint does contain a vision tower (333 model.visual.* tensors in its safetensors index) and the GGUF repo ships two projectors for it. But the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions at all — no --mmproj flag, no vision example, and the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. Every number below is for text generation, and the run command refuses the projector rather than merely omitting it. See Troubleshooting for the upstream vision crash report and for its current, changed status.

⚠️ The model is recent. llama.cpp is not — and the two version floors that matter are about 2,200 builds apart.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990, so it arrived in (b7989, b7990]. That interval is bracketed by adjacent published tags, which is as fine as a tag-granularity probe can resolve — read the endpoints as the limit of the method, not as a commit.
• The qwen35moe-specific MTP wiring — the load_mtp / TENSOR_SKIP handling in src/models/qwen35moe.cpp that keeps block 40 off the card until you ask for it — arrived in (b10211, b10212]. This page leans on it for the weights row of its budget.

Both brackets were bisected by the RTX 3090 and RTX 5090 pages and are quoted here rather than redone. In practice: use b10630 or newer, the build the lead quant was produced with. Every source line cited on this page was read at b10690 (bdf3955159d7184f44b76091973eeff532890a35, published 2026-08-30). A line number is only meaningful together with the tree it was read at, so if you check one, check it there.

Requirements

Installation

1. Get a llama.cpp build that knows the architecture

Take a CUDA release binary at b10630 or newer from ggml-org/llama.cpp releases, or build from source. Nothing model-specific is needed — no pip install flash-attn, no separate kernel package; llama.cpp's FlashAttention lives inside its own CUDA backend.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10690
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

2. Download the Q4_K_M GGUF

The lead quant is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf — 21,864,082,336 bytes (21.86 GB) per the HF tree API, re-checked on 2026-08-30, imatrix-calibrated, produced on llama.cpp b10630.

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

Download the single file by name on purpose: the repo also holds two mmproj projectors, and llama-server's -hf shortcut resolves one automatically and places it in VRAM.

Q4_K_M is also what llama.cpp would pick for you. With no :tag on a -hf argument, find_best_model in common/download.cpp tries Q4_K_M first and Q8_0 second; on this card Q8_0 is 35.217 GiB and does not fit, so the first match is also the only one. The publisher's own card calls Q4_K_M "Good quality, default size for most use cases" — but it is not the only tier this card holds, and the section after Running works out exactly which ones it does.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Four of those flags are load-bearing and none of them is a default. Each was re-read in the b10690 source rather than inherited from a sibling page:

• --no-mmproj — no_mmproj defaults to false and mmproj_use_gpu to true in common/common.h (lines 596 and 594), so a bare -hf …-GGUF fetches a sibling projector and puts it on the GPU. The lead repo ships two, at 899,283,488 and 902,822,432 bytes (0.838 and 0.841 GiB). With the explicit -m path above the flag is belt-and-braces; the moment you switch to -hf it is the difference between loading most of a gigabyte of vision weights you never use and not.
• -np 1 — for the server example common/arg.cpp:1400 sets n_parallel to -1 ("auto by default"), and tools/server/server.cpp:152-155 turns that into 4 slots with a unified KV pool — its own trace line reads n_parallel is set to auto, using n_parallel = 4 and kv_unified = true. The attention cache is shared across those slots, but the Gated-DeltaNet recurrent state is per sequence, so four slots take it from 0.061 GiB to 0.245 GiB. One slot is the right setting for a single-user agent loop, and it is the one this budget assumes.
• -c 131072 — the GGUF declares context_length 262144. Half of it is what fits with real room to spare; the ladder below shows the rest of the curve, including where it stops being comfortable.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 in ggml/src/ggml-common.h is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element rather than 16. It takes the 131,072-token cache from 2.500 GiB to 1.328 GiB, and that 1.172 GiB is most of what pays for the context.

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05). use_jinja defaults to true in common/common.h, so the template embedded in the GGUF renders tool schemas passed through the API's tools= parameter, with reasoning in <think>…</think>.

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

The VRAM budget

Every figure here is derived from the artifact's own tensor table and from llama.cpp's allocation code at b10690. Nothing was measured on hardware.

Each GiB figure is rounded from its own byte count and the total is rounded once from the byte sum, so the GiB column adds to 21.297 while the total reads 21.298. The bytes are the arithmetic; the GiB are a convenience.

None of these terms includes llama.cpp's compute buffers or the CUDA context. That is the single most important caveat on the page, and it is why the remainder column below is a budget rather than a promise.

How the three rows are obtained:

• Weights. The file is 21,864,082,336 bytes, but its tensor payloads sum to 21,853,090,304 and 476,956,672 of that is blk.40, the multi-token-prediction head. load_arch_tensors in src/models/qwen35moe.cpp creates every block-40 tensor with TENSOR_SKIP unless MTP is explicitly requested — llama_model_default_params() sets load_mtp to false — so a normal run loads 21,376,133,632 bytes. That is 19.908 GiB resident against 20.363 GiB on disk: the gap is 487,948,704 bytes, of which blk.40 is 476,956,672 and the last 10,992,032 — 10.5 MiB is the file's non-tensor content: GGUF header, metadata, the tensor-info table and alignment padding, which is exactly the 21,864,082,336 − 21,853,090,304 above. They are not interchangeable. The storage row in Requirements is the disk figure.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so the trunk is 40 layers. full_attention_interval is 4, which puts full attention on blocks 3, 7, 11 … 39 — ten of forty, verified against the tensor list. Each carries head_count_kv 2 at key_length 256 and value_length 256, i.e. 1,024 cache elements per token per layer and 10,240 across the ten: 20,480 bytes per token at f16, 10,880 at q8_0.
• Recurrent state. src/llama-hparams.cpp sizes the other thirty layers' two F32 tensors as (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements — 2,195,456 bytes per layer per sequence row, 65,863,680 bytes (62.8 MiB) across all thirty at one slot. llama-server prints it at startup as RS buffer size. Remember that number; it comes back in the host-RAM section, where it is not the answer.

What the last 2.702 GiB can buy

This is the part of the page that is actually about the card, so it is worth being explicit about the question. The RTX 4090 has the same 24 GB as the RTX 3090 and 8 GB less than the RTX 5090, and our 5090 page spends that difference on one quant tier up and the model's full 262,144-token window. The obvious question for a 4090 owner is whether the headroom this card does not have can be bought some other way. Below is the arithmetic, all of it capacity-driven and therefore true of any 24 GB card, this one included.

A whole quant tier up: no, and not narrowly. Both Q5 rungs from the lead publisher were read out of their own GGUF headers on 2026-08-30, the same way the table above was:

Neither leaves room for a cache, let alone this one — a 131,072-token q8_0 KV cache is 1.328 GiB on its own, and the recurrent state another 0.061, before a single compute buffer. Our 5090 page reaches the same conclusion from the other side, and this is the arithmetic behind it.

A sub-tier up: yes, exactly one, and it is a good one. Q4_K_L is Q4_K_M with the embedding and output weights at Q8_0 — the publisher's card says so in as many words — its row for that file reads "Uses Q8_0 for embed and output weights" and marks the tier recommended, and reading both files' tensor tables shows precisely what that means here. They differ in exactly two tensors of 753, and the routed experts are byte-identical:

So Q4_K_L costs +377,446,400 bytes — 0.352 GiB — resident, and every one of those bytes goes into the two tensors every token passes through, while the 18.586 GiB of routed experts stays exactly as it was. In resident terms the weights row of the budget goes from 19.908 GiB to 20.260 GiB (21,753,580,032 bytes, same blk.40 skipped), which is what the right-hand column of the ladder below is built on. On a mixture-of-experts model where the routed experts of the loaded blocks are 19,503,513,600 of the 21,376,133,632 resident bytes — 91.2% — that is an unusually cheap place to spend headroom. (Both terms exclude blk.40; pairing the all-123-tensor total of 19,956,498,432, which includes it, against a resident figure that does not would be an arithmetic apples-to-oranges.)

The two ladders side by side, at -np 1 and q8_0 KV, against the card's 24 GiB:

A longer context: partly. 196,608 tokens on Q4_K_M still leaves 2.038 GiB, which is the same order of slack as the recommended row. The full 262,144 is arithmetically reachable at 1.374 GiB and this page still does not recommend it, for a concrete reason rather than a cautious one: the table excludes compute buffers and the CUDA context entirely, and llama.cpp's auto-fit is on by default — fit_params = true at common/common.h:476, with fit_params_target initialised to 1024 * 1024*1024 bytes per device at line 481. At 1.374 GiB spare you are inside a rounding error of the margin the fitter itself tries to hold back, and its response when it cannot is to start moving expert tensors to system RAM — which shows up as a collapse in generation speed rather than as an error. The Q4_K_L row at 262,144 is 1.023 GiB, i.e. below that margin outright.

So the summary, in one line: on 24 GB you can buy a sub-tier of quant (Q4_K_L, -c 131072, 2.351 GiB spare) or half again as much context (Q4_K_M, -c 196608, 2.038 GiB spare), but not both, and not a Q5 anything. The page leads with Q4_K_M at 131,072 because it is the configuration the published RTX 3090 page documents, so the two 24 GB pages agree and a reader can compare them directly; if you want the extra precision on the embedding and output head, swap the filename in the download and the run command and take the 2.351 GiB row.

One thing this section deliberately does not claim. Whether the larger output.weight costs measurable decode throughput — Q8_0's 540 MB against Q6_K's 417 MB, read once per token — is a bandwidth question, not a capacity one. It does not transfer between cards, nobody has measured it for this model on any card, and this page will not guess at it.

Results

• Speed: omitted, and not because nobody has measured the model. Throughput figures for Apodex 1.1 mini exist; not one of them is on NVIDIA hardware, and none is on a stock llama.cpp GGUF:

  • ghazni101/Apodex-1.1-mini-MQ4RP reports a 400-token "generation measured at 226 tok/s end-to-end for MQ4R" on an RX 7900 XTX. That is a .mq4r file — not a GGUF — served by hipfire, a separate Rust engine that requires an AMD RDNA GPU.
  • kingjones777 publishes a four-row table — 63.30, 45.05, 32.54 tok/s and one more — all "(128-token generation, greedy)" on a Ryzen AI MAX+ 395, a unified-memory APU rather than a discrete card, and all in ROCmFP4/ROCmFPX tensor formats whose own card is headed "Stock llama.cpp will not load this file". Treat that publisher's STRIX_LEAN row with particular care: the shared variants block prints 64.87 for it while the STRIX_LEAN card's own "Speed — full offload" table shows a dash and says the figure is still being measured. One publisher, two answers about one number.

  So a figure exists for the model, and none exists for this file, on stock llama.cpp, on any NVIDIA card. The space I searched on 2026-08-30: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini (21 fetched and read in full; the 22nd, aykutx21/Apodex-1.1-mini-GGUF, has no README at all — its raw URL is a 404 — and no weights), the llama.cpp issue tracker (a repository-scoped search for apodex returns 0 against a control of 271 for qwen35moe, so the query works and the answer is a real zero), and this site's own catalogue — /check/apodex-1-1-mini/rtx-4090 has no benchmark for this pair. Reddit and LocalScore were not searched and are not inside that space. If you run this, please contribute the numbers — a llama-bench line and the server's own memory breakdown would make this the first measured page for the model on any NVIDIA card.

• VRAM usage: 21.298 GiB derived working set at 131,072 tokens with q8_0 KV, leaving 2.702 GiB of the card before compute buffers. Dropping to -c 32768 brings it to 20.301 GiB; the full ladder is in the section above. See /check/apodex-1-1-mini/rtx-4090 for live data as it lands.

• Quality notes: the vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. Q4_K_M is a 4-bit K-quant of a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated against a corpus rendered through this model's own chat template, published in the repo alongside the imatrix itself, which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-4090.

Checking the budget on your own card

Because the table above stops short of the compute buffers, do not treat it as the last word — llama.cpp will tell you the rest, twice, for free.

Before you commit, ask the fitter what it projects:

./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    -c 131072 -ctk q8_0 -ctv q8_0 -np 1

tools/fit-params prints the CLI arguments the auto-fitter would apply. If it comes back having reduced your context, that is the answer for your machine — take the -c 65536 row and its 3.367 GiB instead.

Afterwards, llama-server prints a memory breakdown [MiB] table with columns total / free / self / model / context / compute / unaccounted. The model and context columns are what this page derives; compute and unaccounted are exactly the two terms it does not carry. If you run this configuration, those two numbers are the single most useful thing you could send to /contribute.

Troubleshooting

The server's host RAM climbs by a hundred-odd MiB per request

It is not a leak, the ceiling is higher than you would guess, and it is invisible in nvidia-smi because it is system RAM rather than VRAM.

llama-server keeps a host-side prompt cache. cache_prompt is true at common/common.h:627 and the cache is bounded by cache_ram_mib, which is 8192 at line 632 of the same file; the flag is -cram / --cache-ram at common/arg.cpp:1713, whose help string reads "set the maximum cache size in MiB (default: %d, -1 - no limit, 0 - disable)" with the default interpolated from that same variable. For an ordinary attention-only model each cache entry is a few MiB and the default is invisible. On a hybrid like this one an entry has to carry recurrent state as well as KV, so entries are two orders of magnitude larger and the cache reaches its bound in tens of requests rather than thousands. Eight gibibytes of system RAM is what "default" means here.

This was chased down in ggml-org/llama.cpp#27894, and the story is worth reading rather than summarising, because the report and its resolution say different things. It was filed as a leak — "Hybrid SSM/attention models (qwen35moe) leak a fixed ~126 MiB RSS per request" — with forty consecutive requests taking RSS from 1,340 to 6,379 MiB in perfectly linear +126 MiB steps, and a controlled matrix that varied build, MTP, context size, vision, generated tokens, KV cache type and flash attention one at a time and moved the number by less than a MiB each. A non-SSM control (gpt-oss-20b mxfp4) on the same binary grew +0 MiB. A llama.cpp contributor then pointed at --cache-ram, and the reporter retracted: "Confirmed — you're right, this isn't a leak." The decisive run is --cache-ram 512 with nothing else changed — +126 MiB per request until it reaches +507 MiB against the 512 MiB bound, then flat for ten consecutive requests. The issue closed as completed on 2026-08-28, by its own author.

Two scoping notes, because this is someone else's measurement. It was taken on an RX 7900 XTX under ROCm, and on an Ornith-1.5-35B-A3B build rather than on Apodex — but the reporter also rebuilt the same source with -DGGML_VULKAN=ON and measured +126.0 MiB/request against ROCm's +126.3, concluding it "is not ROCm/HIP-specific" and that the allocation lives in the model/request layer rather than in any backend. That is what makes it worth putting on a CUDA page. And do not read 126 MiB as one recurrent state: this model's is 65,863,680 bytes, 62.8 MiB, so an entry costs roughly twice that and nobody has instrumented the difference. The 126 MiB is measured, on a different checkpoint; its composition is not established, on any.

Practically: if the box has 16 GB and you are also holding a 21.86 GB file in page cache, set the flag deliberately. --cache-ram 0 disables the cache; a few hundred MiB keeps a useful prefix cache without letting it grow into memory something else needs.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and mainline llama.cpp implements it: src/models/qwen35moe.cpp builds a dedicated MTP graph, enabled with --spec-type draft-mtp (the exact spelling matters — mtp alone is rejected), with no draft model needed because the server drafts against the weights already loaded. The publisher confirms the block survives the conversion: "the MTP layers are stored at Q4_0 in the imatrix quants (except for the Q8_0 quant), since imatrix calibration does not exercise them" — which the tensor table bears out, every one of blk.40's quantised tensors being Q4_0.

The VRAM price is three separate things. The block-40 tensors stop being skipped (+0.444 GiB); the draft depth multiplies the recurrent-state rows; and a second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk, which steers only the main cache (it has its own flag, -ctkd). Against 2.702 GiB of headroom that is most of the budget, before compute buffers — so if you want to try it, drop to -c 32768 first and start from the 3.699 GiB row.

And there is a caution that lands squarely on this optional flag, which is why this section is longer here than on our other pages. ggml-org/llama.cpp#26558 is a hard cublasSgemm_v2 → CUDA_ERROR_INVALID_VALUE → GGML_ABORT crash under --spec-type draft-mtp, reported on an NVIDIA GeForce RTX 4090 (24 GiB) with CMAKE_CUDA_ARCHITECTURES=89. It reproduced in about 21 minutes of soak under KV-cache saturation — the 51- and 57-minute figures in that thread are runs with an experimental mitigation already applied, not the baseline — and the failing op was deterministic: blk.0.ssm_alpha.weight × attn_norm-0, a linear-attention projection of the kind this model has thirty of. The reporter instrumented it down to a cuBLAS handle holding a host-heap pointer where a CUDA stream should be, and found that GGML_CUDA_DISABLE_GRAPHS=1 both kept the server alive through the soak and flattened a separate ~40–50 MiB/min growth in GPU memory.

It is not an RTX 4090 fact, and that changes what you should take from it. A second reporter confirmed the same signature independently from a "different setup (dual RTX 3090, NVLink, Linux, CUDA 12.x, driver 580)" — cublasGemmEx returning CUBLAS_STATUS_INVALID_VALUE, always on device 1, on Qwen3.6-27B-MTP, reproduced across three builds spanning June to August, with --split-mode tensor accelerating time-to-failure. Their mitigation matrix reaches the first reporter's conclusion by a different route: stock 53–69 seconds to failure, LLAMA_GRAPH_REUSE_DISABLE=1 about seven minutes, GGML_CUDA_DISABLE_GRAPHS=1 no failure — 15 minutes clean, then 50 minutes sustained clean. So read this as an architecture-plus-MTP hazard observed on at least two NVIDIA configurations, carrying one mitigation that two independent parties arrived at, rather than as a property of this card.

Read its status precisely, because "closed" is doing less work here than it looks. The issue is closed as completed — but no fix was merged. The PR opened against it, #27368, is closed unmerged; the related #26827 is still open. What actually retired it was a retest. A collaborator asked for one — "please run your repro on latest master and see if it re-occurs (#26574 has been merged)" — and the reporter came back with "Interesting, now it seems not crash any more", after which the collaborator wrote "Closing as it no longer repros apparently". The variable that changed was the build, not the machine: the driver and CUDA versions quoted in that reply, 595.84 and 13.2, are the same pair recorded in the original report's own hardware section. #26574, "ggml-cuda: provide static workspace for cuBLAS handles", was merged, on 2026-08-20, and its merge commit is an ancestor of b10630 — so any build this page recommends already carries it. Its standing is worth stating exactly: it addresses cuBLAS workspace handling rather than the reported abort, and nobody has claimed it fixes that abort — but the retest that closed the issue was requested on the strength of that merge, so it is not unrelated either. Nor is the report this recipe's configuration: it was filed at -np 4 --kv-unified with the context deliberately saturated, and the command above runs one slot. And it is a different checkpoint — a 0.8B qwen35 build, not this 36B qwen35moe one — sharing the architecture family and the Gated-DeltaNet layer the crash landed on.

So: the flag has a known VRAM price, an unmeasured throughput return on this model, and two independent NVIDIA reports of one crash signature — one of them on this exact card — retired without a merged fix. If you enable it, run it under load before you rely on it; GGML_CUDA_DISABLE_GRAPHS=1 is the first thing to reach for, being the one mitigation both reporters found to work outright rather than merely to delay. Then tell us what happened — a clean MTP soak on a 4090 would be a genuinely new datapoint.

The context fits — but check that decode still moves at depth

The 131,072-token budget is an arithmetic statement about VRAM. It is not a statement about how the model behaves at that depth, and this architecture family has two open reports a long-context reader should know about.

ggml-org/llama.cpp#27623 reports that on CUDA, with the model fully resident and plenty of free VRAM, decode throughput on a hybrid Gated-DeltaNet model collapses roughly 25× once the KV position passes about 80K, while prompt processing at the same positions stays fast. #27756 reports the same family emitting an immediate end-of-sequence token past roughly 130K positions, on CUDA and on CPU alike.

Both are filed against Qwen3.8-27B, a 64-layer hybrid with 48 Gated-DeltaNet layers to 16 full-attention ones. Apodex is a 40-layer hybrid with 30 Gated-DeltaNet layers, and that difference is what the second report points at: its author records that the same 243K-token needle test passes on Qwen3.5-35B-A3B — the 30-layer shape, and this model's own parent checkpoint — on the same build and machine, and concludes the effect scales with linear-layer depth rather than being a backend bug. Encouraging for this recipe, and still someone else's measurement of someone else's checkpoint.

So run a long prompt before you rely on one. Time the decode rate a few thousand tokens in and again near your working depth; if it falls off a cliff rather than sloping, that is worth reporting upstream — and to /contribute, so this page can say something firmer than "check".

Do not pass --mmproj with this architecture yet

ggml-org/llama.cpp#25717 documents a CUDA illegal-memory-access crash on vision requests for Qwen3.5-35B-A3B plus its mmproj, traced in the thread to M-RoPE 3D position ids and a find_slot: non-consecutive token position that the image token batch produces. Every row of the reporter's crash matrix sends an image, and nothing in the thread reproduces it on a text-only request, so it does not bear on the configuration this page documents — which is the reason --no-mmproj is in the run command rather than the flag merely being absent.

Its status changed on the day this page was written and the change is easy to misread. The issue was closed as not_planned at 2026-08-30T01:13:13Z, carrying the label stale. That is not a fix. Closed-as-stale means nobody worked on it, so the caution stands exactly as it did while the issue was open; only the word "open" would now be wrong. Anything you read about this issue that was written before 2026-08-30 will call it open, including this site's own earlier pages for this model — that dates the text rather than changing the advice. If you check this yourself, use an authenticated GitHub client — though not for the reason you might assume. An anonymous api.github.com request answers correctly until it exhausts the 60-per-hour quota; past that it returns a two-key body of message and documentation_url with no state field at all, so anything reading state gets null and prints what looks exactly like a clean answer. The tell is that message is present, not that a token is absent.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE in src/llama-arch.cpp, which arrived in (b7989, b7990] — so this error means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

It fits on paper but llama.cpp quietly gives me less

llama.cpp's auto-fit is on by default and will reduce unset parameters to leave a 1024 MiB margin per device, down to a floor of fit_params_min_ctx = 4096 tokens. Because the command above sets -c explicitly, auto-fit will not shrink your context — it will move layers to the CPU instead, which shows up as a collapse in generation speed rather than as an error. If that happens, check the load log for offloaded layers, then either drop a rung on the ladder or turn the behaviour off with -fit off and let the allocation fail loudly instead.

One caveat on -fit itself, since this page points you at the fitter twice: ggml-org/llama.cpp#27171 — a throughput regression reported against a different 35B-A3B model at Q4_K_M when --fit-target is used — is still open, last updated 2026-08-16 (re-checked 2026-08-30). Use llama-fit-params as a projection tool, as above; treat --fit-target as something to measure rather than to set and forget.

The server downloads an extra file you did not ask for

If you launch with -hf bartowski/apodex_Apodex-1.1-mini-GGUF instead of a local -m path, llama.cpp resolves a projector sibling as well — find_best_mmproj in common/download.cpp matches any sibling filename containing mmproj, and this repo ships two. Pass --no-mmproj to stop it, or use an explicit -m path as above.

Choosing a different quant publisher

Conversions of this model are still appearing. A Hub enumeration on 2026-08-30 returned 22 repositories matching Apodex-1.1-mini across 11 namespaces — one more repository and one more namespace than the same query returned two days earlier, the newcomer being stefanprodan/Apodex-1.1-mini-oQ4e-mtp, an MLX build for Apple silicon rather than this card. Treat the paragraph below as a snapshot and re-enumerate before you decide.

The GGUF ladders are not interchangeable, and the Q4_K_M tier alone spans 1.577 GiB across publishers:

The first two both ship an imatrix and either is a reasonable lead. The last is worth a specific warning: 21.800 GiB is 1.437 GiB above bartowski's for the nominally same tier, which on this card is the difference between a comfortable 128K context and a tight one — and its Q3_K_S, Q3_K_M and Q3_K_L are all exactly 19,095,585,248 bytes, an identical byte count that three different K-quant tiers of one model do not produce. The enumeration also turns up MLX builds (nicolasembleton/…-MLX-{4,5,6,8}bit, abenzerps/…-MLX, stefanprodan/…-oQ4e-mtp — Apple silicon, not this card), four kingjones777 repacks in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp can open, one ghazni101 build in the .mq4r container of a different engine entirely, and one repo (aykutx21) with no README and no weights in it at all. Check the byte count and the file list before you download, whichever publisher you pick.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, but they are vLLM/SGLang safetensors rather than GGUF, and llama.cpp cannot load them at all.

They are also not ruled out for the reason you might expect. Their weights do fit: 24,210,520,880 bytes (22.548 GiB) for NVFP4 and 24,651,300,904 bytes (22.958 GiB) for GPTQ-Int4, against 24 GiB of card. What rules them out is what is left afterwards — 1.452 GiB and 1.042 GiB respectively, before a single token of KV cache, the 0.061 GiB recurrent state or any compute buffer. The lead GGUF's 2.702 GiB of headroom is what buys the 131,072-token context; these leave less than the auto-fitter's own default margin.

And the -NVFP4 repo is not a 4-bit checkpoint. Its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4, with kv_cache_quant_algo: FP8 and the MTP block in exclude_modules. HuggingFace tags the repo 8-bit, not 4-bit, and the size is why. A repo name is a label; hf_quant_config.json is the fact. Their own Quick Starts launch on a single GPU — --tp-size 1 on both -NVFP4 and -GPTQ-Int4, --tp 1 on -FP8; the --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex, /apodex-1.1 and /apodex1.1 all return 404, the corresponding registry.ollama.ai manifests return 404, and an Ollama search for apodex returns one unrelated card. Use llama.cpp directly, or point LM Studio at the same GGUF — it runs llama.cpp underneath, so the artifact and the fit arithmetic above are identical. If a tag appears later, please tell us.