Apodex 1.1 mini on RTX 3090 Ti: 128K Agent Context in Q4_K_M, and the CUDA-Graph Caveat

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single RTX 3090 Ti, from a Q4_K_M GGUF, at a 131,072-token working context. The vendor's own launch commands are eight-GPU tensor-parallel servers; this is the one-card version.

A 35B model gets a six-figure context on a 24 GB card because of the architecture, not a trick. Apodex inherits Qwen3.5's hybrid attention: of its 40 transformer blocks only 10 are full-attention, and the other 30 are Gated-DeltaNet linear-attention layers carrying a fixed-size recurrent state instead of a growing KV cache. The KV cache therefore costs 20 KiB per token rather than 80 — so 131,072 tokens is 1.328 GiB at q8_0, and it fits beside the weights.

Hardware data: RTX 3090 Ti (24 GB VRAM) · 21.298 GiB derived working set at a 131,072-token context · See benchmark data

⚠️ The capacity arithmetic on this page is identical to our RTX 3090 page, and that is the finding rather than an oversight.

Both cards hold 24 GB. The working set is decided by the artifact and by the context length, and neither is a function of how fast the silicon is — so every byte in the budget below matches what our RTX 3090 page derives, deliberately. Throughout, conclusions are labelled: a capacity conclusion is arithmetic over bytes and transfers to any 24 GB card, and a bandwidth conclusion does not transfer.

The one axis on which a 3090 Ti could differ from a 3090 here is memory bandwidth, and this page asserts nothing about it. NVIDIA's own RTX 3090 / 3090 Ti product page publishes Memory Size (24 GB for both) and Memory Type (GDDR6X for both) and carries no memory-bandwidth row at all — checked 2026-08-30, and contrast the 40/50-series comparison page, which does publish one. Nobody has measured this model on either card in a citable runtime (see Results). A tokens-per-second claim would therefore be an inference from a spec figure the vendor does not publish, applied to a workload nobody has run. It is not made here.

What this page does add over the 3090 one, all of it verified today rather than inherited: the two CUDA-backend behaviours that are decided by your card's compute capability 8.6 rather than by its capacity (Troubleshooting), the llama-server host-RAM default that neither published 24 GB page carried before today, the corrected status of the upstream vision crash report, and the exact quant sub-tier the leftover headroom can buy.

ℹ️ This recipe is text-only, deliberately. The checkpoint does contain a vision tower (333 model.visual.* tensors in its safetensors index) and the lead GGUF repo ships two projectors for it. But the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions at all — no --mmproj flag, no vision example, and the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. Every number below is for text generation, and the run command refuses the projector rather than merely omitting it. See Troubleshooting for the upstream vision crash report and its current, changed status.

⚠️ The build floor on this page is about speed as well as about loading, which is unusual and is the reason it is stated twice.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990, so it arrived in (b7989, b7990]. That interval is bracketed by adjacent published tags, which is as fine as a tag-granularity probe can resolve — read the endpoints as the limit of the method, not as a commit. Below it, the model does not load at all.
• The qwen35moe-specific MTP wiring — the load_mtp / TENSOR_SKIP handling in src/models/qwen35moe.cpp that keeps block 40 off the card until you ask for it — arrived in (b10211, b10212]. This page leans on it for the weights row of its budget.
• A throughput regression in this architecture's own linear-attention kernels sat on mainline between early June and early August 2026, and it is the subject of its own Troubleshooting section below. It affects what the model costs you per token, not whether it loads, so no error message will tell you about it.

Both brackets were bisected by the RTX 3090 and RTX 5090 pages and are quoted here rather than redone. In practice: use b10630 or newer, the build the lead quant was produced with. Every source line cited on this page was read at b10701 (cc231cb0da565440cf6a3e5b55dfeba477972cb6, published 2026-08-30T18:56:31Z). A line number is only meaningful together with the tree it was read at, so if you check one, check it there — llama.cpp publishes several tags a day and the offsets move.

Requirements

Installation

1. Get a llama.cpp build that knows the architecture

Take a CUDA release binary at b10630 or newer from ggml-org/llama.cpp releases, or build from source. Nothing model-specific is needed — no separate kernel package, no pip install flash-attn; llama.cpp's FlashAttention lives inside its own CUDA backend.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10701
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

If you build from source rather than taking a release binary, CMAKE_CUDA_ARCHITECTURES=86 is this card's target — compute capability 8.6 per NVIDIA's own CUDA GPUs table, which lists the GeForce RTX 3090 Ti and the RTX 3090 together in the 8.6 group. That number is not decoration on this page; two of the Troubleshooting sections below turn on it.

2. Download the Q4_K_M GGUF

The lead quant is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf — 21,864,082,336 bytes (21.86 GB) per the HF tree API, re-checked 2026-08-30, imatrix-calibrated, produced on llama.cpp b10630.

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

Download the single file by name on purpose: the repo also holds two mmproj projectors, and llama-server's -hf shortcut resolves one automatically and places it in VRAM.

Q4_K_M is also what llama.cpp would pick for you. With no :tag on a -hf argument, find_best_model in common/download.cpp tries the tag list {"Q4_K_M", "Q8_0"} in that order; on this card Q8_0 is 35.217 GiB and does not fit, so the first match is also the only one. The publisher's card calls Q4_K_M "Good quality, default size for most use cases" — it is not the only tier this card holds, and the section after Running works out exactly which ones it does.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Four of those flags are load-bearing and none of them is a default. Each was re-read in the b10701 source rather than inherited from a sibling page:

• --no-mmproj — no_mmproj defaults to false and mmproj_use_gpu to true in common/common.h (lines 596 and 594), so a bare -hf …-GGUF fetches a sibling projector and puts it on the GPU. The lead repo ships two, at 899,283,488 and 902,822,432 bytes (0.838 and 0.841 GiB). With the explicit -m path above the flag is belt-and-braces; the moment you switch to -hf it is the difference between loading most of a gigabyte of vision weights you never use and not.
• -np 1 — for the server example common/arg.cpp:1400 sets n_parallel to -1 (// auto by default), and tools/server/server.cpp:152-155 turns that into 4 slots with a unified KV pool — its own trace line reads n_parallel is set to auto, using n_parallel = 4 and kv_unified = true. The attention cache is shared across those slots, but the Gated-DeltaNet recurrent state is per sequence, so four slots take it from 0.061 GiB to 0.245 GiB. One slot is the right setting for a single-user agent loop, and it is the one this budget assumes.
• -c 131072 — the GGUF declares context_length 262144. Half of it is what fits with real room to spare; the ladder below shows the rest of the curve, including where it stops being comfortable.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 in ggml/src/ggml-common.h is a 2-byte scale plus 32 one-byte quants — 34 bytes per 32 elements, i.e. 8.5 bits per element rather than 16. It takes the 131,072-token cache from 2.500 GiB to 1.328 GiB, and that 1.172 GiB is most of what pays for the context.

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05). use_jinja defaults to true at common/common.h:638, so the template embedded in the GGUF renders tool schemas passed through the API's tools= parameter, with reasoning in <think>…</think>.

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

The VRAM budget

Every figure here is derived from the artifact's own tensor table, read out of the GGUF header on 2026-08-30, and from llama.cpp's allocation code at b10701. Nothing was measured on hardware. This is a capacity budget: it holds on any 24 GB card.

Each GiB figure is rounded from its own byte count and the total is rounded once from the byte sum, so the GiB column adds to 21.297 while the total reads 21.298. The bytes are the arithmetic; the GiB are a convenience.

None of these terms includes llama.cpp's compute buffers or the CUDA context. That is the single most important caveat on the page, and it is why the remainder column below is a budget rather than a promise.

How the three rows are obtained:

• Weights. The file is 21,864,082,336 bytes, but its 753 tensor payloads sum to 21,853,090,304, and 476,956,672 of that is blk.40, the multi-token-prediction head. load_arch_tensors in src/models/qwen35moe.cpp creates every block-40 tensor with TENSOR_SKIP unless MTP is explicitly requested — line 45 of that file reads int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0;, and llama_model_default_params() leaves load_mtp false — so a normal run loads 21,376,133,632 bytes. That is 19.908 GiB resident against 20.363 GiB on disk, and the gap is worth doing properly, because a reader who subtracts those two figures finds a remainder rather than the MTP block. The file is 487,948,704 bytes larger than the resident set. 476,956,672 of that is blk.40; the last 10,992,032 bytes — 10.5 MiB — are the part of the file that is not tensor payload at all, and it is exactly the 21,864,082,336 − 21,853,090,304 above: the GGUF header, its key-value metadata, the tensor-info table and the alignment padding between tensors. Those bytes are read at load and are never weights. So the disk and resident figures are not interchangeable, and the storage row in Requirements is the disk one.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so the trunk is 40 layers. full_attention_interval is 4, which puts full attention on blocks 3, 7, 11 … 39 — ten of forty, verified against the tensor list. Each carries head_count_kv 2 at key_length 256 and value_length 256, i.e. 1,024 cache elements per token per layer and 10,240 across the ten: 20,480 bytes per token at f16, 10,880 at q8_0.
• Recurrent state. src/llama-hparams.cpp sizes the other thirty layers' two F32 tensors as (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements — 2,195,456 bytes per layer per sequence row, 65,863,680 bytes (62.8 MiB) across all thirty at one slot. llama-server prints it at startup as RS buffer size. Remember that number; it comes back in the host-RAM section, where it is not the answer.

What the last 2.702 GiB can buy

The RTX 3090 Ti holds the same 24 GB as the RTX 3090 and 8 GB less than the RTX 5090, and our 5090 page spends that difference on one quant tier up and the model's full 262,144-token window. Everything below is capacity-driven and therefore true of any 24 GB card. All of it was re-derived from the GGUF headers today rather than carried across.

A whole quant tier up: no, and not narrowly. Both Q5 rungs from the lead publisher, read from the HF tree API on 2026-08-30:

Neither leaves room for a cache, let alone this one — a 131,072-token q8_0 KV cache is 1.328 GiB on its own, and the recurrent state another 0.061, before a single compute buffer.

A sub-tier up: yes, exactly one, and it is a good one. Q4_K_L is Q4_K_M with the embedding and output weights at Q8_0 — the publisher's own row for that file reads "Uses Q8_0 for embed and output weights" and marks the tier recommended. Reading both files' tensor tables shows precisely what that means here: they differ in exactly two tensors of 753, and every routed-expert tensor is byte-identical.

So Q4_K_L costs +377,446,400 bytes — 0.352 GiB — resident, and every one of those bytes goes into the two tensors every token passes through, while the 18.586 GiB of routed experts stays exactly as it was. The weights row of the budget goes from 19.908 GiB to 20.260 GiB (21,753,580,032 bytes, same blk.40 skipped). On a mixture-of-experts model where the routed experts of the loaded blocks are 19,503,513,600 of the 21,376,133,632 resident bytes — 91.2% — that is an unusually cheap place to spend headroom. (Both terms exclude blk.40. Pairing the all-123-tensor total, which includes it, against a resident figure that does not would be an apples-to-oranges ratio; it comes out at 93.4% and means nothing.)

The two ladders side by side, at -np 1 and q8_0 KV, against the card's 24 GiB:

A longer context: partly. 196,608 tokens on Q4_K_M still leaves 2.038 GiB, the same order of slack as the recommended row. The full 262,144 is arithmetically reachable at 1.374 GiB and this page still does not recommend it, for a concrete reason rather than a cautious one: the table excludes compute buffers and the CUDA context entirely, and llama.cpp's auto-fit is on by default — fit_params = true at common/common.h:476, with fit_params_target initialised to 1024 * 1024*1024 bytes per device at line 481. At 1.374 GiB spare you are inside a rounding error of the margin the fitter itself tries to hold back, and its response when it cannot reach it is to start moving expert tensors to system RAM — which shows up as a collapse in generation speed, not as an error. The Q4_K_L row at 262,144 is 1.023 GiB, i.e. below that margin outright.

So, in one line: on 24 GB you can buy a sub-tier of quant (Q4_K_L, -c 131072, 2.351 GiB spare) or half again as much context (Q4_K_M, -c 196608, 2.038 GiB spare), but not both, and not a Q5 anything. This page leads with Q4_K_M at 131,072 so that it and the RTX 3090 page document the same configuration and a reader can compare them directly; if you want the extra precision on the embedding and output head, swap the filename in the download and the run command and take the 2.351 GiB row.

One thing this section deliberately does not claim. Whether the larger output.weight costs measurable decode throughput — Q8_0's 540 MB against Q6_K's 417 MB, read once per token — is a bandwidth question, not a capacity one. It does not transfer between cards, nobody has measured it for this model on any card, and this page will not guess at it.

Results

• Speed: omitted, and not because nobody has measured the model. Throughput figures for Apodex 1.1 mini exist; not one is on NVIDIA hardware, and not one is on a stock llama.cpp GGUF:

  • ghazni101/Apodex-1.1-mini-MQ4RP reports a 400-token "generation measured at 226 tok/s end-to-end for MQ4R" on an RX 7900 XTX (gfx1100). That is a .mq4r file — not a GGUF — served by hipfire, a separate Rust engine that requires an AMD RDNA GPU. It is also worth reading carefully even on its own terms: the figure is for MQ4R, the repository hosts MQ4RP, and the card then predicts rather than measures the hosted build at "expect a similar ~10% RP decode penalty here."
  • kingjones777 publishes a four-row table of full-offload figures — 63.30, 45.05 and 32.54 tok/s among them, all "(128-token generation, greedy)" — on a Ryzen AI MAX+ 395, a unified-memory APU rather than a discrete card, and all in ROCmFP4/ROCmFPX tensor formats whose own card is headed "Stock llama.cpp will not load this file".

  So a figure exists for the model, and none exists for this file, on stock llama.cpp, on any NVIDIA card. The space searched on 2026-08-30: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini across 11 namespaces — 21 fetched and read, the 22nd (aykutx21/Apodex-1.1-mini-GGUF) returning HTTP 404 for its raw README and holding no weights — searched both for a tok/s-style unit and for table headers carrying the unit, since one of the two publishers above prints bare numbers under a tok/s (full GPU offload) column head; the llama.cpp issue tracker, where a repository-scoped search for apodex returns 0 against a control of 271 for qwen35moe, so the query works and the answer is a real zero; and this site's own catalogue, where /check/apodex-1-1-mini/rtx-3090-ti has no benchmark for this pair. Reddit and LocalScore were not searched and are not inside that space. If you run this, please contribute the numbers — a llama-bench line and the server's own memory breakdown would make this the first measured page for the model on any NVIDIA card.

• VRAM usage: 21.298 GiB derived working set at 131,072 tokens with q8_0 KV, leaving 2.702 GiB of the card before compute buffers. Dropping to -c 32768 brings it to 20.301 GiB; the full ladder is in the section above. See /check/apodex-1-1-mini/rtx-3090-ti for live data as it lands.

• Quality notes: the vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. Q4_K_M is a 4-bit K-quant of a model whose routed experts are already narrow (expert_feed_forward_length 512). The lead quant is imatrix-calibrated, with both the calibration corpus and the imatrix published in the repo alongside the ladder, which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier.

For the full benchmark data, see /check/apodex-1-1-mini/rtx-3090-ti.

Checking the budget on your own card

Because the table above stops short of the compute buffers, do not treat it as the last word — llama.cpp will tell you the rest, twice, for free.

Before you commit, ask the fitter what it projects:

./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    -c 131072 -ctk q8_0 -ctv q8_0 -np 1

tools/fit-params prints the CLI arguments the auto-fitter would apply. If it comes back having reduced your context, that is the answer for your machine — take the -c 65536 row and its 3.367 GiB instead.

Afterwards, llama-server prints a memory breakdown [MiB] table with columns total / free / self / model / context / compute / unaccounted. The model and context columns are what this page derives; compute and unaccounted are exactly the two terms it does not carry. If you run this configuration, those two numbers are the single most useful thing you could send to /contribute.

Troubleshooting

CUDA graphs are enabled on this card, and one upstream crash report depends on that

This is the first of two sections that turn on your card's compute capability rather than on its capacity, so it is worth stating the gate exactly. In a build with CUDA graph support compiled in — the whole mechanism sits behind a USE_CUDA_GRAPH guard, so a build without it has no graphs at all — llama.cpp then disables graph capture by device only for cards below Volta: ggml_cuda_graph_set_enabled in ggml/src/ggml-cuda/ggml-cuda.cu (line 4235) tests cc < GGML_CUDA_CC_VOLTA, and GGML_CUDA_CC_VOLTA is 700 at line 52 of ggml/src/ggml-cuda/common.cuh. The RTX 3090 Ti is compute capability 8.6 per NVIDIA's CUDA GPUs table, so on a stock CUDA release binary graph capture is on for you by default. That matters because the one open hazard in this architecture family with a working mitigation is a graph-capture bug, and the mitigation is to turn capture off — see the MTP section below.

A trap in the environment variable itself, and it is easy to get backwards. The switch is read as a presence test, not a value test: common.cuh line 1258 is

static const bool disable_cuda_graphs_due_to_env = (getenv("GGML_CUDA_DISABLE_GRAPHS") != nullptr);

so GGML_CUDA_DISABLE_GRAPHS=0 disables CUDA graphs exactly as thoroughly as =1 does. If you want graphs back on, unset the variable — do not set it to zero. It is also read once into a static, so it is sampled at first use and cannot be changed while the server is running.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and mainline llama.cpp implements it: src/models/qwen35moe.cpp builds a dedicated MTP graph, described in its own comment at line 116 as "MTP block looks like a full-attention Qwen3.5 decoder block with MoE FFN." You enable it with --spec-type draft-mtp — the exact spelling matters, mtp alone is rejected — and no draft model is needed, because the server drafts against the weights already loaded.

The VRAM price is three separate things. The block-40 tensors stop being skipped (+0.444 GiB); the draft depth multiplies the recurrent-state rows; and a second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk, which steers only the main cache (it has its own flag, -ctkd). Against 2.702 GiB of headroom that is most of the budget, before compute buffers — so if you want to try it, drop to -c 32768 first and start from the 3.699 GiB row.

And there is a caution that lands squarely on this optional flag, on cards with your compute capability. ggml-org/llama.cpp#26558 is a hard cublasSgemm_v2 → CUDA_ERROR_INVALID_VALUE → GGML_ABORT crash under --spec-type draft-mtp, first reported on an RTX 4090 built with CMAKE_CUDA_ARCHITECTURES=89. The reporter instrumented it down to a cuBLAS handle holding a host-heap pointer where a CUDA stream should be, traced the corruption to the per-context CUDA graph cache churning under the MTP draft context, and found GGML_CUDA_DISABLE_GRAPHS=1 both kept the server alive through the soak and flattened a separate growth in GPU memory. The failing op was blk.0.ssm_alpha.weight — a linear-attention projection, built by the same src/models/qwen35moe.cpp that implements this recipe's architecture, and this model has thirty such layers.

Why it is on this page rather than only on the 4090 one. It was independently confirmed on a second machine, and that machine's cards are the ones NVIDIA lists in the same 8.6 group as yours: "Another data point + independent confirmation, from a different setup (dual RTX 3090, NVLink, Linux, CUDA 12.x, driver 580)", with cublasGemmEx returning CUBLAS_STATUS_INVALID_VALUE, always on device 1, reproduced across three builds spanning June to August. That reporter's own summary is "Confirms this isn't specific to your 4090/single-GPU/tiny-model repro" — so read the signature as architecture-family-plus-MTP and not as a property of one card generation. Their mitigation matrix reaches the first reporter's conclusion by a different route: graph-reuse disable only delays failure, while GGML_CUDA_DISABLE_GRAPHS=1 gave "no failure" — 15 minutes clean, then 50 minutes sustained clean.

Now scope it down, because two things in that second report do not describe your machine. It is a dual-GPU rig running --split-mode tensor, which the reporter says accelerates time-to-failure and attributes to concurrent graph churn across devices — their repro "fires reliably in under 2 minutes with graphs enabled" against roughly 21 minutes of soak in the single-card 4090 case. A single RTX 3090 Ti has one device and no tensor split, so that accelerant is absent. And neither report is this model or this configuration: the first is a 0.8B qwen35 build, the second a Qwen3.6-27B, both run at -np 4 --kv-unified with the context deliberately saturated, where the command on this page runs one slot. What transfers is the mechanism and the compute capability, not the time-to-failure.

Read the issue's status precisely, because "closed" is doing less work here than it looks. It is closed as completed — but no fix was merged. The PR opened against it, #27368, is closed unmerged; the related #26827 is still open. What retired it was a retest after #26574 ("ggml-cuda: provide static workspace for cuBLAS handles", merged 2026-08-20, and an ancestor of b10630, so any build this page recommends already carries it) — the reporter came back saying it no longer reproduced, and a collaborator closed it on that basis. That PR addresses cuBLAS workspace handling rather than the reported abort, and nobody has claimed it fixes that abort; it is not unrelated either, since the retest was requested on the strength of it.

So: the flag has a known VRAM price, an unmeasured throughput return on this model, and two independent NVIDIA reports of one crash signature — one of them on cards with your compute capability — retired without a merged fix. If you enable it, run it under load before you rely on it; GGML_CUDA_DISABLE_GRAPHS=1 is the first thing to reach for, being the one mitigation both reporters found to work outright rather than merely to delay. Then tell us what happened — a clean MTP soak on a 3090 Ti would be a genuinely new datapoint.

It loads and runs, but is it running at the speed it should?

This is the second section that turns on your compute capability, and unlike the first it is about a defect that has already been fixed — which is exactly why it belongs next to a build floor rather than in a bug list.

ggml-org/llama.cpp#25162, "Performance regression on Turing GPUs after 9e58d4d69 — Qwen35 SSM kernels affected", reports a "24-42% performance regression" on an RTX 2080 Ti for Qwen3.5 and Qwen3.6 models. The measured, commit-by-commit bisection in it is on Qwen3.6-27B (27.20 tok/s before, 20.66 after). Its Qwen3.5-35B-A3B row — this model's parent checkpoint and the same qwen35moe architecture string, at Q5_K_XL rather than Q4_K_M — is weaker evidence and should be read as such: 46.12 tok/s at HEAD against an "~80 tok/s" figure the table itself labels a user report from about a month earlier, not a run the reporter made. The report is unusually well controlled for the question a reader has: "Gemma4 is NOT affected", because Gemma 4 has no SSM layers, and the fourteen CUDA files the offending commit touched include ssm-scan.cu and ssm-conv.cu — the kernels behind the thirty Gated-DeltaNet layers that are three-quarters of this model.

How far it reaches toward your card is a code question, and the code is unambiguous about the branch while saying nothing about the outcome. The reporter's root cause is that GGML_CUDA_RESTRICT had moved from function parameters to local variables, and that macro is defined in ggml/src/ggml-cuda/common.cuh (lines 1650–1654 at b10701) as empty only when PDL is in use and __CUDA_ARCH__ >= GGML_CUDA_CC_HOPPER, which the same file puts at 900 — and __restrict__ otherwise. Compute capability 8.6 is 860, below 900, so an Ampere build takes the same branch a Turing build does. That is a fact about which code compiles. Whether nvcc degraded its output the same way for sm_86 as for sm_75 is not established: the measurements in that thread are all on one 2080 Ti, and the closest thing to an Ampere claim is the reporter's own closing request — "Restore __restrict__ semantics on function parameters for non-Hopper architectures, or investigate alternative approaches that don't penalize Turing/Volta/Ampere performance." That is an inference by a reporter, not a measurement on an Ampere card, and this page does not upgrade it into one.

The reason it is still worth your attention is the timeline. The issue was closed on 2026-08-03 by its own author, after retesting on the same 2080 Ti and reporting "27.47 tok/s" — back to the pre-regression figure — at commit 0b14b87d7. Compare that commit against b10630, this page's floor: b10630 is 390 commits ahead of it and zero behind, so the retest commit is an ancestor of the floor and whatever restored the speed is already in every build this page asks you to use. So if you are on a binary from June or July that loads qwen35moe perfectly well, you may be paying a large fraction of your tokens per second for the privilege, silently, with no error and no warning. Update before you benchmark anything, and if you have a before-and-after on an Ampere card, that is a measurement nobody has and we would like it at /contribute.

The server's host RAM climbs by a hundred-odd MiB per request

It is not a leak, the ceiling is higher than you would guess, and it is invisible in nvidia-smi because it is system RAM rather than VRAM. Neither published 24 GB page carried this before today; that is an omission this page corrects.

llama-server keeps a host-side prompt cache. cache_prompt is true at common/common.h:627 and the cache is bounded by cache_ram_mib, which is 8192 at line 632 of the same file; the flag is -cram / --cache-ram at common/arg.cpp:1713, whose help string reads "set the maximum cache size in MiB (default: %d, -1 - no limit, 0 - disable)" with the default interpolated from that same variable. For an ordinary attention-only model each cache entry is a few MiB and the default is invisible. On a hybrid like this one an entry has to carry recurrent state as well as KV, so entries are two orders of magnitude larger and the cache reaches its bound in tens of requests rather than thousands. Eight gibibytes of system RAM is what "default" means here.

This was chased down in ggml-org/llama.cpp#27894, and the report and its resolution say different things, so it is worth reading rather than summarising. It was filed as a leak — its title still says so — with forty consecutive requests taking RSS from 1,340 to 6,379 MiB in linear steps of about 126 MiB, and a controlled matrix that varied build, MTP, context size, vision, generated tokens, KV cache type and flash attention one at a time and moved the number by less than a MiB each. A contributor then pointed at --cache-ram, and the reporter retracted: "Confirmed — you're right, this isn't a leak." The decisive run is --cache-ram 512 with nothing else changed — the same +126 MiB per request until it reaches the bound, then flat for ten consecutive requests. The issue closed as completed on 2026-08-28, by its own author.

Two scoping notes, because this is someone else's measurement. It was taken under ROCm and on an Ornith-1.5-35B-A3B build rather than on Apodex — but the reporter also rebuilt the same source with -DGGML_VULKAN=ON and measured +126.0 MiB per request against ROCm's +126.3, concluding the allocation lives in the model/request layer rather than in any backend. That is what makes it worth putting on a CUDA page. And do not read 126 MiB as one recurrent state: this model's is 65,863,680 bytes, 62.8 MiB, so an entry costs roughly twice that and nobody has instrumented the difference. The 126 MiB is measured, on a different checkpoint; its composition is not established, on any.

Practically: if the box has 16 GB and is also holding a 21.86 GB file in page cache, set the flag deliberately. --cache-ram 0 disables the cache; a few hundred MiB keeps a useful prefix cache without letting it grow into memory something else needs.

The context fits — but check that decode still moves at depth

The 131,072-token budget is an arithmetic statement about VRAM. It is not a statement about how the model behaves at that depth, and this architecture family has two open reports a long-context reader should know about.

ggml-org/llama.cpp#27623 reports that on CUDA, with the model fully resident and plenty of free VRAM, decode throughput on a hybrid Gated-DeltaNet model collapses roughly 25× once the KV position passes about 80K, while prompt processing at the same positions stays fast. #27756 reports the same family emitting an immediate end-of-sequence token past roughly 130K positions, on CUDA and on CPU alike. Both were still open when this page was written.

Both are filed against Qwen3.8-27B, a 64-layer hybrid with 48 Gated-DeltaNet layers to 16 full-attention ones. Apodex is a 40-layer hybrid with 30 Gated-DeltaNet layers, and that difference is what the second report points at: its author records that the same 243K-token needle test passes on Qwen3.5-35B-A3B — the 30-layer shape, and this model's own parent checkpoint — on the same build and machine, and concludes the effect scales with linear-layer depth rather than being a backend bug. Encouraging for this recipe, and still someone else's measurement of someone else's checkpoint.

So run a long prompt before you rely on one. Time the decode rate a few thousand tokens in and again near your working depth; if it falls off a cliff rather than sloping, that is worth reporting upstream — and to /contribute, so this page can say something firmer than "check".

Do not pass --mmproj with this architecture yet

ggml-org/llama.cpp#25717 documents a CUDA illegal-memory-access crash on vision requests for Qwen3.5-35B-A3B plus its mmproj, traced in the thread to M-RoPE 3D position ids and a find_slot: non-consecutive token position that the image token batch produces. Every row of the reporter's crash matrix sends an image, and nothing in the thread reproduces it on a text-only request, so it does not bear on the configuration this page documents — which is the reason --no-mmproj is in the run command rather than the flag merely being absent.

Its status changed on the day this page was written and the change is easy to misread. The issue was closed as not_planned at 2026-08-30T01:13:13Z, carrying the label stale. That is not a fix. Closed-as-stale means nobody worked on it, so the caution stands exactly as it did while the issue was open; only the word "open" would now be wrong. Anything you read about this issue that was written before 2026-08-30 will call it open, including our own earlier pages for this model; that dates the text rather than changing the advice.

If you check this yourself, be aware of one way the check fails silently. An anonymous api.github.com request answers correctly until it exhausts the 60-per-hour quota; past that it returns a two-key body of message and documentation_url with no state field at all, so anything reading state gets a null and prints what looks exactly like a clean answer. The tell is that message is present, not that a token is absent — and the cheap guard is to ask about an issue you know is open in the same breath.

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE in src/llama-arch.cpp, which arrived in (b7989, b7990] — so this error means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer. Note that releases/latest on the GitHub API answers v0.3.0 rather than a bNNNN tag, because every bNNNN release is marked as a prerelease; list the releases instead of asking for the latest one.

It fits on paper but llama.cpp quietly gives me less

llama.cpp's auto-fit is on by default and will reduce unset parameters to leave a 1024 MiB margin per device, down to a floor of fit_params_min_ctx = 4096 tokens (common/common.h:478). Because the command above sets -c explicitly, auto-fit will not shrink your context — it will move layers to the CPU instead, which shows up as a collapse in generation speed rather than as an error. If that happens, check the load log for offloaded layers, then either drop a rung on the ladder or turn the behaviour off with -fit off and let the allocation fail loudly instead.

One caveat on -fit itself, since this page points you at the fitter twice: ggml-org/llama.cpp#27171 — a throughput regression reported against a different 35B-A3B model at Q4_K_M when --fit-target is used — is still open, last updated 2026-08-16 (re-checked 2026-08-30). Use llama-fit-params as a projection tool, as above; treat --fit-target as something to measure rather than to set and forget.

The server downloads an extra file you did not ask for

If you launch with -hf bartowski/apodex_Apodex-1.1-mini-GGUF instead of a local -m path, llama.cpp resolves a projector sibling as well — find_best_mmproj in common/download.cpp matches any sibling filename containing mmproj, and this repo ships two. Pass --no-mmproj to stop it, or use an explicit -m path as above.

Choosing a different quant publisher

Conversions of this model are still appearing. A Hub enumeration on 2026-08-30 returned 22 repositories matching Apodex-1.1-mini across 11 namespaces. Treat the paragraph below as a snapshot and re-enumerate before you decide — for a model this young, the repository list is the fastest-decaying fact on the page.

The GGUF ladders are not interchangeable, and the Q4_K_M tier alone spans 1.577 GiB across publishers:

The first two both ship an imatrix and either is a reasonable lead. The last is worth a specific warning: 21.800 GiB is 1.437 GiB above bartowski's for the nominally same tier, which on this card is the difference between a comfortable 128K context and a tight one — and its Q3_K_S, Q3_K_M and Q3_K_L are all exactly 19,095,585,248 bytes, an identical byte count that three different K-quant tiers of one model do not produce. The enumeration also turns up MLX builds for Apple silicon rather than this card (nicolasembleton/…-MLX-{4,5,6,8}bit, abenzerps/…-MLX, stefanprodan/…-oQ4e-mtp, felkf/…-oQ6e-fp16-mtp), four kingjones777 repacks in ROCmFP4/ROCmFPX tensor formats that only a fork of llama.cpp can open, one ghazni101 build in the .mq4r container of a different engine entirely, and one repo (aykutx21) with no README and no weights in it at all. Check the byte count and the file list before you download, whichever publisher you pick.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos, but they are vLLM/SGLang safetensors rather than GGUF, and llama.cpp cannot load them at all.

They are also not ruled out for the reason you might expect. Their weights do fit: 24,210,520,880 bytes (22.548 GiB) for NVFP4 and 24,651,300,904 bytes (22.958 GiB) for GPTQ-Int4, against 24 GiB of card. What rules them out is what is left afterwards — 1.452 GiB and 1.042 GiB respectively, before a single token of KV cache, the 0.061 GiB recurrent state or any compute buffer. The lead GGUF's 2.702 GiB of headroom is what buys the 131,072-token context; these leave less than the auto-fitter's own default margin.

And the -NVFP4 repo is not a 4-bit checkpoint. Its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4, with kv_cache_quant_algo: FP8 and the MTP block in exclude_modules. HuggingFace tags the repo 8-bit, not 4-bit, and the size is why. A repo name is a label; hf_quant_config.json is the fact. The vendor's Quick Start launches all of these at --tensor-parallel-size 8 — an eight-GPU server configuration, not a single-card one.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex, /apodex-1.1 and /apodex1.1 all return 404, the corresponding registry.ollama.ai manifests return 404, and an Ollama search for apodex returns one unrelated card. Use llama.cpp directly, or point LM Studio at the same GGUF — it runs llama.cpp underneath, so the artifact and the fit arithmetic above are identical. If a tag appears later, please tell us.