Apodex 1.1 mini on RX 7900 XTX: a 128K-Context Agent Server on llama.cpp-HIP

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single Radeon RX 7900 XTX, from a Q4_K_M GGUF, at a 131,072-token working context. The vendor's own launch commands are eight-GPU tensor-parallel servers; this is the one-card, one-consumer-Radeon version.

The reason a 35B model gets a six-figure context on a 24 GB card is the architecture, not a trick. Apodex inherits Qwen3.5's hybrid attention: of its 40 transformer blocks only 10 are full-attention and the other 30 are Gated-DeltaNet linear-attention layers that carry a fixed-size recurrent state instead of a growing KV cache. llama.cpp derives that split itself — src/models/qwen35moe.cpp marks block i recurrent when (i + 1) % full_attention_interval != 0, and this GGUF declares full_attention_interval 4, leaving blocks 3, 7, 11 … 39 on full attention. The KV cache therefore costs 20 KiB per token, not 80.

Hardware data: RX 7900 XTX (24 GB VRAM) · 21.298 GiB derived working set at a 131,072-token context · See benchmark data

⚠️ This is a ROCm recipe, not CUDA. RDNA3 has no FP8 and no FP4 hardware, no FlashAttention-2 prebuilt wheel, no ExLlamaV2 and no Marlin. Do not follow a guide that tells you to install a cu12x wheel, pip install flash-attn, or reach for an FP8 checkpoint in the hope of saving memory — on this card an FP8 tensor upcasts and costs more. The reliable path here is a GGUF served by llama.cpp compiled against HIP, which is what this page documents.

ℹ️ This recipe is text-only, deliberately. The upstream checkpoint does contain a vision tower and the GGUF repos ship a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions at all — the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. The two publishers who built this model specifically for AMD both dropped the vision half as well: ghazni101 states the vision tower and MTP head are "intentionally absent from these quants". Every number below is for text generation, and the run command refuses the projector rather than merely omitting --mmproj.

⚠️ Two version floors, about 2,200 builds apart. The architecture string in this file is qwen35moe, and LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990 — an interval bracketed by adjacent published tags, which is as fine as a tag-granularity probe can resolve. The qwen35moe-specific load_mtp / TENSOR_SKIP wiring that keeps block 40 off the card arrived in (b10211, b10212]. In practice, use b10630 or newer — the build the lead quant was produced with, whose publisher bartowski writes that "if this model's architecture is newly supported, you'll need that release or newer to run them". Every source reference on this page was read at b10690 (bdf3955159d7184f44b76091973eeff532890a35, published 2026-08-30), the current release at the time of writing.

Requirements

The RX 7900 XTX is an officially supported ROCm GPU. AMD's install-on-Linux system-requirements matrix lists it as RDNA3, LLVM target gfx1100, supported, alongside the RX 7900 XT and 7900 GRE — and the same table's footnote restricts those cards to the four OS releases above. Because gfx1100 is natively supported, you do not need HSA_OVERRIDE_GFX_VERSION; that variable is a masquerade for cards ROCm ships no kernels for.

Installation

1. Install the ROCm driver stack

ROCm is not bundled with llama.cpp — you install it once at the OS level. On Ubuntu 24.04 (noble), the amdgpu-install package registers AMD's repositories and everything else comes from apt:

# 1. register AMD's repositories
wget https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt install ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt update

# 2. kernel driver — the headers line is not optional, see below
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms
# reboot here to apply all settings

# 3. ROCm itself
sudo apt install python3-setuptools python3-wheel
sudo usermod -a -G render,video $LOGNAME   # Add the current user to the render and video groups
sudo apt install rocm
# reboot again to apply all settings

Do not drop the linux-headers line. amdgpu-dkms builds a kernel module out of source, so it needs the running kernel's headers at /lib/modules/$(uname -r)/build — and apt will not fetch them for you. In AMD's own package index for the repository the .deb above registers, amdgpu-dkms declares Depends: dkms (>= 1.95), libc-dev | libc6-dev, autoconf, automake, initramfs-tools, amdgpu-dkms-firmware and Recommends: dwarves; linux-headers appears nowhere in that index. Without it the DKMS build fails, amdgpu never loads, and every symptom lands much later — rocm-smi showing nothing, the HIP build finding no device, -ngl 99 silently running on the CPU. linux-modules-extra is likewise not guaranteed on server and cloud images, which is why AMD names it explicitly on the Ubuntu tabs.

Those lines are AMD's own, for Ubuntu 24.04, from the ROCm quick-start installation guide — the ROCm 7.14.0 production documentation set. Take the package path from that page rather than from here, because the URL carries the release number and it moves; 7.2.4 is what it shipped when this page was written. Ignore the "This page has moved!" banner at the top of it unless you actually want the ROCm 10.0 installer — it points at the first of the two traps below. Two traps if you go looking for a shortcut. AMD publishes several install doc sets at once and they are on different major versions: the top-level install/rocm.html is the ROCm 10.0 set, whose every mention of 7.2.4 is an instruction to uninstall it first, and the Radeon-and-Ryzen set covers releases only through 7.2.1. And the older amdgpu-install --usecase=graphics,rocm one-liner that most AMD guides still show is no longer this route: the same doc set's installation overview states that "The AMDGPU installer documentation has been removed to encourage the use of the package manager for ROCm installation. While the package manager is the recommended method, you can still install ROCm using the AMDGPU installer" So the one-liner is deprecated rather than dead — but it is undocumented, and the block above is the package-manager path AMD points you to instead. Confirm the card is visible with rocm-smi before going further.

2. Build llama.cpp with the HIP backend

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10690
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release -- -j 16

This is the llama.cpp HIP build recipe with GPU_TARGETS set to this card's LLVM target. Two things worth knowing about that build:

• There is a hard floor on the ROCm version. ggml/src/ggml-hip/CMakeLists.txt fails the configure step outright with "At least ROCM/HIP V6.1 is required". If cmake stops there, upgrade ROCm rather than hunting for a flag.
• The HIP build compiles the same kernels the CUDA build does. The same file globs ../ggml-cuda/*.cu into the ROCm target, so there is no separate AMD kernel set to be missing a feature — including ggml/src/ggml-cuda/gated_delta_net.cu, the kernel this model's 30 linear-attention layers run on. That kernel sizes its warps through ggml_cuda_get_physical_warp_size(), which is HIP-aware by construction, and RDNA3's wave size of 32 is the case it already handles. See Troubleshooting for the source-level argument in full, and for what it does not establish.

3. Download the Q4_K_M GGUF

The lead quant is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf — 21,864,082,336 bytes (21.86 GB) per the HF tree API, imatrix-calibrated, produced on llama.cpp b10630.

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

Q4_K_M is the right tier for this card by elimination. The same publisher's Q5_K_M is 23.741 GiB and Q6_K is 28.430 GiB — the first leaves nothing for a KV cache and the second does not fit at all. Downward, Q4_K_S (19.619 GiB) and IQ4_XS (17.955 GiB) are the next rungs, and they buy roughly 0.7 GiB and 2.4 GiB more spare respectively if you want to run something else on the same GPU. Going below Q4 on a model whose routed experts are already narrow (expert_feed_forward_length 512) is a real quality trade on an agent model, so this page does not lead with one.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Four of those flags are load-bearing and none of them is a default. Each was re-read in the b10690 source rather than inherited:

• --no-mmproj — no_mmproj defaults to false and mmproj_use_gpu to true in common/common.h, so a bare -hf …-GGUF resolves a sibling projector and puts it on the GPU. The lead repo ships two, at 0.838 and 0.841 GiB. With the explicit -m path above the flag is belt-and-braces; the moment you switch to -hf it is the difference between loading vision weights you never use and not.
• -np 1 — for the server example common/arg.cpp sets n_parallel to -1 ("auto"), and tools/server/server.cpp turns that into 4 slots with a unified KV pool. The attention cache is shared across slots but the Gated-DeltaNet recurrent state is per sequence — cparams.n_seq_max is set straight from n_parallel in common/common.cpp — so four slots take the recurrent state from 0.061 GiB to 0.245 GiB. One slot is the right setting for a single-user agent loop.
• -c 131072 — the GGUF declares context_length 262144. Half of it is what fits with room to spare on 24 GB; the ladder below shows why the full window does not.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 in ggml/src/ggml-common.h is a 2-byte scale plus 32 one-byte quants, i.e. 34 bytes per 32 elements, so this is 8.5 bits per element rather than 16, and it takes the 131,072-token cache from 2.500 GiB to 1.328 GiB. On this card the V half has a second consequence worth stating: src/llama-context.cpp enables Flash Attention when the V cache is quantised and refuses to start if you have explicitly turned it off — quantized V cache requires flash_attn to be enabled. So this configuration is also a decision to run the 10 full-attention layers through llama.cpp's own FA kernels, HIP-compiled. That is not the same thing as Dao-AILab's flash-attn package, which does not build on gfx1100 and is not involved here.

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05). The card notes that "Apodex follows the Qwen3.5 chat template"; use_jinja is already true by default for llama-server, so the template embedded in the GGUF is used and <think> blocks and <tool_call> structures are parsed the way the vendor intends without an extra flag.

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

The VRAM budget

Every figure here is derived from the artifact's own tensor table and from llama.cpp's allocation code at b10690. Nothing was measured on hardware.

Each GiB figure is rounded from its own byte count and the total is rounded once from the byte sum, so the GiB column adds to 21.297 while the total reads 21.298. The bytes are the arithmetic; the GiB are a convenience.

Against the card's 24 GiB that leaves 2.702 GiB. Context is the dial:

The full 262,144 window is arithmetically reachable and this page still does not recommend it. The three rows above exclude llama.cpp's compute buffers and the HIP context entirely, and llama.cpp's own auto-fit is on by default — fit_params is true in common/common.h with a target margin of 1024 MiB per device — so at 1.374 GiB spare the fitter is already at the edge of the margin it tries to hold, and its response is to start moving expert tensors to system RAM.

How the three rows are obtained:

• Weights. The file is 21,864,082,336 bytes, but its tensor payloads sum to 21,853,090,304 and 476,956,672 of that is blk.40, the multi-token-prediction head. load_arch_tensors in src/models/qwen35moe.cpp creates every block-40 tensor with TENSOR_SKIP unless MTP is explicitly requested, so a normal run loads 21,376,133,632 bytes. That is 19.908 GiB resident, against 20.363 GiB on disk — the two are not interchangeable, and the storage row above is the disk figure.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so the trunk is 40 layers, of which ten are full-attention. Each carries head_count_kv 2 at key_length 256 and value_length 256 — 1,024 cache elements per token per layer, 10,240 across the ten, i.e. 20,480 bytes per token at f16 and 10,880 at q8_0.
• Recurrent state. The other thirty layers allocate two F32 tensors each. src/llama-hparams.cpp sizes them (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements, which is 2,195,456 bytes per layer per sequence row and 65,863,680 bytes across all thirty. llama-server prints it at startup as RS buffer size.

Checking the budget on your own card

Because the table above stops short of the compute buffers, do not treat it as the last word — llama.cpp will tell you the rest, twice, for free.

Before you commit, ask the fitter what it projects:

./build/bin/llama-fit-params -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    -c 131072 -ctk q8_0 -ctv q8_0 -np 1

tools/fit-params prints the CLI arguments the auto-fitter would apply, so if it comes back having reduced your context, that is the answer for your machine — drop to -c 65536 and take the 3.367 GiB row.

Afterwards, llama-server prints a memory breakdown [MiB] table on shutdown with columns total / free / self / model / context / compute / unaccounted (common/fit.cpp). The model and context columns are what the table above derives; compute and unaccounted are exactly the two terms it does not carry. If you run this configuration, those two numbers are the most useful thing you could send to /contribute.

Results

• Speed: omitted — no figure below is a measurement of this recipe. Not because nobody has measured the model on this card: somebody has, and neither figure transfers. Both are quoted with their disqualification attached, and neither is a number for this file on mainline llama.cpp-HIP:

  • ghazni101/Apodex-1.1-mini-MQ4RP reports a 400-token "generation measured at 226 tok/s end-to-end for MQ4R" on an RX 7900 XTX, validated on gfx1100. It is the only Apodex throughput figure anywhere on this exact card — and it is a .mq4r file, not a GGUF, served by hipfire, which ghazni101's own card describes as "a Rust-native LLM inference engine for AMD RDNA GPUs". Different container, different quantisation (uniform 4.25-bit MQ4, 18.70 GB), different engine, and a vision tower and MTP head that were stripped at conversion. Right card, wrong everything else; quoting it here would be a number about somebody else's software.
  • kingjones777/Apodex-1.1-mini-ROCmFP4-COHERENT-GGUF reports its COHERENT tier at "63.30 tok/s (128-token generation, greedy)." Those are on a Ryzen AI MAX+ 395 — a unified-memory APU, not a discrete card — and its own card is headed "Stock llama.cpp will not load this file", because its tensor format "is a custom tensor format that exists only in the" ROCmFPX fork. Treat the rest of that publisher's table with care: its STRIX_LEAN build is printed at 64.87 in a shared variants block whose caption says "A dash means I haven't measured that one yet", while the STRIX_LEAN card carries both that block and its own "Speed — full offload" table, which shows a dash for the same build and says "Full-offload speed being measured on an idle box, card will be updated." One page, two answers about one figure.

  So a figure exists for the model, and one even exists for this card, and none exists for this file on mainline llama.cpp-HIP on this card. The space I searched, on 2026-08-30: the model cards of all 21 HuggingFace repositories matching Apodex-1.1-mini (20 read; the 21st, aykutx21/Apodex-1.1-mini-GGUF, contains one file, .gitattributes, and no README or weights), their discussions tabs (one thread across all 21, a vendor community announcement), the llama.cpp tracker (a search for apodex returns nothing, against a non-empty control — 269 results for the architecture string qwen35moe at the time of writing), and this site's own catalogue — /check/apodex-1-1-mini/rx-7900-xtx has no benchmark, and the RX 7900 XTX carries three in total, none of them a mixture-of-experts model. If you run this, please contribute the numbers.

• VRAM usage: 21.298 GiB derived working set at 131,072 tokens with q8_0 KV, leaving 2.702 GiB of the card before compute buffers. Dropping to -c 32768 brings it to 20.301 GiB. See /check/apodex-1-1-mini/rx-7900-xtx for live data as it lands.

• Quality notes: the vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. The lead quant is imatrix-calibrated against a corpus rendered through this model's own chat template, which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier.

For the full benchmark data, see /check/apodex-1-1-mini/rx-7900-xtx.

Troubleshooting

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE in src/llama-arch.cpp, which arrived in (b7989, b7990] — so this means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer.

Note that architecture registration lives in src/, not in a backend, so it is not something a HIP build can lack relative to a CUDA one. Whether the kernels are there is the separate question below.

Does a HIP build actually run this architecture?

Nobody has published a run of this model on this card under mainline llama.cpp, so the honest answer is an argument from the source plus three adjacent reports, and it is worth seeing where each stops.

From the source, at b10690:

• The ROCm target compiles ../ggml-cuda/*.cu wholesale, so gated_delta_net.cu, ssm_conv and the rest are in the binary. One kernel set, compiled twice.
• ggml_backend_cuda_device_supports_op — the function that decides whether an operation runs on the GPU or falls back to CPU — contains zero GGML_USE_HIP guards anywhere in its body — against 15 elsewhere in the same file — in ggml/src/ggml-cuda/ggml-cuda.cu. It contains four GGML_USE_MUSA guards, and one of them sits on GGML_OP_GATED_DELTA_NET itself: the single backend carved out of this model's linear-attention operator is Moore Threads', not AMD's.
• The kernel is HIP-aware rather than merely HIP-compatible: it derives its warp width from ggml_cuda_get_physical_warp_size(), which branches on GGML_USE_HIP and returns 64 only for GFX8/GFX9. RDNA3 takes the 32 branch, which is what a real gfx1100 device line reports (Wave Size: 32).
• The kernel has had AMD-specific attention: llama.cpp PR #20282, titled ggml-cuda: gdn use shared mem for HIP, is a HIP-targeted revision of this exact kernel. It was closed without being merged, so read it as evidence that people compile and profile this path on AMD, not as a shipped optimisation.

From the field, and this is the part that matters most, because it is a run rather than an argument:

• #27894 is a qwen35moe model running on this exact card. Its reporter served Ornith-1.5-35B-A3B — the same hybrid architecture string, the same 3-in-4 SSM layer pattern — on an RX 7900 XTX (Navi 31, gfx1100, 24 GB) under ROCm 7.2.0 on Ubuntu, from a build configured -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100, at two llama.cpp builds ~950 apart (b9722 and b10673). Forty consecutive completions in one configuration, -ngl 99 --n-cpu-moe 0 --ctx-size 147456 — the same full offload this page documents, at a longer context than it recommends. A separate controlled matrix, same protocol, then varied one thing at a time: the MTP head on and off, --mmproj on and off, flash attention on and off, KV at both q4_0 and f16, and two different model files. The issue was filed about host-memory growth, and it was not a generation defect: nothing in it reports wrong or missing output, and it closed as "Confirmed — you're right, this isn't a leak." once a llama.cpp contributor identified the cause (see the host-RAM section below). So the strongest thing the tracker says about this architecture on gfx1100 is that someone ran it hard for forty requests and the only thing they had to report was a --cache-ram default.

• Someone analysing an unrelated qwen35moe bug writes "We have not reproduced the bug — we run this architecture on CPU and ROCm, not Metal — so this is analysis of your artifacts, not a second data point." (#27442). Their GitHub author_association is CONTRIBUTOR — someone with a merged PR, not a maintainer — so read it as a community statement that the architecture gets run under ROCm as a matter of course. It is not a measurement, and they say so themselves.

• #27698 reports a qwen35moe GGUF loading and generating at ~6.4 t/s under the ROCm server container at build b10108, and failing to load at b10615 — on an RX 6700 XT, which is gfx1031 / RDNA2, a 12 GB card, with a partial offload (-ngl 20 --n-cpu-moe 0 --fit off) that this recipe does not use. Open, no comments, no maintainer response, and not bisected. It is a positive datapoint for the architecture under HIP and a caution about partial offload on a smaller RDNA2 card; it is not evidence about a full offload on 24 GB of RDNA3.

What none of that establishes is that a full 131,072-token run of this checkpoint is correct on gfx1100 — #27894 is a sibling model, and its reporter was watching resident memory rather than checking recall. Read the next two sections before you rely on a long context.

It generates fluently but seems to have forgotten the start of the prompt

This is the failure mode to know about on AMD, because nothing about it looks like a failure. There is an open cluster of reports — #27556, #27579 — in which the HIP backend silently drops the oldest part of the context while the output stays fluent and deterministic, and the Vulkan build of the identical commit is correct. It is not a Gated-DeltaNet defect, and the wider scope is a reason to take it more seriously rather than less: #27579's own matrix has a plain dense llama model (Llama-3.1-8B Q4_K_M) failing at every depth it tested — 0/16 at 30,516 tokens and 0/6 at a 1,616-token prompt, while a 512-expert bailingmoe3 MoE passes 16/16 at 29,275 tokens. Its qwen35 row — the architecture family closest to this page — is correct at shallow depth and fails past roughly 29K.

Every report in those two issues — both reporters and every commenter — is on gfx1151, the Radeon 8060S inside a Ryzen AI MAX+ 395, and the mechanism the threads converged on is specific to integrated GPUs. #27579 draws its own boundary in as many words: "Only gfx1151. Discrete RDNA/CDNA cards untested." And on the cause, a commenter on #27556 puts it directly: "The cause is a write after read race on graph input tensors when the backend computes directly from host memory on integrated GPUs." The two candidate fixes are named accordingly — PR #27311 is a scheduler UMA ring buffer and PR #25863 is titled avoid direct ROCm_Host compute on HIP integrated GPUs. Both were still open at the time of writing.

The source agrees about the scope. ggml-cuda.cu sets a per-device integrated flag from the device property on HIP builds, and ggml_backend_cuda_device_supports_buft accepts a host-visible buffer for compute only when that flag is set. A discrete RX 7900 XTX does not report itself as integrated, so the path those reports blame is not one this card takes. That is a real narrowing and it is not a clean bill of health: the issues are open, unlabelled and unconfirmed, every person who has posted in the two threads carries a GitHub author_association of NONE, and nobody has run a long-context needle test on gfx1100 with this checkpoint.

So run one. The same commenter gives the method, and it costs a single prompt: "You need a retrieval probe: put a unique marker early in the prompt and ask for it back verbatim." Put a distinctive string in the first few hundred tokens of a 100K-token prompt, ask for it back, and check the answer rather than the prose around it. Fluent output is not evidence — that is the whole point of this class of bug. If the marker comes back, tell us; a single confirmed needle test on this card would be worth more than every paragraph above.

The server's host RAM climbs by ~126 MiB per request

It is not a leak, and the ceiling is higher than you would guess. llama-server keeps a host-side prompt cache bounded by -cram / --cache-ram, which defaults to 8192 MiB (common/common.h). For an ordinary attention-only model each cache entry is a few MiB and the default is invisible. For a hybrid like this one an entry has to carry recurrent state as well as KV, so entries are two orders of magnitude larger and the cache reaches its bound in tens of requests rather than thousands.

Two numbers, and they are not the same number. This model's per-sequence recurrent state is 65,863,680 bytes — 62.8 MiB, the same figure budgeted for the GPU above, and derived. The only measured entry size is 126 MiB, from #27894 in the next paragraph. An entry therefore costs about twice one recurrent state, and this page does not know what the second half is: a trivial prompt at q4_0 KV cannot account for it, and nobody has instrumented it. If you follow the link you will meet a third answer — the reporter identifies the entry as "a cache entry for a hybrid model is one full recurrent state, a constant ~126 MiB", which is an inference rather than a measurement, and on this geometry one state is 62.8 MiB, not 126. The 126 is measured and stands; the identification of what it consists of does not. Size a host from the measured 126 MiB, not from the derived 62.8.

This was chased down on this exact card. The reporter of #27894 measured a flat +126 MiB per request across forty requests on an RX 7900 XTX, ruled out nine other variables, and filed it as a leak; a llama.cpp contributor pointed at --cache-ram, and --cache-ram 512 reproduced the growth and then held flat for ten consecutive requests at +507 MiB — about four entries against a 512 MiB bound. That was measured on an Ornith-1.5-35B-A3B build rather than on Apodex, so the transfer needs checking — and it checks out against the measured file itself. The artifact #27894 names, Ornith-1.5-35B-A3B-APEX-MTP-I-Compact.gguf, is published at mudler/Ornith-1.5-35B-A3B-APEX-MTP-GGUF (17,437,861,408 bytes) — not at the repository the issue itself links, which carries eighteen files under different names. Match the filename exactly: the same publisher ships an APEX-MTP-Compact 256 bytes smaller, so the I- is load-bearing. Its header declares qwen35moe with block_count 41, nextn_predict_layers 1, ssm.conv_kernel 4, ssm.state_size 128, ssm.group_count 16, ssm.inner_size 4096 and full_attention_interval 4 — the same 753 tensors and the same 30 recurrent layers as this file, so its per-sequence state is byte-for-byte the same 62.8 MiB. The comparison is therefore between two files with identical recurrent geometry, which is what makes the factor of two a real gap rather than a mismatch between models. Their own summary of the ergonomics is worth reading before you size a machine: "the 8192 MiB default is invisible for attention-only models where entries are a few MiB, but for a hybrid SSM model each entry is ~126 MiB, so the same default can commit 8 GiB of host RAM".

Why this is sharper on an AMD box than the number suggests: in that report the host-RAM pressure collided with an amdgpu VRAM-to-GTT eviction of the model, and the evicted weights then had nowhere to go. If you are running this on 16 GB of system RAM, set --cache-ram deliberately — 0 disables the cache entirely, and a few hundred MiB keeps a useful prefix cache without letting it grow into the space the driver may need.

Do not reach for the Vulkan backend as a workaround here

On RDNA3 the usual advice is that llama.cpp's Vulkan backend sometimes beats ROCm at token generation, and for ordinary dense models that is fair. For this architecture family on this card it is the wrong reflex. #27237 reports a hybrid Gated-DeltaNet model producing garbage or truncated output on the Vulkan backend on an RX 7900 XTX, reproducibly, at batch size 512, and generating correctly at 1024 and 4096. Flash Attention on or off, KV f16 or q8_0, MTP on or off — all reproduce. A third party then tried it on Intel ANV and NVIDIA's proprietary ICD and could not reproduce, concluding that "this does not reproduce for me at ubatch 512 on either ICD, so the fault may be specific to the AMD drivers rather than to the Vulkan backend generally".

Two scoping notes, because this is a report about a neighbour rather than about this exact model: the checkpoint is a Qwen3.5-27B derivative on the qwen35 architecture rather than qwen35moe, and the reporter drove it through koboldcpp's bundled Vulkan backend on Windows. What carries across is the mechanism they point at — the Gated-DeltaNet kernel's path selection at small batch — which is a property of the hybrid layer type this model also has 30 of. Note what the reproduction actually pins: both -b 512 and ubatch 512 — and 512 is llama.cpp's own default -ub (common/common.h; the default -b is 2048). So the failing configuration contains the stock micro-batch, the report does not isolate which of the two knobs is the trigger, and the third party's negative result is framed at ubatch 512 as well. I could not determine the trigger either. Read that as: this is not a configuration you can be sure of having avoided by leaving the defaults alone — which is the argument for staying on HIP here, not for tuning batch sizes on Vulkan.

The server runs on the CPU instead of the GPU

Confirm ROCm is installed and the card is visible (rocm-smi should list the 7900 XTX), and that your user is in the render and video groups — groups should show both, and the change only takes effect after logging out and back in. For a source build, confirm you configured with -DGGML_HIP=ON and that the startup banner names a ROCm device. You should not need HSA_OVERRIDE_GFX_VERSION on this card; if you find yourself setting it, the build is probably targeting the wrong architecture, and -DGPU_TARGETS=gfx1100 is the fix.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and mainline llama.cpp implements it — src/models/qwen35moe.cpp builds a dedicated MTP graph, enabled with --spec-type draft-mtp (the exact spelling matters), with no draft model needed. It is not free: the block-40 tensors stop being skipped (+0.444 GiB), the draft depth multiplies the recurrent-state rows, and a second single-layer KV cache is allocated for the draft context at f16 unless you also pass -ctkd. At this recipe's context that eats most of the 2.702 GiB the budget above leaves, which is already before compute buffers, so if you want to try it drop to -c 32768 first and take the 3.699 GiB row.

On AMD there is a second reason for caution and it is a plain absence: nobody has published an MTP run of this architecture under HIP, on any card. The two publishers who built this model specifically for AMD both went the other way — ghazni101 strips the MTP draft head from its builds (quoted in full above) and kingjones777 converted with --no-mtp. Treat the flag as an experiment with a known VRAM price and an unmeasured throughput return, and compare tokens per second with and without it on your own prompts before keeping it.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos. On this card they are ruled out twice over.

First, they are vLLM/SGLang safetensors rather than GGUF, and llama.cpp cannot load them at all. Second, and specific to RDNA3: the card has no FP8 and no FP4 tensor hardware, so an FP8 or NVFP4 checkpoint is not a memory-saving path here — the weights upcast on load and you pay more memory for no compute acceleration. Size alone would not have settled it, incidentally: at 22.548 GiB and 22.958 GiB the NVFP4 and GPTQ-Int4 weight sets do fit inside 24 GiB. What rules them out is what is left afterwards, before a single token of KV cache. And the -NVFP4 repo is not even a 4-bit checkpoint: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4.

If you need the VRAM back for something else

You should not need expert offload on this card — the whole model is resident with room to spare — but -ncmoe N / --n-cpu-moe N is the lever if you want the GPU for something else at the same time. It is architecture-agnostic: common/arg.cpp hands it LLM_FFN_EXPS_REGEX, defined in common/common.h as "\\.ffn_(up|down|gate|gate_up)_(ch|)exps", and the matched tensors are re-pointed at ggml_backend_cpu_buffer_type(). In this file 123 tensor names match, 18.164 GiB of them across the 40 loaded blocks — 91.2% of the resident weights — so -ncmoe 40 would leave a 1.744 GiB floor of attention, norms, shared experts and embeddings on the card, with the expert tensors resident in system RAM. That 18.164 GiB is a host-RAM requirement, and it is larger than the 16 GB this page's Requirements row states — which is sized for the configuration above, where nothing is offloaded. Taking the full ladder means a 32 GB host, and that is before the prompt cache described under The server's host RAM climbs by ~126 MiB per request above.

Two warnings if you go there. The step size is not uniform: the publisher upcasts ffn_down_exps to Q6_K in 20 of the 40 loaded blocks (0-4, then every third from 7 to 34, then 35-39), so one step costs either 0.4219 GiB or 0.4863 GiB depending on which block it is, and any rule of thumb of the form one step frees X GB is wrong for this file. And nobody has measured what offload does to throughput here, on this card or any other — the only ROCm partial-offload report for this architecture is the RDNA2 one above, where it stopped loading.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex, /apodex-1.1-mini and /apodex1.1 all return 404, and so does the corresponding registry.ollama.ai manifest, against a control of qwen3.5 which returns 200 on both surfaces. Use llama.cpp directly. If a tag appears later, please tell us.