Apodex 1.1 mini on RX 7800 XT: a 36B agent in 16 GB with ROCm and llama.cpp expert offload

What You'll Build

A local OpenAI-compatible endpoint serving Apodex-1.1-mini — an Apache-2.0 agent model that Apodex AI fine-tuned from Qwen/Qwen3.5-35B-A3B — on a single Radeon RX 7800 XT, at a 131,072-token working context, from a Q4_K_M GGUF that is larger than the card.

Two things are happening at once on this page and they are worth separating before you start. The first is capacity: the Q4_K_M file loads 19.908 GiB of weights and this card holds 16 GiB, so the model runs only because it is a sparse Mixture-of-Experts and llama.cpp will leave the routed expert tensors of the first N blocks in system RAM. 123 tensors carry 18.586 GiB of the file, and across the forty loaded blocks that is 18.164 GiB of the 19.908 GiB resident — 91.2 %. Everything the card actually has to keep — attention, the Gated-DeltaNet mixers, the shared experts, the router, the embeddings and the output head — is 1.744 GiB. The recipe is a choice of how much of that 91.2 % to push across.

The second is that this is an AMD card, and the two facts meet here in a way they do not meet anywhere else in this model's coverage. On a 24 GB Radeon nothing is offloaded and host RAM is nearly free; here 8.174 GiB of host RAM is spoken for before the operating system gets a look in, and the amdgpu driver's habit of evicting VRAM into host memory rather than failing an allocation lands on exactly that budget. System RAM is the section that matters most on this page, and it is not the one you would expect to matter.

Hardware data: RX 7800 XT (16 GB VRAM, gfx1101) · 13.124 GiB derived working set at -ncmoe 18, 131,072-token context · See benchmark data

⚠️ This is a ROCm recipe, not CUDA. RDNA3 has no FP8 and no FP4 hardware, no FlashAttention-2 prebuilt wheel, no ExLlamaV2 and no Marlin. Do not follow a guide that tells you to install a cu12x wheel, pip install flash-attn, or reach for an FP8 checkpoint in the hope of saving memory — on this card an FP8 tensor upcasts and costs more. The reliable path is a GGUF served by llama.cpp compiled against HIP, which is what this page documents.

⚠️ This model does not fit 16 GB on its own, and system RAM is a prerequisite rather than a footnote. At the setting below, 8.174 GiB of expert weights are resident in host RAM for as long as the server runs, and llama-server's prompt cache can commit several GiB more on top. bartowski's own sizing rule for this file is to "add both your system RAM and your GPU's VRAM together" and pick a quant a couple of GB below the sum (model card); with 20.363 GiB of weights that is satisfied on paper by 16 GB of RAM, and on an AMD box you want 32 for the reason in the eviction section below.

ℹ️ This recipe is text-only, deliberately. The upstream checkpoint contains a vision tower and the GGUF repos ship a projector for it, but the vendor documents the model as pipeline_tag: text-generation and its model card carries no image-input instructions at all — the words "vision", "visual", "multimodal" and "mmproj" appear zero times in it. Both publishers who built this model specifically for AMD dropped the vision half as well: ghazni101 states the vision tower and MTP head are "intentionally absent from these quants". Every number below is for text generation, and the run command refuses the projector explicitly rather than merely omitting --mmproj.

⚠️ Two version floors, about 2,200 builds apart. They are properties of the model rather than of the card, and they were bisected once by the sibling pages rather than re-bisected here.

• The architecture. LLM_ARCH_QWEN35MOE is absent from src/llama-arch.cpp at release b7989 and present at b7990, so it arrived in (b7989, b7990].
• The qwen35moe-specific MTP wiring — the load_mtp / TENSOR_SKIP handling in src/models/qwen35moe.cpp that keeps block 40 off the card until you ask for it — arrived in (b10211, b10212].

Both are intervals on purpose: each is bracketed by adjacent published tags, which is as fine as a tag-granularity probe can resolve. Use b10630 or newer — the build the lead quant was produced with. That is a sufficient floor, not a bisected minimum. Every source reference on this page was read at release b10690 (bdf3955159d7184f44b76091973eeff532890a35, published 2026-08-30); the release track moves several times a day and b10701 was the newest tag while this was being written. See Troubleshooting for one open ROCm regression report, on a different card, whose first-known-bad build is older than both of these.

Requirements

The RX 7800 XT is an officially supported ROCm GPU, and it is worth checking that yourself rather than taking a forum's word for it. AMD's install-on-Linux system-requirements matrix lists AMD Radeon RX 7800 XT · RDNA3 · gfx1101 · supported, alongside the RX 7700 XT and RX 7700 on the same target, and the same table's footnote restricts every consumer Radeon to the four OS releases above. Because gfx1101 is natively supported you do not need HSA_OVERRIDE_GFX_VERSION=11.0.0; that variable is a masquerade for cards ROCm ships no kernels for, and older community threads that present it as a required step for this card predate the current support matrix.

gfx1101 is not gfx1100. The RX 7900 XTX, XT and GRE are Navi 31 / gfx1100; this card is Navi 32 / gfx1101, a different LLVM target with its own code objects in the fatbin. Anything you read about the 7900 series is a statement about a neighbour, and this page keeps the two apart wherever it matters — including in the one open llama.cpp issue that names this exact card.

Installation

1. Install the ROCm driver stack

ROCm is not bundled with llama.cpp — you install it once at the OS level. The block below is AMD's own Ubuntu 24.04 track from the ROCm quick-start installation guide, which is the ROCm 7.14.0 production documentation set:

# 1. register AMD's repositories
wget https://repo.radeon.com/amdgpu-install/7.2.4/ubuntu/noble/amdgpu-install_7.2.4.70204-1_all.deb
sudo apt install ./amdgpu-install_7.2.4.70204-1_all.deb
sudo apt update

# 2. kernel driver — remove any previous AMD GPU driver first, and do not drop the headers line
sudo apt install "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install amdgpu-dkms
# reboot here to apply all settings

# 3. ROCm itself
sudo apt install python3-setuptools python3-wheel
sudo usermod -a -G render,video $LOGNAME   # Add the current user to the render and video groups
sudo apt install rocm
# reboot again to apply all settings

Four things about that block, in the order they will bite you.

Take the package path from AMD's page rather than from here. The URL carries the release number and it moves; 7.2.4 is what the guide shipped when this page was written. Ignore the "This page has moved!" banner at the top of it unless you actually want the ROCm 10.0 installer — AMD publishes several install doc sets simultaneously and they are on different major versions. The top-level install/rocm.html is the ROCm 10.0 set, whose every mention of 7.2.4 is an instruction to uninstall it first, and the Radeon-and-Ryzen set covers releases only through 7.2.1.

Do not drop the linux-headers line. amdgpu-dkms builds a kernel module out of source, so it needs the running kernel's headers at /lib/modules/$(uname -r)/build — and apt will not fetch them for you. In AMD's own package index for the repository the .deb above registers, amdgpu-dkms declares Depends: dkms (>= 1.95), libc-dev | libc6-dev, autoconf, automake, initramfs-tools, amdgpu-dkms-firmware; linux-headers appears nowhere in that index. Without it the DKMS build fails, amdgpu never loads, and every symptom lands much later — rocm-smi showing nothing, the HIP build finding no device, -ngl 99 silently running on the CPU. linux-modules-extra is likewise not guaranteed on server and cloud images, which is why AMD names it explicitly on the Ubuntu tabs.

Remove any earlier AMD GPU driver before step 2. AMD prints a Caution to that effect above the kernel-driver commands, pointing at its own Uninstall kernel driver steps. A left-over driver from a distribution package or a previous ROCm release is the common reason amdgpu-dkms installs cleanly and the card still does not come up.

And note what the usermod line does not do. AMD follows it with a Note reading "Quick Start enables GPU access for the current user only." — and it links its own Configuring permissions for GPU access page for the rest — so if you intend to run the server as a service account rather than as yourself, that account needs render and video too. Group membership only takes effect after logging out and back in; groups should show both. Confirm the card is visible with rocm-smi before going further.

The older amdgpu-install --usecase=graphics,rocm one-liner that most AMD guides still show is not this route. The same doc set's installation overview states that "The AMDGPU installer documentation has been removed to encourage the use of the package manager for ROCm installation. While the package manager is the recommended method, you can still install ROCm using the AMDGPU installer" — so the one-liner is deprecated rather than dead, but it is undocumented, and the block above is the package-manager path AMD points you to instead.

2. Build llama.cpp with the HIP backend, targeting gfx1101

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10690
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release -- -j 16

This is the llama.cpp HIP build recipe with GPU_TARGETS set to this card's LLVM target — not gfx1100. Three things worth knowing:

• There is a hard floor on the ROCm version. ggml/src/ggml-hip/CMakeLists.txt:54-55 fails the configure step outright with "At least ROCM/HIP V6.1 is required". If cmake stops there, upgrade ROCm rather than hunting for a flag.
• The HIP build compiles the same kernels the CUDA build does. The same file's line 63 is file(GLOB GGML_SOURCES_ROCM "../ggml-cuda/*.cu"), so the whole CUDA tree is compiled into the ROCm target — there is no separate AMD kernel set that could be missing a feature, including ggml/src/ggml-cuda/gated_delta_net.cu, the kernel this model's 30 linear-attention layers run on. That kernel takes its warp width from ggml_cuda_get_physical_warp_size() in ggml/src/ggml-cuda/common.cuh:374-380, which returns 64 only under GGML_USE_HIP and __GFX9__/__GFX8__; RDNA3 takes the 32 branch, which is what a gfx1101 device reports.
• Build for your target explicitly rather than relying on a generic package. Issue #25620 — filed by a GitHub CONTRIBUTOR, still open — documents MMVQ producing NaN at batch-1 decode when the loaded HIP code object was compiled for --offload-arch=gfx11-generic and the runtime device has no native code in the fatbin, with the symptom being "coherent-looking prompt processing followed by tokens rendered as ?????... at plausible generation rates". Its scope is RDNA3.5 (gfx1150/1152/1153), not gfx1101, and a source build pinned to -DGPU_TARGETS=gfx1101 produces a native code object either way — but it is the reason this page builds from source with a named target instead of taking a distribution's generic ROCm package.

3. Download the Q4_K_M GGUF

pip install -U huggingface_hub
hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
    apodex_Apodex-1.1-mini-Q4_K_M.gguf --local-dir ./apodex-1.1-mini

That is bartowski/apodex_Apodex-1.1-mini-GGUF apodex_Apodex-1.1-mini-Q4_K_M.gguf, 21,864,082,336 bytes (20.363 GiB), imatrix-calibrated and produced on llama.cpp b10630 — re-fetched from the HF blob API on 2026-08-30, when the repository's lastModified was 2026-08-26T09:14:35Z.

Q4_K_M on a 16 GB card is a deliberate choice, and the obvious alternative is narrower than it looks. Ten of bartowski's quant rungs are under 16 GiB on disk — IQ2_XXS 9.551, IQ2_XS 10.500, IQ2_S 10.698, IQ2_M 11.682, Q2_K 12.187, Q2_K_L 12.650, IQ3_XXS 14.287, Q3_K_S 14.886, IQ3_XS 15.543 and Q3_K_M 15.550 GiB (counted over the repo's 29 .gguf files with the two mmproj projectors and the imatrix file excluded, since none of the three is a rung), so "small enough to load" is not the constraint. The constraint is what has to sit beside the weights. Hold this page's own terms — 32,768 tokens of q8_0 KV (0.332 GiB), the recurrent state (0.061 GiB) and the ~2.9 GiB reserve this page keeps — and a fully resident configuration needs weights at or below 12.707 GiB. That is Q2_K_L (12.650 GiB) and everything smaller; the whole 3-bit tier misses it by a mile, IQ3_XXS at 14.287 GiB and Q3_K_S at 14.886 GiB. So the escape from offload is not "drop to 3-bit", it is "drop to 2-bit" — and that is exactly where the publisher of a second GGUF set draws the line: abenzerps suggests IQ2_M or IQ3_M as a 16 GB starting point and then closes the section with "Prefer Q4_K_M or higher when system memory permits." This recipe follows that advice, because the model is an agent that runs tool calls and multi-step plans, and 2-bit weights are the wrong place to save on a workload where a single malformed argument ends the run. A 3-bit rung with a shallower -ncmoe is a reasonable middle and this page does not measure it either.

Running

./build/bin/llama-server \
    -m ./apodex-1.1-mini/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
    --no-mmproj \
    -ngl 99 \
    -ncmoe 18 \
    -np 1 \
    -c 131072 \
    -ctk q8_0 -ctv q8_0 \
    --load-mode none \
    --cache-ram 2048 \
    --temp 1.0 --top-p 0.95 --repeat-penalty 1.05 \
    --host 127.0.0.1 --port 8080

Then talk to it:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [{"role": "user", "content": "Plan a three-step verification pass for a CSV of quarterly revenue."}]
  }'

Sampling values are the vendor's own recommendation from the model card (temperature 1.0, top_p 0.95, repetition_penalty 1.05), whose text notes that "Apodex follows the Qwen3.5 chat template"; use_jinja is already true by default for llama-server, so the template embedded in the GGUF is used and <think> blocks and <tool_call> structures are parsed the way the vendor intends without an extra flag.

Eight of those flags are load-bearing and none of them is a default. Each was re-read in the b10690 source rather than inherited from a sibling page:

• -ngl 99 puts every layer on the GPU as the starting point. -ncmoe then walks part of it back off.
• -ncmoe 18 is this recipe's subject; the next three sections derive the number.
• --no-mmproj — no_mmproj defaults to false and mmproj_use_gpu to true in common/common.h:594,596, so a bare -hf …-GGUF resolves a sibling projector and puts it on the GPU. The lead repo ships two, at 899,283,488 and 902,822,432 bytes (0.838 and 0.841 GiB). On this card that is just under the 0.844 GiB that the two cheapest steps of -ncmoe would free, spent on a capability the recipe does not use.
• -np 1 — for the server example common/arg.cpp:1400 sets n_parallel to -1 ("auto"), which tools/server/server.cpp turns into 4 slots. The Gated-DeltaNet recurrent state is per sequence, so four slots take it from 0.061 GiB to 0.245 GiB. One slot is right for a single-user agent loop.
• -c 131072 — the GGUF declares context_length 262144. Half of it is what this recipe budgets; the ladder below prices the other half.
• -ctk q8_0 -ctv q8_0 — quantised KV. block_q8_0 in ggml/src/ggml-common.h is a 2-byte scale plus 32 one-byte quants, i.e. 34 bytes per 32 elements — 8.5 bits per element rather than 16. It takes the 131,072-token cache from 2.500 GiB to 1.328 GiB, which here is two to three steps of -ncmoe you do not have to spend. It also decides a flag you are not passing: a quantised V cache requires FlashAttention, so with the default -fa auto src/llama-context.cpp enables it, while an explicit -fa off is a hard error and the context is never created. That is fine here — the same file checks that q8_0's 32-element block divides n_embd_head_k and n_embd_head_v, both 256 on this model — but it means this configuration runs the ten full-attention layers through llama.cpp's own HIP-compiled FA kernels. That is not the same thing as Dao-AILab's flash-attn package, which does not build on RDNA3 and is not involved here.
• --load-mode none — llama.cpp asks for this itself. When a CPU tensor override meets memory mapping, src/llama-model-loader.cpp prints "tensor overrides to CPU are used with mmap enabled - consider using --load-mode none for better performance". The mechanism is worth knowing: an overridden tensor is given the first workable buffer type from a list src/llama-model.cpp builds in the order "ACCEL -> GPU host -> CPU extra -> CPU", whose GPU-host entry exists precisely because "storing the tensors in a host buffer is useful when the processing of large batches" is offloaded to a GPU — pinned memory the card can DMA from. The loader downgrades that choice to ordinary CPU memory whenever mmap is on. So mmap silently costs you the pinned buffer this whole path was built around.
• --cache-ram 2048 — see System RAM. The default is 8192 MiB and on this architecture, on this card, that is the single most dangerous default on the page.

What -ncmoe actually moves

-ncmoe N / --n-cpu-moe N is documented as "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" (common/arg.cpp:2789-2790), and on a hybrid recurrent model like this one it is easy to assume that more moves than actually does.

It matches tensor names, not architectures. The flag appends one buffer-type override per block index i in 0 … N-1, each a regex built from LLM_FFN_EXPS_REGEX in common/common.h:1130:

const char * const LLM_FFN_EXPS_REGEX = "\\.ffn_(up|down|gate|gate_up)_(ch|)exps";

In this GGUF exactly 123 tensors match — ffn_down_exps, ffn_gate_exps and ffn_up_exps, three per block across all 41 blocks. The override is applied at tensor-creation time, in the loader, before any buffer exists: a matching tensor is given a CPU-side buffer type (common/common.h:1139 pairs the regex with ggml_backend_cpu_buffer_type()) and is then allocated and read into it. Those bytes never enter VRAM at all. Nothing about the architecture is consulted, which is why the flag works identically on a hybrid model and on a plain transformer MoE — and why it works identically on HIP and on CUDA, since the buffer type it names is the CPU one on both.

Three things it does not move, all of which stay on the card:

• The Gated-DeltaNet mixers and the attention weights. A GDN block's ssm_* tensors, its attn_qkv and attn_gate, and a full-attention block's attn_q/k/v/output are untouched by the regex.
• The shared expert and the router. ffn_down_shexp, ffn_gate_shexp, ffn_up_shexp, ffn_gate_inp and ffn_gate_inp_shexp do not match _exps, so the dense path every token takes stays resident.
• The KV cache and the recurrent state. Both pick their buffer type from the layer's device, which is decided by -ngl alone; -ncmoe rewrites weight tensors and never touches the layer assignment. So a block whose experts are in RAM still keeps its recurrent state in VRAM — which is what makes the budget below decomposable at all.

Add it up and -ncmoe 40 would leave 1.744 GiB on the card.

The VRAM budget

Every figure here is derived from the artifact's own tensor table and from llama.cpp's allocation code at b10690. Nothing on this page was measured on an RX 7800 XT.

Every GiB figure is rounded from its own byte count and the total is rounded once from the byte sum, which is why the GiB column adds to 13.123 while the total reads 13.124. The bytes are the arithmetic; the GiB are a convenience.

How the three rows are obtained:

• Weights. The file is 21,864,082,336 bytes, its tensor payloads sum to 21,853,090,304, and 476,956,672 of that is blk.40, the multi-token-prediction head, which src/models/qwen35moe.cpp creates with TENSOR_SKIP unless MTP is explicitly requested. A normal run therefore loads 21,376,133,632 bytes = 19.908 GiB resident, against 20.363 GiB on disk — the two are not interchangeable, and the Storage row above is the disk figure. -ncmoe 18 moves the experts of blocks 0–17 (8,776,581,120 B = 8.174 GiB) into host RAM, leaving 12,599,552,512 B on the card.
• KV cache. block_count is 41 and nextn_predict_layers is 1, so the trunk is 40 layers, of which ten are full-attention — full_attention_interval is 4, and the tensor list confirms blocks [3, 7, 11, 15, 19, 23, 27, 31, 35, 39]. Each carries head_count_kv 2 at key_length 256 and value_length 256, i.e. 1,024 cache elements per token per layer and 10,240 across the ten — 20,480 bytes per token at f16 and 10,880 at q8_0.
• Recurrent state. The other thirty layers allocate two F32 tensors each: (conv_kernel − 1) × (inner_size + 2 × group_count × state_size) = 24,576 elements and state_size × inner_size = 524,288 elements, which is 2,195,456 bytes per layer per sequence and 65,863,680 bytes across all thirty. llama-server prints it at startup as RS buffer size.

Against the card's 16 GiB that leaves 2.876 GiB, and none of the three rows above accounts for what has to live in it:

• llama.cpp's compute buffers and the HIP context, as on any recipe.
• The display driver's reservation, if the card is also driving a monitor, and the device's reported total sitting slightly under the nominal 16,384 MiB.
• A GPU-side staging copy of the expert weights during prompt processing, which only exists on the offloaded path — see the next-but-one section for why. The scheduler's copy slots are keyed per tensor, and the largest single expert tensor here is ffn_down_exps at 220,200,960 B; one block's three expert tensors together come to 0.486 GiB at the largest, which is the bound to reserve against rather than a figure the allocator is known to hold at once.

So 2.876 GiB is a reservation, not a measurement. Read the real numbers off the server's own logs: llama.cpp prints model buffer size once per buffer, so with -ncmoe you get a ROCm line and a host line and the split above is directly checkable, followed by KV buffer size, RS buffer size and compute buffer size; on shutdown it prints a memory breakdown [MiB] table with total / free / self / model / context / compute / unaccounted columns, and compute and unaccounted are exactly the two terms this table does not carry. Cross-check against rocm-smi --showmeminfo vram before you rely on the configuration. If you run it, those two columns are the most useful thing you could send us; this pair has no benchmark in our catalogue.

The offload ladder

The ladder is not linear, and this is the part that does not transfer from any other recipe. bartowski upcasts ffn_down_exps to Q6_K in 20 of the 40 loaded blocks — 0, 1, 2, 3, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, and then 34 through 39 — so one step of -ncmoe costs either 0.4219 GiB or 0.4863 GiB depending on which block it lands on. Any sentence of the form "a step of -ncmoe costs X GB" is wrong on this file; read the cumulative column instead.

Weights only — no KV cache, no recurrent state, no compute buffers. The left column is a system-RAM requirement, and it is the number to size your host against.

Put the KV cache and the recurrent state back on top and hold the reserve near 2.9 GiB, and the whole recipe collapses to one trade. The KV cache is linear in context, so the price of a doubling is the whole cache you already had:

All four rows use -ctk q8_0 -ctv q8_0 and -np 1, and all four are arithmetic rather than measurement. The middle row is the one the command above runs, and the context is chosen so that this page and the RTX 3090 page document the same window — the entire difference between a 24 GB card and this one is that 8.174 GiB of experts have moved off the card and into RAM. Note what the table does not show: the full 262,144-token window is reachable here, for three more steps of offload and 1.330 GiB more host RAM. Capacity is not what stops you going there; the unmeasured decode cost of reading three more blocks of experts out of system RAM is.

Why the reserve here is wider than a CUDA card of the same size would need, and it is the one place this page deliberately diverges from its NVIDIA sibling. On an NVIDIA card, asking for more VRAM than exists is an allocation failure and you find out immediately. On amdgpu it may not be: the driver can evict device memory into GTT — host memory reached over the bus — and the run continues, slowly, with no error. The reporter of llama.cpp #27894 hit exactly that on an RDNA3 Radeon under ROCm 7.2.0, and describes an amdgpu "VRAM→GTT eviction of the model (~16.6 GB) which then could not fit in host RAM, taking the machine down." That is a 24 GB card with nothing offloaded; on this card 8.174 GiB of host RAM is already committed to experts before the driver wants any. So the failure this reserve is protecting against is not a clean OOM — it is a silent migration into a host budget that this recipe has already spent. One extra step of -ncmoe relative to a 16 GB NVIDIA card costs 0.422 GiB of host RAM and buys 0.422 GiB of the margin that keeps that migration from starting. This is a capacity argument and it transfers to any 16 GB card; the amdgpu half of it transfers to any AMD card and to no NVIDIA one.

Where the work happens: prompt processing versus generation

The single most useful thing to understand about this configuration is that the offloaded experts are not treated the same way in the two phases, and llama.cpp's scheduler says so in one rule — a rule that is in the HIP build too, which is not obvious and is worth showing.

An operation whose weights live in a host buffer is normally run on the CPU: ggml/src/ggml-backend.cpp puts it plainly, "operations with weights are preferably run on the same backend as the weights". But immediately after that the scheduler will "check if a backend with higher prio wants to offload the op", and asks the GPU. The CUDA backend answers with a single test — ggml/src/ggml-cuda/ggml-cuda.cu:5343-5346, get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size — whose threshold defaults to 32 at line 5517 (getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(...) : 32).

That rule applies on ROCm, and here is why it is not an assumption. The HIP target compiles ../ggml-cuda/*.cu wholesale, ggml-cuda.cu included. The device-interface struct at line 5380 of that file wires .offload_op to ggml_backend_cuda_device_offload_op at line 5392, and contains zero GGML_USE_HIP occurrences — against 15 elsewhere in the same file, so the guard exists and is used, just not there. One source, compiled twice, one rule. (This does not hold for the Vulkan backend, which is a different code path and is not what this page builds.)

So the same weights take two completely different paths:

• Prompt processing runs a batch of hundreds of tokens (-ub defaults to 512), so the expert matmuls are offloaded to the GPU and the expert weights are copied host → device over the PCIe link, layer by layer, for every physical batch. This phase is bus-bound. A community reporter profiling exactly this path with Nsight — a different card and a different model of this same qwen35moe architecture, at -ncmoe 26 — found that "prefill is bottlenecked by PCIe transfers, not GPU compute", with the GPU idle a large fraction of each pass waiting on those copies (ggml-org/llama.cpp#25859, open, no maintainer verdict on the finding itself).
• Token generation runs one token at a time. One is less than 32, so the offloaded expert matmuls stay on the CPU backend and those weights never cross the bus at all. This phase is bound by your system RAM bandwidth and your CPU, and the only thing moving over the link is a small activation vector per layer.

Two practical consequences:

1. Dual-channel memory and memory clock matter more than the PCIe link for the tokens per second you feel while the model is answering. Single-channel RAM roughly halves the bandwidth the decode path reads those experts at. Note that the card's own 624 GB/s of GDDR6 (AMD product specifications: 16 GB, 256-bit, up to 624 GB/s) is not the number that governs decode here — a DDR5 dual-channel host is an order of magnitude below it, and 41% of this model's weights are on the far side of that gap.
2. The link matters for time-to-first-token on long prompts. This page does not assert a lane count for this card: AMD's own product-specification page publishes memory size, type, interface width, bandwidth and compute units, and does not publish the PCIe lane count at all. Check what yours negotiated — rocm-smi --showbus names the PCIe device, and lspci -vv on that address reports LnkCap/LnkSta with the negotiated generation and width. It is a property of your card and your slot, and it is the one term in this page's reasoning that varies between machines that are otherwise identical. If the link turns out to be the constraint, --no-op-offload disables that offload rule globally, moving prompt-processing expert matmuls onto the CPU as well and removing the transfers entirely. It is a real trade, it is untested here, and it is one flag to try.

Throughput at any of these settings is unmeasured on this pair, and the ladder above is a capacity statement rather than a speed one. The two do not even have the same shape: an owner of a 16 GB card tuning a different model of the same architecture on stock llama.cpp b10644 found -ncmoe 20 to be an isolated peak rather than a plateau edge — 22 and 21 both near 36.7 tok/s, 20 jumping to 42.18, and 19 falling back to 35.33 — and was careful to add "I am not claiming a cause for this boundary; I did not profile PCIe transfers or GPU idle time." (the same thread — an RX 9070 XT under Vulkan on Windows, so a different backend on a different generation of card, and nothing there is a number for this one). Sweep the two or three rungs either side of 18 on your own machine, and please contribute what you find.

System RAM, the prompt cache, and the amdgpu eviction that lands on it

At -ncmoe 18 the expert weights hold 8.174 GiB of host RAM for the life of the process. That is the part the ladder tells you about. There are two more consumers it does not, and on an AMD box they interact.

llama-server keeps a host-side prompt cache, capped by -cram / --cache-ram at a default of 8192 MiB (common/common.h:632, flag at common/arg.cpp:1713). For an attention-only model a cache entry is a few MiB and the default is invisible. For a hybrid like this one an entry has to carry recurrent state as well as KV, so entries are two orders of magnitude larger and the cache reaches its bound in tens of requests rather than thousands.

Two numbers, and they are not the same number. This model's per-sequence recurrent state is 65,863,680 bytes — 62.8 MiB, the same figure budgeted for the GPU above, and derived. The only measured entry size is 126 MiB, from #27894, where a reporter measured a flat +126 MiB per request across forty requests on an RDNA3 Radeon under ROCm, filed it as a leak, and retracted after a contributor pointed at --cache-ram: "Confirmed — you're right, this isn't a leak." Setting --cache-ram 512 reproduced the growth and then held flat for ten consecutive requests at +507 MiB, about four entries against a 512 MiB bound. An entry therefore costs about twice one recurrent state, and nobody has instrumented what the second half is. Size a host from the measured 126 MiB, not from the derived 62.8. The report's own summary is the thing to read before sizing a machine: "the 8192 MiB default is invisible for attention-only models where entries are a few MiB, but for a hybrid SSM model each entry is ~126 MiB, so the same default can commit 8 GiB of host RAM".

Two scoping notes, because a citation that is not scoped is the failure this page is trying to avoid. It was measured on an Ornith-1.5-35B-A3B build rather than on Apodex — the transfer holds, because that file's header declares the same qwen35moe with block_count 41, nextn_predict_layers 1, full_attention_interval 4 and the same SSM geometry, so its per-sequence state is byte-for-byte the same 62.8 MiB. And it is not an AMD defect: the same reporter rebuilt from the same source with -DGGML_VULKAN=ON and measured +126.0 MiB/request against ROCm's +126.3, which puts the allocation in the model and request layer rather than in HIP.

What is AMD-specific is what the pressure collides with. In that report the host-RAM growth met an amdgpu VRAM→GTT eviction of the model, whose ~16.6 GB then "could not fit in host RAM, taking the machine down." Read that with this page's own arithmetic in front of you: at -ncmoe 18 you have already given 8.174 GiB of host RAM to experts, and an uncapped prompt cache will ask for up to 8 GiB more, on a machine where the driver may at any point decide it needs to put several more gigabytes of device memory into the same place. That is why --cache-ram 2048 is in the command above and why 32 GB is the RAM row rather than 16. --cache-ram 0 disables the cache entirely and is the right setting if you are memory-constrained and your prompts do not share prefixes.

-fit turns itself off, and that is what you want here

llama.cpp can size the offload for you: auto-fit is on by default — fit_params = true at common/common.h:476, with a per-device target of 1024 MiB at line 481 — so "not passing -fit" is not the same as "not using it". It was still true at b10690.

It does not compete with an explicit -ncmoe. The fitter's contract in common/fit.h is that "only parameters that have the same value as in llama_default_model_params are modified", and common/fit.cpp makes the buffer-override case an outright bail: it throws "model_params::tensor_buft_overrides already set by user, abort", which the caller catches and logs as a failed fit before loading the model with your parameters untouched. So passing -ncmoe gives you the explicit offload and disables the fitter, at the cost of one warning line in the log. Passing -fit off as well is harmless and removes the warning.

Prefer the explicit number anyway, for a second reason. There is an open regression report against --fit-target on a Q4_K_M of another 35B-A3B model — llama.cpp issue #27171, filed 2026-08-16, still open and still labelled bug-unconfirmed when this page was written, with a contributor's fix in flight at PR #27207, also still open and unmerged. Different card, different model — a reason to pin the offload by hand until the fix lands, not a claim about this pair.

If you would rather see what the fitter projects before committing, llama-fit-params -m <file> -c 131072 -ctk q8_0 -ctv q8_0 -np 1 prints the CLI arguments it would apply; if it comes back having reduced your context, that is the answer for your machine and the -c 65536 row above is where to go.

Results

• Speed: omitted — no figure below is a measurement of this recipe, and the omission is not for want of looking. Two throughput figures exist for this model and neither travels to this page. ghazni101/Apodex-1.1-mini-MQ4RP reports a 400-token "generation measured at 226 tok/s end-to-end for MQ4R" — validated on an RX 7900 XTX, which is gfx1100 / Navi 31, not this card, from a .mq4r file served by hipfire, which that card describes as "a Rust-native LLM inference engine for AMD RDNA GPUs". It is not a GGUF, not llama.cpp, not this card and not this quantisation, and the vision tower and MTP head were stripped at conversion. It does not even measure the file that repo is named for: the repository ships MQ4RP, the 226 is for MQ4R, and the same card goes on to "expect a similar ~10% RP decode penalty here" on the strength of a different model's pair of runs. It is the nearest thing to an AMD number that exists for this model, and it is nearest on the axis that matters least. kingjones777 publishes a four-rung ladder — 63.30, 45.05, 32.54 tok/s and a fourth build at 64.87 whose own card shows a dash for that measurement and says the run is still pending — all of it on a Ryzen AI MAX+ 395 (gfx1151), a unified-memory APU with no expert offload and no PCIe hop, under a fork whose card is headed "Stock llama.cpp will not load this file". There is no throughput figure for this file, on stock llama.cpp-HIP, at any -ncmoe setting, on any card. The space searched, on 2026-08-30: the model cards of all 22 HuggingFace repositories matching Apodex-1.1-mini (one, aykutx21/Apodex-1.1-mini-GGUF, has no README at all — its raw fetch 404s), read in full rather than pattern-matched, because a tok/s regex misses a table that prints its unit in the column header; the llama.cpp tracker, where a search for apodex returns 0 against a control of 271 for the architecture string qwen35moe; and this site's own catalogue, where /check/apodex-1-1-mini/rx-7800-xt has no benchmark. If you run this, please contribute the numbers.
• VRAM usage: 13.124 GiB derived working set at -ncmoe 18 and 131,072 tokens with q8_0 KV, leaving 2.876 GiB of the card before compute buffers, with 8.174 GiB of expert weights in system RAM. Dropping to -ncmoe 16 at 32,768 tokens gives 13.036 GiB and 7.266 GiB respectively. See /check/apodex-1-1-mini/rx-7800-xt for live data as it lands.
• Quality notes: the vendor publishes agentic evaluation scores on the model card, but they are run in Apodex AI's own harness and several of the benchmarks are the vendor's own creations, so they are a vendor claim rather than an independent measurement and are not reproduced here. The lead quant is imatrix-calibrated — the publisher ships both the calibration corpus and the imatrix alongside the ladder — which is the main reason to prefer it over an uncalibrated conversion at the same nominal tier. Note that the routed experts are already narrow (expert_feed_forward_length 512), which is part of why the sub-4-bit rungs give up more than the file sizes suggest.

For the full benchmark data, see /check/apodex-1-1-mini/rx-7800-xt.

Troubleshooting

unknown model architecture: 'qwen35moe'

Your binary predates LLM_ARCH_QWEN35MOE in src/llama-arch.cpp, which arrived in (b7989, b7990] — so this means a genuinely old build, not a merely recent one. Check llama-server --version against the release list and move to b10630 or newer. Note that architecture registration lives in src/, not in a backend, so it is not something a HIP build can lack relative to a CUDA one. Whether the kernels are there is the separate question below.

Does a HIP build on gfx1101 actually run this architecture?

Nobody has published a run of this model on this card under mainline llama.cpp, so the answer is an argument from source plus three field reports, and it is worth seeing where each stops.

From the source at b10690: the ROCm target compiles ../ggml-cuda/*.cu wholesale, so gated_delta_net.cu, ssm_conv and the rest are in the binary; ggml_backend_cuda_device_supports_op — the function that decides whether an operation runs on the GPU or falls back to the CPU — begins at line 4869 of ggml-cuda.cu and contains zero GGML_USE_HIP guards in its body, against 15 elsewhere in the file; and the warp-width helper returns 64 only for GFX8/GFX9, so RDNA3 takes the 32 branch that gfx1101 reports.

From the field, and this is the part that matters, because it is a run rather than an argument:

• #22351 is a qwen35moe model on this exact card. Its reporter ran Qwen_Qwen3.6-35B-A3B-Q4_K_L.gguf — the same architecture string, the same hybrid layer pattern — on 2× AMD Radeon RX 7800 XT (gfx1101), 16GB VRAM each, built -DGGML_HIP=ON for gfx1101 at build b8929 under ROCm 7.2.0. Across two GPUs with --tensor-split 1,1 it segfaults inside libhsa-runtime64.so during load_tensors. On one card it does not: the report states in as many words that "With single GPU (ROCR_VISIBLE_DEVICES=0) the model loads and runs correctly." That single-GPU sentence is the strongest positive evidence this page could find for the architecture on gfx1101 — searched across the llama.cpp tracker (gfx1101, 51 issues; qwen35moe ROCm, 66; both non-empty, against apodex, which returns 0) and the model cards of all 22 Apodex-1.1-mini repositories — and it is the configuration this page documents. The issue was closed as a duplicate; its only comment is from a GitHub CONTRIBUTOR writing "These are all related", so read it as one of a cluster of multi-GPU ROCm crashes and not as a verdict on the single-card path. If you have two of these cards, do not use --tensor-split for this model.
• #27698 is the report to read before you trust a recent build, and it is open with no comments and no labels. A qwen35moe GGUF loaded and generated under the official server-rocm container at b10108 and fails to load at b10615. Read the window carefully, because it is worse than "inside a bisection range": b10108 is the last build the reporter saw work and b10615 the earliest they saw fail, and the b10630 and b10690 this page names are both after b10615 — on the far side of the failure, not bracketed by it, on a report with no fix and no responder. Its card is an RX 6700 XT, gfx1031, RDNA2, 12 GB, which is a different architecture generation from this one, and its configuration is a partial layer offload (--n-gpu-layers 20 --n-cpu-moe 0 --fit off) rather than the expert offload here. It is not bisected and nobody has responded. It is not the only ROCm load failure this page cites — #22351 above is one too, and its title says so — but it is the only one that is a build regression on a single card, which is the shape that would reach a reader following these instructions. Search space for that: every issue returned by the llama.cpp tracker for qwen35moe ROCm (66) and for gfx1101 (51), read by title and, where the title was ambiguous, by body. It is not evidence about gfx1101 — but this page is a partial-offload page on ROCm, so if your build refuses to load and b10108 does load, you have found the same thing and the issue is where to say so.
• Someone analysing an unrelated qwen35moe bug writes "We have not reproduced the bug — we run this architecture on CPU and ROCm, not Metal — so this is analysis of your artifacts, not a second data point" (#27442). Their author_association is CONTRIBUTOR. It is a community statement that the architecture is run under ROCm as a matter of course, not a measurement, and they say so themselves.

What none of that establishes is that a full 131,072-token run of this checkpoint is correct on gfx1101. Run the needle test below before you rely on a long context, and tell us what happens.

It generates fluently but seems to have forgotten the start of the prompt

Worth knowing about on AMD, because nothing about it looks like a failure. There is an open cluster of reports — #27556, #27579 — in which the HIP backend silently drops the oldest part of the context while the output stays fluent and deterministic. Every report in those two threads is on gfx1151, the integrated Radeon 8060S, and the mechanism the threads converged on is specific to integrated GPUs: a commenter on #27556 puts it as "a write after read race on graph input tensors when the backend computes directly from host memory on integrated GPUs", and #27579 draws its own boundary — "Only gfx1151. Discrete RDNA/CDNA cards untested." The source agrees about the scope: ggml-cuda.cu sets a per-device integrated flag from the device property on HIP builds, and accepts a host-visible buffer for compute only when that flag is set. A discrete RX 7800 XT does not report itself as integrated, so the path those reports blame is not one this card takes.

That is a real narrowing and not a clean bill of health, and on this page there is a second reason to test rather than assume: the whole point of -ncmoe is that a large fraction of the model is being read out of host memory, which is the neighbourhood those reports are about even if the mechanism differs. So run the probe. The same commenter gives the method and it costs a single prompt: "You need a retrieval probe: put a unique marker early in the prompt and ask for it back verbatim." Put a distinctive string in the first few hundred tokens of a 100K-token prompt, ask for it back, and check the answer rather than the prose around it. Fluent output is not evidence — that is the whole point of this class of bug. A single confirmed needle test on gfx1101 with this checkpoint would be worth more than every paragraph above; if you run one, send the result.

It loads fine, then slows to a crawl or takes the machine down under a long prompt

On an NVIDIA card the symptom of an over-committed budget is an out-of-memory error. On amdgpu it may instead be an eviction of device memory into GTT, which is host memory, on a host that this recipe has already committed 8.174 GiB of — see the System RAM section and #27894, where exactly that sequence ended with the machine going down. The fix order is: cap --cache-ram (2048, or 0 if you are tight), raise -ncmoe by two or three — on this file that frees between 0.844 and 1.459 GiB depending on which blocks the steps land on — or drop -c a row on the table. If the card is also driving a display, start two steps higher than the table and check rocm-smi --showmeminfo vram after load. Loading allocates the weights and the full KV cache, but the compute buffers grow with the batch, and prompt processing additionally needs room for the expert weights it is staging onto the card.

Prompt processing is slow and the GPU looks idle

This is the expected shape of the offloaded path rather than a fault, and there are four things to try before suspecting the model. Confirm mmap is off (--load-mode none); llama.cpp prints a warning about it, and the RX 9070 XT owner cited above reported a gain on both phases from following it, larger on prompt processing (the same comment — a different card, a different model and a different backend, so a direction to test rather than a number to expect). Check what your PCIe link negotiated with rocm-smi --showbus plus lspci -vv, since prompt processing is the phase that spends it. Raising -b and -ub is the largest lever that same reporter found: 8192/2048 gave him ~1524 t/s of prompt processing against ~705 at 1024/512. Read that gap carefully before expecting it — his baseline is below the 2048 default on -b and exactly at the 512 default on -ub, and he published no row at the actual default pair, so a reader starting from defaults already begins with more logical batch than his 705 t/s row had; his nearest at-or-above-default row (2048/1024, ~1008 t/s) makes it about 1.5×. His 16384 row was slightly worse than 8192, so this is a sweep and not a direction. It costs compute buffer, which the 2.876 GiB reserve above has not been sized for, so raise them and re-read rocm-smi rather than assuming. And --no-op-offload keeps the expert matmuls on the CPU in both phases, which removes the transfers entirely and is worth an A/B if your link is narrow.

The server's host RAM climbs by ~126 MiB per request

Not a leak — it is the prompt cache filling to --cache-ram, which defaults to 8192 MiB and costs about 126 MiB per entry on this architecture. Cap it, as the command above does. If it climbs and never plateaus, your system RAM is simply short: at -ncmoe 18 more than a third of the file lives in RAM permanently, on top of the OS and on top of whatever the driver may evict into it.

Do not reach for the Vulkan backend as a workaround here

On RDNA3 the usual advice is that llama.cpp's Vulkan backend sometimes beats ROCm at token generation, and for ordinary dense models that is fair. For this architecture family it is the wrong reflex, and there are two open reports rather than one. #26795 reports Qwen3.6-35B-A3B — the same qwen35moe, the same Gated-DeltaNet operator — with Vulkan decode collapsing to ~4 t/s while prefill runs at ~3,200 t/s; that is on RDNA4 (gfx1201) under RADV, a generation newer than this card. #27237 reports a hybrid Gated-DeltaNet model producing garbage or truncated output on Vulkan on an RX 7900 XTX, reproducibly at batch size 512 and correctly at 1024 and 4096, with a third party unable to reproduce it on Intel and NVIDIA ICDs and concluding "the fault may be specific to the AMD drivers rather than to the Vulkan backend generally". Neither is this card and neither is this checkpoint — the second is a qwen35 dense derivative rather than qwen35moe, driven through koboldcpp on Windows. What carries is that the failing configuration in #27237 contains llama.cpp's own default -ub of 512, so it is not a configuration you can be confident of having avoided by leaving the defaults alone. That is an argument for staying on HIP here, not for tuning batch sizes on Vulkan. It is also why the batch-32 offload rule above was verified in the HIP build specifically: it is a CUDA-and-HIP rule and it does not describe Vulkan.

The server runs on the CPU instead of the GPU

Confirm ROCm is installed and the card is visible (rocm-smi should list the 7800 XT), and that your user is in the render and video groups — groups should show both, and the change only takes effect after logging out and back in. For a source build, confirm you configured with -DGGML_HIP=ON and that the startup banner names a ROCm device with gfx1101. You should not need HSA_OVERRIDE_GFX_VERSION on this card; if you find yourself setting it, the build is probably targeting the wrong architecture, and -DGPU_TARGETS=gfx1101 is the fix. Note that a build made for gfx1100 will not run here — Navi 31 and Navi 32 are different targets, and this is the most common way a guide written for a 7900 XTX fails on a 7800 XT.

Should you turn on the MTP head?

The GGUF carries a working multi-token-prediction block and mainline llama.cpp implements it behind --spec-type draft-mtp (the exact spelling matters; mtp alone is rejected). On this card, under offload, the honest answer is that nobody has measured it and the costs are all on the wrong side of the ledger.

The costs are concrete: block 40's tensors stop being skipped (+0.444 GiB of weights, which -ncmoe 18 does not touch — the flag only covers blocks 0 to 17); the recurrent state is multiplied by the draft depth; and a second, single-layer KV cache is allocated for the draft context, f16 by default and untouched by your -ctk. Against a 2.876 GiB reserve the weights alone are one full step of -ncmoe, spent before any of it has been shown to help.

The benefit is unknown and the nearest evidence is discouraging on both axes. unsloth's published guidance for this class of model is that "dense models are much more accelerated with MTP (1.4-2x) vs MoE models (1.15-1.25x)" (Qwen3.6 docs) — so 1.15–1.25× at best, before offload is taken into account. And under offload even that goes: an RTX 3060 owner running --n-cpu-moe 25 on a different model of this architecture with the MTP flags on reports "Absolutely same decoding speed as on ordinary Q4_K_M GGUF" (discussion #18 on that model's GGUF repo — a community thread, no org member in it). That is at least consistent with the mechanism above: if the pace is set by the CPU reading expert weights out of system RAM, drafting more tokens per step does not help, because the bottleneck is not the GPU waiting on itself. On AMD there is a third reason for caution and it is a plain absence — bounded, so you can check it: no MTP run of this qwen35moe architecture under HIP has been published on any surface searched. That space is the model cards and discussions of all 22 HuggingFace repositories matching Apodex-1.1-mini, plus the llama.cpp tracker, where searches for qwen35moe ROCm (66 results) and gfx1101 (51 results) return nothing carrying an MTP measurement on an AMD device — both non-empty, so the zero is about MTP-on-HIP and not about a query that could not match. And and both publishers who built this model specifically for AMD went the other way — ghazni101 strips the MTP draft head and kingjones777 converted with --no-mtp. Try it if you like; do not budget VRAM for a speed-up you have not seen on your own machine.

Choosing a different quant publisher

Conversions of this model are still appearing, and the pace is worth respecting: a Hub enumeration on 2026-08-30 returned 22 repositories matching Apodex-1.1-mini across eleven publishers, one of which had not existed in an enumeration run earlier the same day. Re-enumerate before you decide, and check the byte count and the file list rather than the tier name — across publishers a file named Q4_K_M spans more than a gigabyte for the same nominal tier. Within bartowski's own ladder the rungs adjacent to the lead are Q4_K_S at 19.619 GiB and IQ4_XS at 17.955 GiB; each is worth roughly two and six steps of -ncmoe respectively, which is a legitimate way to buy back host RAM rather than VRAM.

You wanted the vendor's own quantised weights

Apodex AI publishes -NVFP4, -GPTQ-Int4 and -FP8 repos. On this card they are ruled out three times over. They are vLLM/SGLang safetensors rather than GGUF, and llama.cpp cannot load them at all. They are much too large before anything else is counted — 22.548 GiB and 22.958 GiB of weights for the first two, against 16 GiB. And specific to RDNA3: the card has no FP8 and no FP4 tensor hardware, so an FP8 or NVFP4 checkpoint is not a memory-saving path here — the weights upcast on load and you pay more memory for no compute acceleration. The -NVFP4 repo is not even a 4-bit checkpoint: its hf_quant_config.json declares quant_algo: MIXED_PRECISION over 290 quantized layers, of which 250 are FP8 and only 40 — the routed experts — are NVFP4. One thing this page will not claim about them, because it is false and it was inherited from a sibling page rather than checked: their Quick Starts do not launch at eight GPUs. Re-read at the source, apodex/Apodex-1.1-mini-NVFP4 and -GPTQ-Int4 both launch --tp-size 1 and -FP8 launches --tp 1; the --tp 8 and --tensor-parallel-size 8 lines belong to the unquantised base repo alone. The three reasons above are what rule these out on a 16 GB desktop, and none of them needs the vendor to be running a datacentre.

No Ollama tag

There is no Ollama library entry for this model: ollama.com/library/apodex and /apodex-1.1-mini both return 404, and the corresponding registry.ollama.ai manifest returns 404 while the same two request shapes return 200 for qwen3.5, which is in the library. Use llama.cpp directly. If a tag appears later, please tell us.