Qwen3.8-27B on RX 7800 XT: a vision-capable 27B inside 16 GB on ROCm

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — a 27B dense vision-language model — on a single 16 GB RX 7800 XT, with the vision projector loaded and a 32,768-token context. The install is llama-server from llama.cpp built against ROCm/HIP for gfx1101, a 3-bit dynamic GGUF from unsloth, and an 8-bit KV cache.

Hardware data: RX 7800 XT (16GB VRAM) · derived working set 14.594 GiB of 16 GiB · See benchmark data

⚠️ Nothing on this page was measured on an RX 7800 XT. /check/qwen3-8-27b/rx-7800-xt returns verdict: unknown with zero benchmarks, and no measurement of this model on this card surfaced anywhere during research — not in llama.cpp's issue tracker, not in the model's 104 discussion threads, not in the 57 threads on the GGUF repo this recipe downloads from. Every number below is either a byte count read from the artifact you download or arithmetic derived from llama.cpp's own source, shown in full so you can check it. If you run this, please contribute your numbers — that is how the /check page gets real data.

⚠️ On AMD, running out of VRAM does not announce itself. Where an NVIDIA card OOMs at load, ROCm spills into GTT — system memory addressed through the GPU — and the model starts fine, then collapses mid-request. On a 1.4 GiB headroom this is the failure mode you are most likely to meet; Troubleshooting has the monitoring command and the ladder back down.

ℹ️ AMD's own guidance for this model is a bigger card than this one. Its launch post points Radeon owners at the 32 GB AI PRO R9700 and says the model "requires roughly 24GB of variable graphics memory" to run comfortably, adding that it "can also run on supported AMD hardware with more than 24 GB of Variable Graphics Memory or VRAM". That is a statement about the ~18 GB quants their LM Studio path pulls. This page does something narrower and states its arithmetic: a 3-bit build with the projector and a 32K window, inside 16 GiB.

Requirements

Where this card sits, so the absent speed number is not a mystery

Nine cards in this catalogue hold 16 GB, and they are not one performance tier. These are nameplate specifications, not measurements — for the seven NVIDIA members, memory bandwidth as published in Wikipedia's Nvidia GPU specification tables, whose rows cite TechPowerUp (which blocks direct fetching): RTX 5080 960.0, RTX 5070 Ti 896.0, RTX 4080 Super 736.3, RTX 4080 716.8, RTX 4070 Ti Super 672.3, RTX 5060 Ti 448.0 and RTX 4060 Ti 288.0 GB/s — a 3.33× span at identical capacity. (The 5060 Ti figure is carried by a three-row rowspan owned by the RTX 5060 row and covering both 5060 Ti entries; it is resolved rather than read off the adjacent line, because a table offset is exactly how a spec gets misattributed.) AMD's own specification page — first-party, and the only figure here that is not second-hand — puts the RX 7800 XT at "Up to 624 GB/s" over a "256-bit" interface at "Up to 19.5 Gbps", with 60 compute units, 120 AI accelerators and 64 MB of Infinity Cache. So within that set of eight characterised cards it is sixth: below the RTX 4070 Ti Super's 672.3 GB/s, 1.39× the RTX 5060 Ti's 448.0 and 2.17× the RTX 4060 Ti's 288.0. (The catalogue's ninth 16 GB machine, the M2 Pro, is not characterised in that comparison.)

Decode on a 27B model is memory-bound, so that ordering is the useful prior in the absence of a measurement here. It is a prior, not a number: nothing below is presented as a throughput figure for this card, and the one published benchmark of this model on any RDNA 3 GPU is on a different card and a different backend (see Results).

What ROCm changes, and what it doesn't

One backend, compiled twice. llama.cpp has no separate AMD kernel tree: ggml/src/ggml-hip/CMakeLists.txt globs ../ggml-cuda/*.cu and compiles it through HIP with GGML_USE_HIP defined. The model-level arithmetic on this site's NVIDIA pages for Qwen3.8-27B is therefore reading the same source your build compiles. What differs is the branch taken at runtime, keyed on the device.

Matrix-multiply dispatch is type-dependent here and is not on NVIDIA. ggml_cuda_should_use_mmq in mmq.cu returns true outright once turing_mma_available(cc) holds — and that predicate is GGML_CUDA_CC_IS_NVIDIA(cc) && … in common.cuh, so on a Radeon control reaches the AMD branch instead. RDNA 3's branch carries a per-type switch: Q2_K takes MMQ only up to a batch of 128, Q6_K likewise on this generation, IQ2_XS/IQ2_S likewise, everything else falls to default: return true. The Q3_K tier this recipe leads is in that default. Worth knowing before you drop a tier: Q2_K is one of the four types the switch does name.

The WMMA flash-attention kernel does not apply to this model. fattn.cu picks the AMD matrix-core path only when amd_wmma_available(cc) && gqa_opt_applies && Q->ne[0] <= 128. This model's head dimension is 256, so that condition is false and attention runs on llama.cpp's generic vector kernel during decode and its tile kernel during prefill. That is which kernel gets selected, not a measured slowdown — but do not assume this model inherits the throughput of a 128-head-dim model on the same card.

What does not transfer: the FP8 and FP4 escape hatches. RDNA 3's WMMA units take FP16, BF16, INT8 and INT4; there is no FP8 and no FP4 in hardware, which arrived with RDNA 4. On a 16 GB NVIDIA card the reflex squeeze is an FP8 or NVFP4 build; here those buy nothing, because an FP8 tensor upcasts on load. There is no flash-attn wheel to install either, and no ExLlamaV2 or Marlin path — subtract all of it. The compensation is that BF16 output is kept rather than promoted to F32 on RDNA 3, which is why this recipe uses the BF16 vision projector.

Why a 27B fits at all: the architecture

Qwen3.8-27B is not a conventional 64-layer dense transformer, and that is the whole reason a 16 GB card is in the conversation. Its config.json declares 64 layers of which 48 are linear_attention and 16 are full_attention, at indices 3, 7, 11 … 63 — the model card describes the pattern as "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))".

llama.cpp implements this as architecture qwen35 and derives the same split independently: src/models/qwen35.cpp marks a layer recurrent when (i + 1) % full_attention_interval != 0, with full_attention_interval = 4 in every published build. Only 16 layers hold a KV cache instead of 64, so KV costs a quarter of what a conventional 27B would charge; the other 48 hold a fixed-size recurrent state that does not grow with context at all.

Count the layers from 64, not from the GGUF's block_count. Publishers disagree: ggml-org's files declare 64 and ship the multi-token-prediction head separately, while unsloth, bartowski and lmstudio-community declare 65 with nextn_predict_layers = 1 and the head inline. llama.cpp reconciles them in llama-hparams.cpp with n_layer() { return n_layer_all - n_layer_nextn; }, and the MTP block is excluded from the attention cache by the same test. Sizing KV off a raw 65 would give 16.25 layers and overstate the cache by about 6%.

VRAM budget

Every figure here is either a byte count from the HuggingFace tree API or arithmetic over values read out of the model's config and llama.cpp's source.

On units, because this is where fit calculations go wrong. Everything below is GiB (2^30 bytes) on both sides of the comparison. A "16 GB" graphics card is 16 GiB = 16,384 MiB — GPU memory is quoted in binary units, unlike the decimal GB HuggingFace prints for file sizes. Reading the card's capacity as decimal would understate it by 7% and turn a fit into a miss; reading a file's decimal GB as GiB does the reverse. Two real deductions do apply, and both come out of the headroom line rather than the components: the device-local heap the driver exposes sits under the nominal capacity, and the display driver reserves more on top if a monitor is attached. Check yours before sizing the context:

amd-smi monitor --vram-usage

KV cache, per token. 16 full-attention layers × 4 KV heads × 256 head dim × 2 (K and V) = 32,768 elements per token. The config confirms the shape: num_key_value_heads: 4, head_dim: 256. At fp16 that is 65,536 bytes per token; at q8_0 (34 bytes per 32-element block) it is 34,816 bytes per token.

Recurrent state, per sequence. n_embd_r() returns (ssm_d_conv - 1) * (ssm_d_inner + 2 * ssm_n_group * ssm_d_state) = 3 × (6144 + 2×16×128) = 30,720 elements of convolution state, and n_embd_s() returns ssm_d_state * ssm_d_inner = 128 × 6144 = 786,432 elements, per layer. Both are allocated GGML_TYPE_F32. Across 48 layers for one sequence that is 48 × (30,720 + 786,432) × 4 = 156,893,184 bytes, constant regardless of context length.

The whole budget at 32,768 context, --parallel 1, q8_0 KV:

(The GiB column is rounded for display — the KV row is 1.0625 exactly — and the total is computed from the unrounded byte column, rounded once.)

That 1.406 GiB is what the HIP context, the compute buffers, the vision encoder graph and the display driver's own reservation have to live in. It is a reservation, not a measurement, and on this platform it is also the thing standing between you and a GTT spill rather than a clean error. Treat 32K as a starting point to verify rather than a guarantee; the ladder under Troubleshooting walks you down.

One thing works in your favour that the table does not credit: llama.cpp skips the model's multi-token-prediction block unless you ask for speculation — qwen35.cpp creates those tensors with int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0; — so the extra blk.64 carried inside the unsloth file is on disk but not in VRAM by default.

Picking a quant on 16 GB of Radeon

There is no first-party GGUF. Qwen's HuggingFace organisation carries exactly four Qwen3.8 repositories — Qwen3.8-27B, Qwen3.8-27B-FP8 and the two 2.4T-A95B variants — and none is a GGUF conversion, though the same org publishes 54 -GGUF repositories for other families. Re-checked by enumeration on 2026-08-16. Every GGUF below is a community conversion, so the publisher is part of what you are choosing.

Why the 4-bit tier is out, as arithmetic rather than a size floor. Plenty of 4-bit builds are small enough to load: bartowski's IQ4_XS is 15,567,824,480 B = 14.499 GiB, comfortably inside the card. What decides it is what has to sit beside the weights — projector 0.867 + recurrent 0.146 + KV at 32K q8_0 1.062 = 2.075 GiB of fixed load. Add that to the smallest uniform 4-bit build and you get 16.574 GiB, over the card before a single byte of runtime buffer. Shrinking the window does not rescue it: at 16K the same build totals 16.043 GiB, still over; at 8K it totals 15.778 GiB, leaving 0.222 GiB for every compute buffer, the HIP context, the vision graph and the driver combined. That is the reason this recipe is 3-bit, and dropping the projector to buy the 4-bit tier trades away the model's defining capability.

The 3-bit shelf, from the same tree fetch:

Your tool may call the lead file Q4_K_S. unsloth's Dynamic builds carry a wrong general.file_type — the repo's own discussion #38 enumerates it across the UD-* files and unsloth acknowledged it there. It is a label, not the tensors: the file is 12.518 GiB, which no real Q4_K_S of this model could be.

Two AMD-shaped detours that look like the right answer

Nine repositories whose names begin "ROCm" will surface if you search, and stock llama.cpp cannot load them. The ROCmFP4 / ROCmFPX family uses tensor formats from a third-party fork, and that fork's own documentation says so: "The ROCmFP4 / ROCmFPX tensor formats (ggml types 100–106) exist only in this fork." The -STRIX variants additionally target gfx1151, the Ryzen AI Max+ APU, not a discrete RDNA 3 card. Treat the prefix as a fork marker, not as "the AMD build".

AMD's own quantizations are real and are not for this card. The amd org publishes Qwen3.8-27B-Quark-AWQ-INT4-W4A16 and MXFP4/Qronos siblings — safetensors for vLLM-class runtimes, not GGUF. Two reasons they do not reach this page: the INT4 build is 19,535,853,721 B (18.194 GiB) of weights before any cache, which is over this card on its own; and its card states it needs a runtime with W4A16Int4 scheme support, linking a vLLM pull request that was still open and unmerged when this page was written.

Installation

1. Confirm your gfx target — this card is gfx1101

rocminfo | grep gfx | head -1 | awk '{print $2}'

AMD's ROCm 7.14.0 compatibility matrix lists AMD Radeon RX 7800 XT (gfx1101), alongside the RX 7700 XT and RX 7700 on the same target, with the RX 7900 series on gfx1100 and the RX 7600 on gfx1102. That last one is the trap: a widely-surfaced community setup guide for this exact card prints gfx1102 as its architecture, which is the RX 7600's target, and building -DGPU_TARGETS=gfx1102 produces code objects your card cannot run. Read the target off your own machine and cross-check it against AMD's matrix; do not take it from a guide, this one included.

2. Install ROCm

Follow AMD's ROCm quick start for Linux. Two constraints before you start: llama.cpp's HIP build asserts a floor in CMake — "At least ROCM/HIP V6.1 is required" — and AMD's system-requirements table restricts consumer Radeon cards to Ubuntu 24.04.4, Ubuntu 22.04.5, RHEL 10.1 and RHEL 9.7 (footnote 7 there; the shorter compatibility matrix carries the gfx targets but not this restriction). ROCm 7.14.0 is the current production release.

You should not need HSA_OVERRIDE_GFX_VERSION. llama.cpp's build documentation frames it as a fallback — "If your GPU is not officially supported you can use the environment variable" — and gfx1101 is officially supported. Community threads that tell 7800 XT owners to masquerade as gfx1100 predate that support; setting it turns a working configuration into an unexplained one.

3. Build llama.cpp with HIP

There is no prebuilt Linux ROCm binary — the project's release assets carry a win-rocm-7.14-x64 zip for Windows and no ROCm tarball for Linux.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release -- -j 16

That is llama.cpp's own documented invocation with the target switched to this card. Its docs note that GPU_TARGETS "is optional, omitting it will build the code for all GPUs in the current system" — naming it compiles faster and removes any doubt about which ISA you got.

Container alternative. ghcr.io/ggml-org/llama.cpp:server-rocm is built from .devops/rocm.Dockerfile, whose ROCM_DOCKER_ARCH is a fat list that includes gfx1101, on a rocm/dev-ubuntu-24.04:7.2.1-complete base. llama.cpp's docker documentation is candid that the GPU images are "not currently tested by CI beyond being built".

On build numbers. Both structural requirements — the qwen35 architecture in src/llama-arch.cpp and the qwen3vl_merger projector in tools/mtmd/clip-impl.h — are present as far back as b8001, so any current build clears the floor. One flag below is newer: --reasoning-effort is absent at b10433 and present at b10434. Prefer a recent build and pin the one you tested.

4. Download the weights and the projector

pip install -U huggingface_hub
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q3_K_XL.gguf --local-dir ./qwen38
hf download unsloth/Qwen3.8-27B-GGUF mmproj-BF16.gguf --local-dir ./qwen38

Both files must come down. Without the projector the server loads text-only and rejects image requests.

Running

./build/bin/llama-server \
  --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf \
  --mmproj ./qwen38/mmproj-BF16.gguf \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --image-min-tokens 1024 \
  --jinja \
  --no-reasoning-preserve \
  --reasoning-effort medium \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

On a HIP build the device enumerates as ROCm0. Four of those flags are doing load-bearing work on a 16 GB card and are not defaults:

• --parallel 1 is worth 0.438 GiB. common_params initialises n_parallel to 1, but that is not the value llama-server runs with: common/arg.cpp overrides it to -1 for the server example before your arguments are parsed, and tools/server/server.cpp resolves any negative value to four slots with kv_unified = true. This is backend-independent and observable on AMD hardware: the startup log in llama.cpp #27124, from a command line carrying no -np flag at all, prints n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true'. Because the recurrent state is allocated per sequence, the default costs 627,572,736 bytes instead of 156,893,184 — 0.438 GiB, or about 13,500 more tokens of context. The KV cache itself is not multiplied, because the auto path also turns on a unified cache: four slots share one window rather than dividing it. What --parallel 1 buys you is the extra recurrent copies back, and a window a concurrent request cannot evict.
• --cache-type-k q8_0 --cache-type-v q8_0 halves KV against fp16. Do not go to 4-bit KV, and do not mix the two types — see Troubleshooting.
• --image-min-tokens 1024 is what llama.cpp asks for at load time: Qwen-VL models need at least 1024 image tokens to work correctly on grounding tasks. Budget for it — at that floor and q8_0 KV, each image costs about 35 MB of cache.
• --reasoning-effort medium and --no-reasoning-preserve. Thinking is on by default at the card's xhigh level (it documents that "supported levels are xhigh, medium, and low"), and preserved thinking keeps every past reasoning trace in your window. Both are context-budget decisions on a card whose entire KV cache is the 1.062 GiB budgeted above. On a build older than b10434 the first flag does not parse — use --chat-template-kwargs '{"reasoning_effort":"medium"}'.

One default worth knowing about: llama.cpp now fits parameters to free device memory unless you pass --fit off. common/fit.h scopes it — "only parameters that have the same value as in llama_default_model_params are modified", with context size changed only if left at 0 — so the explicit --ctx-size and --n-gpu-layers above are untouched. Its sibling --fit-print on asks llama.cpp for its own estimate of the memory this configuration needs, which is worth comparing against the table above precisely because that table is derived rather than measured.

Point any OpenAI-compatible client at http://127.0.0.1:8080/v1; images go in as image_url content parts, and GET /props reports whether the build accepted the projector.

Does the vision path work on ROCm? Yes, with a named limit on what has been shown. On 2026-08-16 llama.cpp #27124 — an image-input crash report against this model — was closed after a second reporter ran the same weights and the same mmproj-BF16.gguf on a ROCm build and answered "I am able to input images and the model correctly describes their content", with a log showing the device as ROCm0; the original reporter, whose Vulkan build was crashing, switched to ROCm and confirmed it. That machine is a Ryzen AI Max+ 395 (gfx1151), not this discrete gfx1101 card, and it is a functional confirmation rather than a measurement. What underwrites the transfer is that llama.cpp's projector is not backend-specific code — tools/mtmd/clip.cpp has no CUDA/HIP branches and initialises through the generic GPU path — so the projector is one backend compiled twice, like the rest of the tree. If it fails on your build, that is new information worth filing upstream and sending to /contribute.

Text-only, if you want the context back

Dropping the projector frees 931,146,432 bytes — about 26,700 more tokens of q8_0 KV. It is a real option and it costs the model's defining capability:

./build/bin/llama-server --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf --no-mmproj \
  --n-gpu-layers 99 --ctx-size 65536 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja

Why not Ollama on this card

Ollama's library page for this model lists twelve tags and the smallest is 18 GB — there is no tag below this card's capacity, so any of them partially offloads. On AMD that is worse than it sounds: the overflow lands in GTT rather than failing, so you get a server that works and crawls. Pull the GGUF directly instead.

Results

• Speed: omitted. There is no measurement of this model on an RX 7800 XT anywhere this research reached — not on /check, not in the model's 104 HuggingFace discussion threads (enumerated individually), not in the 57 threads on the unsloth GGUF repo, not in llama.cpp's issue tracker. The nearest datapoints are all on other hardware: a Vulkan llama-bench run of this model on an RX 7900 XTX (llama.cpp #27097), a ROCm run on a Ryzen AI Max+ 395 APU (#27124), and AMD's own launch figures on a Ryzen AI Max+ 395 and a Radeon AI PRO R9700 — the last two measured, per AMD's footnote, "in llama.cpp on Windows with the Vulkan backend and MTP=4". Different cards, and in three of the four cases a different backend from the one this page installs. The bandwidth placement earlier in this page is the honest substitute. If you run this configuration, please post your numbers via /contribute so /check/qwen3-8-27b/rx-7800-xt stops being empty.
• VRAM usage: derived, not measured — 14.594 GiB of 16 GiB at 32K context with the projector loaded, itemised above. Live data, when it exists, will be at /check/qwen3-8-27b/rx-7800-xt.
• Quality notes: this card runs the model at 3 bits, and vision is the part that suffers first. In discussion #69 a community user — a reply in the thread, not its opener — reports that "the multi modal ability is inferior to 3.6 27b when comes to 4 bit quant": a comparison to the previous generation at 4 bits, one tier above what fits here, with no equivalent report at 3 bits either way. Expect the text side to hold up better than the vision side, and verify OCR and fine grounding work before relying on them.

For the full benchmark data, see /check/qwen3-8-27b/rx-7800-xt.

Troubleshooting

It runs out of memory quietly

On AMD this is the first thing to suspect, because it does not look like an error. llama.cpp #26432 documents it on an RX 7900 XT under HIP, running the previous generation of this same qwen35 architecture: the model loads cleanly, then the first large request pushes buffers into GTT and decode falls from 42–46 tok/s to 18. The reporter watched /sys/class/drm/card1/device/mem_info_gtt_used climb from 0.59 GB to 14.65 GB during a single prefill and drop back afterwards, and — having tested hipMalloc directly and found that it fails cleanly rather than spilling — concluded that llama.cpp's load-time accounting misses runtime-peak buffers, so "on this platform the overage shows up as a silent GTT cliff instead of a load-time error". Watch it:

watch -n1 'cat /sys/class/drm/card0/device/mem_info_gtt_used; amd-smi monitor --vram-usage'

If GTT moves during a request, you are over the line, and the fact that the model loaded proves nothing. The ladder, in the order that costs you least:

1. Drop --ctx-size to 16384. At q8_0 that halves the KV line to 0.531 GiB and takes the total to 14.063 GiB.
2. Confirm --parallel 1 is actually on the command line — the auto default is 4 and costs 0.438 GiB of recurrent state.
3. Drop the projector with --no-mmproj and run text-only: 0.867 GiB.
4. Drop a quant tier: unsloth's Q3_K_S is 11.711 GiB and UD-IQ3_XXS is 11.095 GiB. Do not chase this much further down. The one comparative measurement published at this tier is vendor-run and self-interested — a quant publisher's own KLD sweep against 20 community files — but its publisher-independent finding is worth having: "below 10 GB every quant of this model degrades fast".

Prompt processing collapses to tens of tokens per second

You quantised the KV cache to 4 bits, or mixed the two cache types. This is not a CUDA-only trap despite where it was reported: the gates live in fattn.cu, which your HIP build compiles from the same source. ggml_cuda_fattn_kv_type_supported returns false for Q4_1/Q5_0/Q5_1 under #ifndef GGML_CUDA_FA_ALL_QUANTS, and the same guard returns BEST_FATTN_KERNEL_NONE whenever K->type != V->type; with no kernel available the scheduler quietly relocates attention to the CPU backend and prints nothing. The default HIP build compiles exactly the same four FlashAttention vector instances as the default CUDA one — f16-f16, q4_0-q4_0, q8_0-q8_0 and bf16-bf16 — as the else branch of ggml-hip/CMakeLists.txt shows. It was measured on an RTX 3090 in llama.cpp #27109, where prefill fell from 991–1276 t/s to 34–106 t/s on a mixed q4_1/q8_0 cache; read that as the size of the cliff on another vendor's card, not as throughput here. The rule that survives is match the two types. This recipe's q8_0/q8_0 is in the default set, so no flag and no newer build is needed; if you want the exotic combinations the fix is a compile flag, not a version bump:

cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DGGML_CUDA_FA_ALL_QUANTS=ON

ROCm does not see the card, or the build produces the wrong ISA

Two distinct causes with the same symptom. On Linux, the usual one is a GPU_TARGETS that does not match rocminfo — see step 1, and note that this card's neighbours in the product line are on a different target than the RX 7600. On Windows, there is a packaging defect rather than anything you did: llama.cpp #26996 shows ggml-hip.dll in the win-rocm-7.14-x64 release archive importing a hipblas.dll the archive does not ship, so --list-devices prints nothing and exits 0. A project contributor's answer there is to install ROCm 7.14 from AMD's Windows tarball and complete its post-installation environment steps; a reporter who copied the whole ROCm bin\ directory next to the binaries got his GPU recognised.

Images crash the server

Which backend you are on matters more than which card you have. Every image-input crash report found for this model is on a Vulkan build (#27124, on an AMD Ryzen AI MAX+ 395 under Windows), and that thread was closed by moving to a ROCm build. If you hit an image crash, try the ROCm binary before you try a different quant.

The model thinks for minutes on a trivial question

Widely reported for this model, and on this card it is a memory problem as much as a latency one: a runaway trace is what exhausts a 32K window whose entire KV cache is 1.062 GiB. The lever is on the model card — reasoning_effort accepts xhigh (the default), medium and low, and thinking can be turned off per request.

Turning on MTP makes things worse here

The model ships a multi-token-prediction head inline in unsloth's file, and --spec-type draft-mtp works against it with no separate download. On a 16 GB card, leave it off. It costs VRAM you do not have: llama_memory_recurrent allocates mem_size * (1 + n_rs_seq) rows of recurrent state, so --spec-draft-n-max 4 takes that line from 156,893,184 to 784,465,920 bytes — 0.584 GiB — and it adds a 17th attention layer's worth of KV, because llama.cpp gives the MTP block a plain attention cache rather than the hybrid one. The payoff is doubtful on this backend family besides: the one ROCm log of this model records draft acceptance = 0.35929, and the CUDA reports in #26750 sit in the same band against roughly 92% on Vulkan.

Pushing past 32K

It is possible, and it is where the GTT warning above earns its place. A reader in the unsloth repo's 16 GB thread reports a stable 58,880-token window on a Q3_K_XL build by running q8_0 K against an f16 V with --fit off and --kv-unified — but he does not name his card, and he also runs MTP, which this page recommends against. Read it as an existence proof for the window rather than a setting for this card. The arithmetic for this page's own combination says 49,152 tokens totals 15.125 GiB, leaving 0.875 GiB for everything else. Verify with --fit-print on and the GTT watch before you rely on it.