Bonsai 27B on RX 7900 XTX: a 27B Multimodal Model on AMD via 1-bit GGUF + llama.cpp (ROCm)

What You'll Build

A local, private 27B multimodal setup — text and image input — running on an RX 7900 XTX. Bonsai 27B is PrismML's 1-bit compression of Qwen3.6-27B: every weight is binary (±scale) with one FP16 scale per 128-weight group, which lands the whole 27B at 3.54 GB on disk. On a 24 GB card that is not a squeeze — it is roughly a seventh of the card.

Bonsai is a reasoning model — each turn opens with a <think> block before the answer — and it ships a separate vision tower (0.59 GB mmproj) for image input.

Hardware data: RX 7900 XTX (24GB VRAM, gfx1100) · Bonsai 27B Q1_0 GGUF (3.54 GB weights + 0.59 GB vision tower) · memory is not the constraint on this card · See benchmark data

Two corrections to what the vendor tells you, and one finding of our own.

1. You do not need PrismML's llama.cpp fork. The model card says to clone it "for the Q1_0_g128 hybrid-attention kernels". The Q1_0 type was merged into upstream llama.cpp in PR #21273 (CPU, 2026-04-06), followed by Metal (04-08), Vulkan (04-10) and CUDA (04-15). The shipped file's group size (128) matches upstream's QK1_0. The fork is genuinely required for the ternary build (Q2_0) — PrismML ships that at group size 128 while upstream's Q2_0 is group size 64, so the type names collide but the formats don't. This recipe uses the 1-bit build, so upstream is all you need.

2. ROCm is not on PrismML's support matrix. It works anyway. PrismML's format matrix lists Q1_0 as merged for CPU/Metal/CUDA/Vulkan and says nothing about ROCm/HIP; the model card mentions no AMD path at all. That reads as "AMD users must use Vulkan". It is misleading. llama.cpp's HIP backend has no kernels of its own — it compiles the CUDA sources. ggml/src/ggml-hip/CMakeLists.txt globs ../ggml-cuda/*.cu and builds them with LANGUAGE HIP, so the Q1_0 CUDA kernel is the ROCm kernel. There is no separate ROCm port to wait for.

3. gfx1100 specifically is covered. In ggml/src/ggml-cuda/common.cuh, RDNA3 is defined as 0x1100 — "RX 7000, minimum for WMMA" — and amd_wmma_available() returns true for RDNA3, which routes the RX 7900 XTX to the rdna4 MMQ config table. That table carries explicit GGML_TYPE_Q1_0 tuning rows, and the Q1_0 tile loaders are gated on AMD_MFMA_AVAILABLE || TURING_MMA_AVAILABLE || AMD_WMMA_AVAILABLE — the last of which is defined for RDNA3. Caveat, stated plainly: that is a reading of the source tree, not a measured run. See Results.

Requirements

Item	Value
GPU	RX 7900 XTX, 24 GB VRAM, gfx1100 (RDNA3 / Navi 31)
Weights	Bonsai-27B-Q1_0.gguf — 3.54 GB
Vision tower	Bonsai-27B-mmproj-Q8_0.gguf — 0.59 GB (optional; text-only works without it)
Runtime	Upstream llama.cpp, build b9988 or newer, HIP/ROCm enabled
Context window	262,144 tokens — and on 24 GB it actually fits (see Running)
License	Apache-2.0

Installation

1. Build upstream llama.cpp with HIP

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release \
  && cmake --build build --config Release -- -j 16

gfx1100 is the RX 7900 XTX/XT/GRE target. Confirm yours if you're unsure:

rocminfo | grep gfx | head -1 | awk '{print $2}'

On RDNA3 you can also enable the rocWMMA flash-attention path, which needs the rocwmma-dev headers:

  -DGGML_HIP_ROCWMMA_FATTN=ON

Why b9988 and not just "recent": Q1_0 CUDA/HIP kernels have existed since roughly b8800 (2026-04-15), but the RDNA3→WMMA config mapping described above arrived with the MMQ kernel-configuration refactor merged 2026-07-13, first released in b9988. That is the build where we can point at the source and show gfx1100 selecting a Q1_0-tuned config. Older builds may well work; we have not traced their dispatch, so b9988 is the floor we'll actually stand behind.

./build/bin/llama-cli --version

2. Download the weights

pip install -U "huggingface_hub[cli]"

hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .
# optional — only if you want image input:
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-mmproj-Q8_0.gguf --local-dir .

3. If ROCm gives you trouble — Vulkan is a real fallback

Upstream ships a genuine Q1_0 Vulkan shader (ggml/src/ggml-vulkan/vulkan-shaders/dequant_q1_0.comp, wired into the shader generator), so the Vulkan backend is a legitimate path on this card, not a downgrade to CPU:

cmake -B build-vk -DGGML_VULKAN=ON
cmake --build build-vk -j --config Release

Vulkan is the easier build and the one PrismML's matrix implicitly points AMD users toward. ROCm is the one we'd try first on gfx1100 — but since neither is measured here, if ROCm misbehaves, Vulkan costs you one cmake to rule it out.

Running

1. Text

./build/bin/llama-cli -m Bonsai-27B-Q1_0.gguf \
  -c 8192 -ngl 99 -st \
  -p "Explain why 1-bit quantization loses less on factual recall than on arithmetic."

Use -st (--single-turn), not -no-cnv — in current builds -no-cnv does not suppress the interactive UI, and if stdin is closed the binary will sit there printing an empty prompt forever.

2. Long context — where 24 GB actually pays off

This is the part that differs from every 8 GB Bonsai recipe. PrismML's peak-memory table on the model card reads 5.2 GB at 4K, 5.6 GB at 10K and 11.6 GB at 100K with no KV compression — the 100K figure that forces 8 GB cards into 4-bit KV fits this card outright:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 100000 -ngl 99 \
  --host 127.0.0.1 --port 8080

PrismML further reports that 4-bit KV fits the full 262K window in ~9.4 GB, which also fits with room to spare:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 262144 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 127.0.0.1 --port 8080

PrismML does not publish an unquantized 262K figure, and we're not going to extrapolate one for you — if you measure it, send it in.

3. Image input

./build/bin/llama-mtmd-cli -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  --image screenshot.png \
  --image-min-tokens 1024 \
  -p "What is in this image? Describe exactly what you see, including any text."

llama.cpp identifies Bonsai as a Qwen-VL model and will warn you to pass --image-min-tokens 1024; take the advice if you care about grounding or reading fine text. Budget generously for -n — the <think> block alone can eat 80+ tokens before the model says anything to you.

Results

No RX 7900 XTX measurement exists — not ours, not anyone's. We are not going to print a number we didn't take. What we can tell you is what we verified and how:

• The ROCm code path exists and is arch-gated to include gfx1100. That is established by reading upstream's source tree (ggml-hip/CMakeLists.txt, ggml-cuda/common.cuh, ggml-cuda/mmq-config-rdna4.cuh), not by running it. Source-reading tells you a kernel will compile and dispatch; it does not tell you the output is numerically correct or fast. The CUDA PR's own author noted AMD support was a partial surprise ("works for AMD in some cases"), and a reviewer had to correct the WMMA guard — that history is a reason to verify on hardware, not to assume.
• The only real numbers we hold are Apple's. On an M4 (Metal, upstream b9960): 50.86 ± 0.18 t/s for pp512, against 14.24 ± 0.07 t/s for tg128. Prompt processing on this model is slow relative to a conventional Q4_K_M of similar footprint — at that rate a 10K-token prompt is roughly three minutes of ingestion before a single token comes back. This is Metal on Apple Silicon and says nothing reliable about RDNA3. A different backend, a different memory system, a different kernel.
• The one third-party run we trust is also not AMD. Independent testing on a Jetson Orin covers quality, on NVIDIA embedded hardware.

If you own this card, a llama-bench run is the single most useful thing anyone could contribute to this page — contribute it, and say whether you built ROCm or Vulkan.

Quality does not degrade evenly, and the averages hide it. PrismML reports ~89.5% of the FP16 baseline (76.11 average across 15 thinking-mode benchmarks). The Jetson Orin testing found general knowledge and coding holding at 97–100%, while mathematics fell to ~64% and multilingual performance roughly halved. Treat Bonsai as a strong reader, describer and coder at this size, and reach for a smaller model at Q4 when you need arithmetic or non-English fluency. That split is a property of the compression, not of the backend — it will follow the model onto AMD.

Troubleshooting

unknown model architecture or the file won't load

Your llama.cpp is too old. Q1_0 and the qwen35 architecture both need a recent upstream build — use b9988 or newer. Don't reach for the vendor's fork; it isn't the problem.

Wrong/garbage output on ROCm, but the model loads

This is the failure mode to watch for on gfx1100, because the AMD path is source-verified rather than hardware-verified. Rebuild with -DGGML_VULKAN=ON and compare the same prompt. If Vulkan is coherent and ROCm isn't, you've found a real kernel bug — please report it upstream and tell us. You can also force the non-MMQ path to isolate it:

cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DGGML_CUDA_FORCE_CUBLAS=ON

The build compiles for the wrong GPU / no kernel image is available

GPU_TARGETS didn't match your card. Check with rocminfo | grep gfx | head -1 | awk '{print $2}' and rebuild. Omitting -DGPU_TARGETS builds for every GPU in the system, which is slower to compile but harder to get wrong.

The ternary/Q2_0 file won't load, even on a new build

Expected. PrismML ships ternary at group size 128; upstream's Q2_0 is group size 64. The type names collide but the formats differ. Either use the 1-bit build (this recipe) or build the vendor's fork — and note the fork's AMD story is its own unverified question.

llama-cli hangs, printing > forever, pinning a core

You passed -no-cnv and closed stdin. Use -st / --single-turn instead.

Out of memory

On 24 GB with a 3.54 GB model this should not happen below 100K context. If it does, you likely have -c set near 262144 without KV quantization — add --cache-type-k q4_0 --cache-type-v q4_0.

Ollama can't run it

Expected, on any card. Ollama's Q1_0 support PR (#15213) was closed without merging, and users report Bonsai failing to load with a 500 error (#15359). Use llama.cpp directly — that's why this recipe doesn't offer an Ollama path.