How much VRAM does Bonsai 27B need?

About 10 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Bonsai 27B on RTX 4090: the Full 262K Window Plus the Vision Tower

What You'll Build

A local, private 27B multimodal setup — text and image input — running on a 24 GB RTX 4090 with the model's entire 262,144-token context window loaded and the vision tower resident alongside it, and room left over. Bonsai 27B is PrismML's 1-bit compression of Qwen3.6-27B: every weight is binary (±scale) with one FP16 scale per 128-weight group, which lands the whole 27B at 3.54 GB on disk.

On a 24 GB card the usual framing inverts. There is no fit problem to solve here — PrismML's peak-memory table on the model card puts the full 262K window at ~9.4 GB with 4-bit KV, which is under half the card. The interesting question on the 4090 is not what can I squeeze in but what can I stop compromising on: full-precision KV cache, the vision tower loaded permanently, generous batch sizes, and a second model resident beside it if you want one.

Bonsai is a reasoning model — each turn opens with a <think> block before the answer — and it ships a separate vision tower (0.59 GB mmproj) for image input.

Hardware data: RTX 4090 (24GB VRAM) · Bonsai 27B Q1_0 GGUF (3.54 GB weights + 0.59 GB vision tower) · full 262K window at a PrismML-reported ~9.4 GB with 4-bit KV · See benchmark data

The vendor's model card tells you to clone their llama.cpp fork. You do not need it. The Q1_0 quantization type was merged into upstream llama.cpp in PR #21273, and the shipped file's group size (128) matches upstream's QK1_0. We loaded and ran this exact file on a stock build of upstream llama.cpp (b9960) with no vendor code involved. The fork is genuinely required for the ternary build (Q2_0), because PrismML ships that one at group size 128 while upstream's Q2_0 is group size 64 — same type name, different format, won't load. This recipe uses the 1-bit build, so upstream is all you need.

Requirements

Item	Value
GPU	RTX 4090, 24 GB VRAM
Weights	`Bonsai-27B-Q1_0.gguf` — 3.54 GB on disk (PrismML's table lists 3.79 GB resident)
Vision tower	`Bonsai-27B-mmproj-Q8_0.gguf` — 0.59 GB (keep it loaded; you have the room)
Runtime	Upstream llama.cpp, build b9960 or newer, CUDA enabled
Context window	262,144 tokens — the full window, ~9.4 GB with 4-bit KV per PrismML
License	Apache-2.0

Installation

1. Build upstream llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release

Confirm the build is new enough — older builds will reject the file outright:

./build/bin/llama-cli --version

2. Download the weights

pip install -U "huggingface_hub[cli]"

hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-mmproj-Q8_0.gguf --local-dir .

Grab the mmproj unconditionally on this card. On an 8 GB box it is a tradeoff; here 0.59 GB is noise.

Running

1. The full 262K window, with the vision tower loaded

PrismML reports the complete 262,144-token context at ~9.4 GB with 4-bit KV. Add the vision tower and you are still under 10 GB on a 24 GB card:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  -c 262144 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 127.0.0.1 --port 8080

That is the headline configuration: every token the model can hold, plus image input, in well under half the card.

2. Un-quantized KV, further out than any smaller card can go

PrismML's uncompressed rows read 5.2 GB at 4K, 5.6 GB at 10K, 11.6 GB at 100K. On 24 GB, the 100K row — the one that defines a 12 GB card's ceiling — leaves you more than half the GPU free:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  -c 100000 -ngl 99 \
  --host 127.0.0.1 --port 8080

PrismML's table stops at 100K for uncompressed KV. There is clearly headroom above it on this card, and full-precision cache past 100K is worth trying — but we have no published figure for where it lands, and we will not extrapolate one for you. Raise -c and watch nvidia-smi; if you map the curve, contribute it.

For a one-shot from the CLI:

./build/bin/llama-cli -m Bonsai-27B-Q1_0.gguf \
  -c 100000 -ngl 99 -st \
  -f long-document.txt

Use -st (--single-turn), not -no-cnv — in current builds -no-cnv does not suppress the interactive UI, and if stdin is closed the binary will sit there printing an empty prompt forever.

3. Image input

./build/bin/llama-mtmd-cli -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  --image screenshot.png \
  --image-min-tokens 1024 \
  -p "What is in this image? Describe exactly what you see, including any text."

llama.cpp identifies Bonsai as a Qwen-VL model and will warn you to pass --image-min-tokens 1024; take the advice if you care about grounding or reading fine text. Budget generously for -n — the <think> block alone can eat 80+ tokens before the model says anything to you.

Results

There is no RTX 4090 measurement for this model. We have not run it on this card and we are not going to invent a number for it. If you run it, please contribute your figures — a 4090 is the card most likely to expose whether Bonsai's slow prompt processing is a kernel problem or a format problem, and nobody has published that yet.

What exists today, and what it is worth:

Ours, on Apple Silicon (M4, Metal, upstream b9960): 50.86 ± 0.18 t/s for pp512, 14.24 ± 0.07 t/s for tg128. Different vendor, different backend, different kernel. Do not read these as 4090 numbers.
A third-party Jetson Orin run reports 82.9% accuracy at 14.7 tok/s in 3.5 GB for Q1_0. Also not an RTX card, and an embedded part at that.

The honest read on the 262K headline. Memory is genuinely not the constraint on this card — PrismML's ~9.4 GB figure settles that. But on every backend measured so far, prompt processing on this model is slow relative to a conventional Q4_K_M of similar footprint, and a 262K prompt is the maximal stress on exactly that axis. Whether a 4090's CUDA path makes a quarter-million-token ingest practical is unmeasured. The window is real; the wall-clock cost of filling it is unknown.

Quality does not degrade evenly, and the averages hide it. PrismML reports ~89.5% of the FP16 baseline. Independent testing found general knowledge and coding holding at 97–100%, while mathematics fell to ~64% and multilingual performance roughly halved. Worth weighing on a 24 GB card specifically: you have the VRAM to run a 12–14B model at Q4 instead, and for arithmetic or non-English work that is very likely the better trade. Bonsai earns its place here as a long-context reader, describer and coder, not as a general-purpose default.

Troubleshooting

Ollama can't run it

Expected, on any card. Ollama's Q1_0 support PR (#15213) was closed without merging, and users report Bonsai failing to load with a 500 error (#15359). Use llama.cpp directly — that is why this recipe does not offer an Ollama path.

`unknown model architecture` or the file won't load

Your llama.cpp is too old. Q1_0 and the qwen35 architecture both need a recent upstream build — use b9960 or newer. Don't reach for the vendor's fork; it isn't the problem.

The ternary/`Q2_0` file won't load, even on a new build

Expected. PrismML ships ternary at group size 128; upstream's Q2_0 is group size 64. The type names collide but the formats differ. Either use the 1-bit build (this recipe) or build the vendor's fork.

llama-cli hangs, printing `>` forever, pinning a core

You passed -no-cnv and closed stdin. Use -st / --single-turn instead.

Out of memory on 24 GB

Rare enough that it usually means something else. Check you are not requesting un-quantized KV far past 100K — PrismML publishes no figure above that row, and the cache grows with context. Add --cache-type-k q4_0 --cache-type-v q4_0 and the full 262K drops to ~9.4 GB.

The GPU sits mostly idle and it still feels slow

That is consistent with the shape of this model rather than a misconfiguration — prompt processing is the slow axis. -ngl 99 already puts every layer on the card. There is no 4090 measurement to compare against, so if yours looks wrong you have nothing to check it against; publishing a figure helps the next person.

Image encoding takes ages

Expected — a single image took 32 s on an M4. The find_slot: non-consecutive token position warnings that scroll past during encoding are noisy but did not corrupt output in our testing.