How much VRAM does Bonsai 27B need?

About 6 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Bonsai 27B on RTX 5060: the Cheapest Current-Gen Card That Runs a 27B

What You'll Build

A local, private 27B multimodal setup — text and image input — running on the entry-level card of the current generation. Bonsai 27B is PrismML's 1-bit compression of Qwen3.6-27B: every weight is binary (±scale) with one FP16 scale per 128-weight group, which lands the whole 27B at 3.54 GB on disk. That is smaller than most 8B models at Q4.

The RTX 5060 is the cheapest Blackwell card you can buy, and 8 GB is the tier where "which models can I even load" is normally the whole conversation — it tops out around 8B–14B at Q4. Bonsai walks straight past that: a 27B parameter count on a card that costs less than the model's own reputation suggests. The catch is not the weights. It is the context.

Bonsai is a reasoning model — each turn opens with a <think> block before the answer — and it ships a separate vision tower (0.59 GB mmproj) for image input.

Hardware data: RTX 5060 (8GB VRAM, Blackwell) · Bonsai 27B Q1_0 GGUF (3.54 GB weights + 0.59 GB vision tower) · comfortable fit, context is the only budget that bites · See benchmark data

The vendor's model card tells you to clone their llama.cpp fork. You do not need it. The Q1_0 quantization type was merged into upstream llama.cpp in PR #21273, and the shipped file's group size (128) matches upstream's QK1_0. We loaded and ran this exact file on a stock build of upstream llama.cpp (b9960) with no vendor code involved. The fork is genuinely required for the ternary build (Q2_0), because PrismML ships that one at group size 128 while upstream's Q2_0 is group size 64 — same type name, different format, won't load. This recipe uses the 1-bit build, so upstream is all you need.

Requirements

Item	Value
GPU	RTX 5060, 8 GB VRAM (Blackwell)
Weights	`Bonsai-27B-Q1_0.gguf` — 3.54 GB on disk (PrismML's table lists 3.79 GB resident)
Vision tower	`Bonsai-27B-mmproj-Q8_0.gguf` — 0.59 GB (optional; text-only works without it)
Runtime	Upstream llama.cpp, build b9960 or newer, CUDA enabled
Context window	262,144 tokens on paper; 8 GB caps you well short of that — see Running
License	Apache-2.0

Installation

1. Build upstream llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build -j --config Release

120 is the RTX 5060's compute capability (sm_120, Blackwell). Build this with CUDA 12.8, not the newest toolkit you can find — NVIDIA's own Blackwell migration guide recommends 12.8 for llama.cpp on this architecture, and newer is actively worse here: CUDA 13.1 builds the Blackwell MMQ kernel into a segfault. This is the one place in this recipe where "just update everything" is the wrong instinct.

Confirm the build is new enough — older builds will reject the file outright:

./build/bin/llama-cli --version

Blackwell is a newer target than Ada or Ampere, so a stale CUDA toolkit is a more likely stumbling block here than on a 3060 Ti. If the build completes but the card never lights up, see Troubleshooting.

2. Download the weights

pip install -U "huggingface_hub[cli]"

hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .
# optional — only if you want image input:
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-mmproj-Q8_0.gguf --local-dir .

Running

1. Text, with the context capped to fit 8 GB

The weights are only 3.54 GB, so on an 8 GB card the KV cache is the entire question. PrismML publishes a peak-memory table on the model card: 5.2 GB at 4K context, 5.6 GB at 10K, and 11.6 GB at 100K with no KV compression. The first two fit an 8 GB card; the third does not, on this card or any other 8 GB card. If 100K uncompressed is what you came for, that is a 12 GB conversation, not a 5060 one.

./build/bin/llama-cli -m Bonsai-27B-Q1_0.gguf \
  -c 8192 -ngl 99 -st \
  -p "Explain why 1-bit quantization loses less on factual recall than on arithmetic."

Use -st (--single-turn), not -no-cnv — in current builds -no-cnv does not suppress the interactive UI, and if stdin is closed the binary will sit there printing an empty prompt forever.

2. Reaching for long context — quantize the KV cache

PrismML reports that 4-bit KV pulls the 100K peak down to ~6.8 GB and fits the full 262K window in ~9.4 GB. On 8 GB, that puts 100K within reach and leaves the full window out:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 100000 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 127.0.0.1 --port 8080

6.8 GB on an 8 GB card is a real margin but not a generous one — a desktop compositor and a browser can eat the difference.

3. Image input

./build/bin/llama-mtmd-cli -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  --image screenshot.png \
  --image-min-tokens 1024 \
  -p "What is in this image? Describe exactly what you see, including any text."

llama.cpp identifies Bonsai as a Qwen-VL model and will warn you to pass --image-min-tokens 1024; take the advice if you care about grounding or reading fine text. Budget generously for -n — the <think> block alone can eat 80+ tokens before the model says anything to you.

Results

There is no RTX 5060 measurement for this model. We have not run it on this card and we will not invent a number for it. Blackwell is new enough that nobody has published Bonsai figures on it at all, as far as we can find. If you have this card, you are in a good position to be the first — please contribute your figures.

What exists today, and what it is worth:

Ours, on Apple Silicon (M4, Metal, upstream b9960): 50.86 ± 0.18 t/s for pp512, 14.24 ± 0.07 t/s for tg128. Different vendor, different backend, different kernel. Do not read these as 5060 numbers.
A third-party Jetson Orin run reports 82.9% accuracy at 14.7 tok/s in 3.5 GB for Q1_0. Also not an RTX card.

The pattern to expect, without a number attached. On every backend measured so far, prompt processing on this model runs slow relative to a conventional Q4_K_M of similar footprint — the ratio between prompt-processing and generation rates is much tighter than you would expect. Whether Blackwell's CUDA kernels change that is unmeasured. Plan for a model that is fast to load and slow to think, and let the first real 5060 measurement — maybe yours — replace this paragraph with a figure.

Quality does not degrade evenly, and the averages hide it. PrismML reports ~89.5% of the FP16 baseline. Independent testing found general knowledge and coding holding at 97–100%, while mathematics fell to ~64% and multilingual performance roughly halved. Treat Bonsai as a strong reader, describer and coder at this size, and reach for a smaller model at Q4 when you need arithmetic or non-English fluency.

Troubleshooting

Ollama can't run it

Expected, on any card. Ollama's Q1_0 support PR (#15213) was closed without merging, and users report Bonsai failing to load with a 500 error (#15359). Use llama.cpp directly — that is why this recipe does not offer an Ollama path.

The build succeeds but llama.cpp never uses the GPU

The RTX 5060 is compute capability 12.0 (sm_120). A CUDA toolkit that cannot target it builds cleanly and then quietly falls back to CPU — you get no error, just glacial output. Check nvcc --version, rebuild with -DCMAKE_CUDA_ARCHITECTURES=120, and confirm the offload by watching the layer-count line in the load log with -ngl 99.

It segfaults on the newest CUDA toolkit

Don't reach for the latest toolkit. NVIDIA's Blackwell migration guide recommends CUDA 12.8 for llama.cpp on Blackwell; CUDA 13.1 compiles the Blackwell MMQ kernel into a segfault. Downgrade the toolkit, don't upgrade it.

`unknown model architecture` or the file won't load

Your llama.cpp is too old. Q1_0 and the qwen35 architecture both need a recent upstream build — use b9960 or newer. Don't reach for the vendor's fork; it isn't the problem.

The ternary/`Q2_0` file won't load, even on a new build

Expected. PrismML ships ternary at group size 128; upstream's Q2_0 is group size 64. The type names collide but the formats differ. Either use the 1-bit build (this recipe) or build the vendor's fork.

llama-cli hangs, printing `>` forever, pinning a core

You passed -no-cnv and closed stdin. Use -st / --single-turn instead.

Out of memory once the conversation gets long

The weights aren't the problem — the KV cache is. Lower -c, or add --cache-type-k q4_0 --cache-type-v q4_0. Per PrismML's table, 10K uncompressed is 5.6 GB and 100K with 4-bit KV is ~6.8 GB; the full 262K window (~9.4 GB) does not fit 8 GB at any setting.

Image encoding takes ages

Expected — a single image took 32 s on an M4. The find_slot: non-consecutive token position warnings that scroll past during encoding are noisy but did not corrupt output in our testing.