How much VRAM does Bonsai 27B need?

About 12 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Bonsai 27B on RTX 4070: 100K Context with No KV Compression on 12GB

What You'll Build

A local, private 27B multimodal setup — text and image input — running on a 12 GB RTX 4070, with a 100,000-token context window and no KV-cache compression at all. Bonsai 27B is PrismML's 1-bit compression of Qwen3.6-27B: every weight is binary (±scale) with one FP16 scale per 128-weight group, which lands the whole 27B at 3.54 GB on disk.

The 4070 is the sweet-spot card for this model, and the reason is arithmetic rather than horsepower. PrismML's peak-memory table on the model card puts a 100K-token context at 11.6 GB with the KV cache left at full precision. That fits 12 GB. It does not fit any 8 GB card — not at Q4 KV, not at any setting. On the 3060 Ti you either cap the context or quantize the cache; here you do neither.

Bonsai is a reasoning model — each turn opens with a <think> block before the answer — and it ships a separate vision tower (0.59 GB mmproj) for image input.

Hardware data: RTX 4070 (12GB VRAM) · Bonsai 27B Q1_0 GGUF (3.54 GB weights + 0.59 GB vision tower) · 100K context uncompressed at a PrismML-reported 11.6 GB peak · See benchmark data

The vendor's model card tells you to clone their llama.cpp fork. You do not need it. The Q1_0 quantization type was merged into upstream llama.cpp in PR #21273, and the shipped file's group size (128) matches upstream's QK1_0. We loaded and ran this exact file on a stock build of upstream llama.cpp (b9960) with no vendor code involved. The fork is genuinely required for the ternary build (Q2_0), because PrismML ships that one at group size 128 while upstream's Q2_0 is group size 64 — same type name, different format, won't load. This recipe uses the 1-bit build, so upstream is all you need.

Requirements

Item	Value
GPU	RTX 4070, 12 GB VRAM
Weights	`Bonsai-27B-Q1_0.gguf` — 3.54 GB on disk (PrismML's table lists 3.79 GB resident)
Vision tower	`Bonsai-27B-mmproj-Q8_0.gguf` — 0.59 GB (optional; text-only works without it)
Runtime	Upstream llama.cpp, build b9960 or newer, CUDA enabled
Context window	262,144 tokens max; 100K is the uncompressed ceiling on 12 GB
License	Apache-2.0

Installation

1. Build upstream llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release

Confirm the build is new enough — older builds will reject the file outright:

./build/bin/llama-cli --version

2. Download the weights

pip install -U "huggingface_hub[cli]"

hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .
# optional — only if you want image input:
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-mmproj-Q8_0.gguf --local-dir .

Running

1. The headline path — 100K context, KV cache untouched

PrismML's published peak-memory table reads: 5.2 GB at 4K context, 5.6 GB at 10K, 11.6 GB at 100K, all with no KV compression. That last row is the one that matters here — it clears 12 GB, and it is the reason to reach for this card over an 8 GB one.

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 100000 -ngl 99 \
  --host 127.0.0.1 --port 8080

Leaving the cache at full precision means no quantization error stacked on top of an already 1-bit model. Given how unevenly Bonsai degrades (see Results), that is worth the VRAM.

For a one-shot from the CLI:

./build/bin/llama-cli -m Bonsai-27B-Q1_0.gguf \
  -c 100000 -ngl 99 -st \
  -f long-document.txt

Use -st (--single-turn), not -no-cnv — in current builds -no-cnv does not suppress the interactive UI, and if stdin is closed the binary will sit there printing an empty prompt forever.

2. Pushing past 100K — quantize the KV cache

PrismML reports that 4-bit KV pulls the 100K peak down to ~6.8 GB and fits the full 262K window in ~9.4 GB. Both clear 12 GB comfortably, so the entire context range of this model is available to you — the only question is whether you want to pay for it in cache precision:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 262144 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 127.0.0.1 --port 8080

Read Results before you commit to a prompt that long. The constraint at 262K is not memory on this card — it is how long you are willing to wait for ingestion.

3. Image input

./build/bin/llama-mtmd-cli -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  --image screenshot.png \
  --image-min-tokens 1024 \
  -p "What is in this image? Describe exactly what you see, including any text."

llama.cpp identifies Bonsai as a Qwen-VL model and will warn you to pass --image-min-tokens 1024; take the advice if you care about grounding or reading fine text. Budget generously for -n — the <think> block alone can eat 80+ tokens before the model says anything to you.

Results

There is no RTX 4070 measurement for this model. We have not run it on this card and we are not going to guess at a number for it. If you run it, please contribute your figures — prompt-processing and token-generation rates on the CUDA path are the single most useful missing data point for this recipe.

What exists today, and what it is worth:

Ours, on Apple Silicon (M4, Metal, upstream b9960): 50.86 ± 0.18 t/s for pp512, 14.24 ± 0.07 t/s for tg128. Different vendor, different backend, different kernel. Do not read these as 4070 numbers.
A third-party Jetson Orin run reports 82.9% accuracy at 14.7 tok/s in 3.5 GB for Q1_0. Also not an RTX card.

The shape of the problem is what transfers, not the numbers. On every backend measured so far, prompt processing on this model is slow relative to a conventional Q4_K_M of similar footprint — which is exactly the thing a 100K context stresses. The 4070 gives you the memory to hold 100K tokens uncompressed; whether it gives you the throughput to ingest 100K tokens in a tolerable amount of time is unmeasured. Treat the context ceiling as a capability, not a promise.

Quality does not degrade evenly, and the averages hide it. PrismML reports ~89.5% of the FP16 baseline. Independent testing found general knowledge and coding holding at 97–100%, while mathematics fell to ~64% and multilingual performance roughly halved. Treat Bonsai as a strong reader, describer and coder at this size, and reach for a smaller model at Q4 when you need arithmetic or non-English fluency.

Troubleshooting

Ollama can't run it

Expected, on any card. Ollama's Q1_0 support PR (#15213) was closed without merging, and users report Bonsai failing to load with a 500 error (#15359). Use llama.cpp directly — that is why this recipe does not offer an Ollama path.

`unknown model architecture` or the file won't load

Your llama.cpp is too old. Q1_0 and the qwen35 architecture both need a recent upstream build — use b9960 or newer. Don't reach for the vendor's fork; it isn't the problem.

The ternary/`Q2_0` file won't load, even on a new build

Expected. PrismML ships ternary at group size 128; upstream's Q2_0 is group size 64. The type names collide but the formats differ. Either use the 1-bit build (this recipe) or build the vendor's fork.

llama-cli hangs, printing `>` forever, pinning a core

You passed -no-cnv and closed stdin. Use -st / --single-turn instead.

Out of memory at `-c 100000`

11.6 GB is PrismML's figure on a 12 GB card — the margin is thin, and anything else on the GPU eats it. Close the browser, kill the compositor's GPU load, or drop to -c 65536. If you need the headroom back permanently, add --cache-type-k q4_0 --cache-type-v q4_0 and the same 100K drops to ~6.8 GB.

A 100K prompt takes forever to ingest

Expected, and not a bug you can configure away. See Results — prompt processing is the slow axis on this model. No 4070 figure exists, so we cannot tell you how long; if you measure one, send it in.

Image encoding takes ages

Expected — a single image took 32 s on an M4. The find_slot: non-consecutive token position warnings that scroll past during encoding are noisy but did not corrupt output in our testing.