Bonsai 27B on RTX 3060 Ti: a 27B Multimodal Model on 8GB via 1-bit GGUF + llama.cpp

What You'll Build

A local, private 27B multimodal setup — text and image input — running on an 8 GB RTX 3060 Ti. Bonsai 27B is PrismML's 1-bit compression of Qwen3.6-27B: every weight is binary (±scale) with one FP16 scale per 128-weight group, which lands the whole 27B at 3.54 GB on disk. That is smaller than most 8B models at Q4, on a card that normally tops out around 8B–14B.

Bonsai is a reasoning model — each turn opens with a <think> block before the answer — and it ships a separate vision tower (0.59 GB mmproj) for image input.

Hardware data: RTX 3060 Ti (8GB VRAM) · Bonsai 27B Q1_0 GGUF (3.54 GB weights + 0.59 GB vision tower) · comfortable fit, context is the only budget that bites · See benchmark data

The vendor's model card tells you to clone their llama.cpp fork. You do not need it. The Q1_0 quantization type was merged into upstream llama.cpp in PR #21273, and the shipped file's group size (128) matches upstream's QK1_0. We loaded and ran this exact file on a stock Homebrew build of upstream llama.cpp (b9960) with no vendor code involved. The fork is genuinely required for the ternary build (Q2_0), because PrismML ships that one at group size 128 while upstream's Q2_0 is group size 64 — same type name, different format, won't load. This recipe uses the 1-bit build, so upstream is all you need.

Requirements

Installation

1. Build upstream llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build -j --config Release

Confirm the build is new enough — Q1_0 support predates b9960, but older builds will reject the file outright:

./build/bin/llama-cli --version

2. Download the weights

pip install -U "huggingface_hub[cli]"

hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .
# optional — only if you want image input:
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-mmproj-Q8_0.gguf --local-dir .

Running

1. Text, with the context capped to fit 8 GB

The weights are only 3.54 GB, so on an 8 GB card the KV cache is the entire question. PrismML publishes a peak-memory table on the model card: 5.2 GB at 4K context, 5.6 GB at 10K, and 11.6 GB at 100K with no KV compression. The first two fit an 8 GB card; the third does not.

./build/bin/llama-cli -m Bonsai-27B-Q1_0.gguf \
  -c 8192 -ngl 99 -st \
  -p "Explain why 1-bit quantization loses less on factual recall than on arithmetic."

Use -st (--single-turn), not -no-cnv — in current builds -no-cnv does not suppress the interactive UI, and if stdin is closed the binary will sit there printing an empty prompt forever.

2. Reaching for long context — quantize the KV cache

PrismML reports that 4-bit KV pulls the 100K peak down to ~6.8 GB and fits the full 262K window in ~9.4 GB. On 8 GB, that puts 100K within reach and leaves the full window out:

./build/bin/llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 100000 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 127.0.0.1 --port 8080

3. Image input

./build/bin/llama-mtmd-cli -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  --image screenshot.png \
  --image-min-tokens 1024 \
  -p "What is in this image? Describe exactly what you see, including any text."

llama.cpp identifies Bonsai as a Qwen-VL model and will warn you to pass --image-min-tokens 1024; take the advice if you care about grounding or reading fine text. Budget generously for -n — the <think> block alone can eat 80+ tokens before the model says anything to you.

Results

Bonsai's OCR survives 1-bit compression well. Handed a dense dark-theme screenshot with no hint about its contents, it read the navigation, the page subheading, and six separate filter counts off the image correctly.

The 262K context headline is throughput-bound, not memory-bound — this is the thing to understand before you commit. Prompt processing on this model is far slower than a conventional Q4_K_M of similar footprint. Measured on Apple Silicon (M4, Metal, upstream b9960): 50.86 ± 0.18 t/s for pp512, against 14.24 ± 0.07 t/s for tg128. At that rate a 10K-token prompt takes roughly three minutes just to ingest, before a single token comes back. Those figures are Apple-measured and the CUDA kernel may well behave differently — no RTX 3060 Ti measurement exists yet. If you take one, please contribute it.

Quality does not degrade evenly, and the averages hide it. PrismML reports ~89.5% of the FP16 baseline. Independent testing on a Jetson Orin found general knowledge and coding holding at 97–100%, while mathematics fell to ~64% and multilingual performance roughly halved. Treat Bonsai as a strong reader, describer and coder at this size, and reach for a smaller model at Q4 when you need arithmetic or non-English fluency.

Troubleshooting

Ollama can't run it

Expected, on any card. Ollama's Q1_0 support PR (#15213) was closed without merging, and users report Bonsai failing to load with a 500 error (#15359). Use llama.cpp directly — that is why this recipe does not offer an Ollama path.

unknown model architecture or the file won't load

Your llama.cpp is too old. Q1_0 and the qwen35 architecture both need a recent upstream build — use b9960 or newer. Don't reach for the vendor's fork; it isn't the problem.

The ternary/Q2_0 file won't load, even on a new build

Expected. PrismML ships ternary at group size 128; upstream's Q2_0 is group size 64. The type names collide but the formats differ. Either use the 1-bit build (this recipe) or build the vendor's fork.

llama-cli hangs, printing > forever, pinning a core

You passed -no-cnv and closed stdin. Use -st / --single-turn instead.

Out of memory once the conversation gets long

The weights aren't the problem — the KV cache is. Lower -c, or add --cache-type-k q4_0 --cache-type-v q4_0.

Image encoding takes ages

Expected — a single image took 32 s on an M4. The find_slot: non-consecutive token position warnings that scroll past during encoding are noisy but did not corrupt output in our testing.