Bonsai 27B on Apple M2 Pro: a 27B Multimodal Model on 16GB Unified Memory via 1-bit GGUF + Metal

What You'll Build

A local, private 27B multimodal setup — text and image input — running on an Apple M2 Pro with 16 GB of unified memory. Bonsai 27B is PrismML's 1-bit compression of Qwen3.6-27B: every weight is binary (±scale) with one FP16 scale per 128-weight group, which lands the whole 27B at 3.54 GB on disk. That is smaller than most 8B models at Q4.

Bonsai is a reasoning model — each turn opens with a <think> block before the answer — and it ships a separate vision tower (0.59 GB mmproj) for image input.

The install story on Apple is genuinely short. We ran this exact GGUF on Apple Silicon (M4, 16 GB) on a stock Homebrew build of upstream llama.cpp b9960 — Metal backend, no fork, no patches. It loaded, generated, and did vision. brew install llama.cpp is the whole setup step.

Hardware data: Apple M2 Pro, 16 GB unified memory · Bonsai 27B Q1_0 GGUF (3.54 GB weights + 0.59 GB vision tower) · comfortable fit; context and the working-set ceiling are the budgets that bite · See benchmark data

The vendor's model card tells you to clone their llama.cpp fork. You do not need it. The Q1_0 quantization type was merged into upstream llama.cpp in PR #21273, and the shipped file's group size (128) matches upstream's QK1_0. We loaded and ran this exact file on a stock Homebrew build of upstream llama.cpp (b9960) with no vendor code involved. The fork is genuinely required for the ternary build (Q2_0), because PrismML ships that one at group size 128 while upstream's Q2_0 is group size 64 — same type name, different format, won't load. This recipe uses the 1-bit build, so upstream is all you need.

PrismML also ships an MLX pack and claims their ternary 2-bit version runs on stock MLX — we did not verify that path and this recipe does not use it. Given the fork claim above was wrong in one direction and their format matrix omits a mainline-compatible file they actually ship, treat any unverified vendor routing advice with suspicion. The GGUF/Metal path below is the one we ran ourselves.

Requirements

Item	Value
Chip	Apple M2 Pro, 16 GB unified memory
Weights	Bonsai-27B-Q1_0.gguf — 3.54 GB
Vision tower	Bonsai-27B-mmproj-Q8_0.gguf — 0.59 GB (optional; text-only works without it)
Runtime	Upstream llama.cpp, build b9960 or newer, Metal
Context window	262,144 tokens (fits — see Running)
License	Apache-2.0

Unified memory is not 16 GB of VRAM. macOS shares that pool with everything else, and Metal enforces a working-set ceiling well below the installed total. On our 16 GB Apple machine llama.cpp reported recommendedMaxWorkingSetSize = 12713.12 MB — about 12.4 GB usable, roughly 78% of installed. Every memory figure below should be read against that ceiling, not against 16.

Installation

1. Install llama.cpp

brew install llama.cpp

That's it. The Homebrew bottle is upstream llama.cpp with Metal already enabled — no CUDA toolkit, no CMake flags, no vendor fork. Confirm the build is new enough (Q1_0 support predates b9960, but older builds will reject the file outright):

llama-cli --version

2. Download the weights

pip install -U "huggingface_hub[cli]"

hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .
# optional — only if you want image input:
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-mmproj-Q8_0.gguf --local-dir .

Running

1. Text

llama-cli -m Bonsai-27B-Q1_0.gguf \
  -c 8192 -ngl 99 -st \
  -p "Explain why 1-bit quantization loses less on factual recall than on arithmetic."

Use -st (--single-turn), not -no-cnv — in current builds -no-cnv does not suppress the interactive UI, and if stdin is closed the binary will sit there printing an empty prompt forever.

2. Long context — where 16 GB of unified memory actually pays off

The weights are only 3.54 GB, so the KV cache is the entire question. PrismML publishes a peak-memory table on the model card: 5.2 GB at 4K context, 5.6 GB at 10K, and 11.6 GB at 100K with no KV compression. With 4-bit KV they report ~6.8 GB at 100K and the full 262K window in ~9.4 GB.

Line those up against the ~12.4 GB working-set ceiling and this is the interesting part of the recipe: on a 16 GB Apple machine the entire ladder fits, including the full 262K context window at 4-bit KV, with ~3 GB of headroom. An 8 GB discrete card cannot say that. The one config to be careful with is 100K uncompressed at 11.6 GB — that clears 12.4 GB by ~0.8 GB, which is not the margin you want to hand macOS. Quantize the KV instead:

llama-server -m Bonsai-27B-Q1_0.gguf \
  -c 262144 -ngl 99 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --host 127.0.0.1 --port 8080

Memory is the budget you have; throughput is the one you don't — see Results before you actually feed it 262K tokens.

3. Image input

llama-mtmd-cli -m Bonsai-27B-Q1_0.gguf \
  --mmproj Bonsai-27B-mmproj-Q8_0.gguf \
  --image screenshot.png \
  --image-min-tokens 1024 \
  -p "What is in this image? Describe exactly what you see, including any text."

llama.cpp identifies Bonsai as a Qwen-VL model and will warn you to pass --image-min-tokens 1024; take the advice if you care about grounding or reading fine text. Budget generously for -n — the <think> block alone can eat 80+ tokens before the model says anything to you.

Results

Bonsai's OCR survives 1-bit compression well. Handed a dense dark-theme screenshot with no hint about its contents, it read the navigation, the page subheading, and six separate filter counts off the image correctly.

We have no M2 Pro speed measurement, and we will not extrapolate one. Our numbers come from an Apple M4 (base chip), 16 GB, on upstream llama.cpp b9960 with Metal: pp512 = 50.86 ± 0.18 t/s and tg128 = 14.24 ± 0.07 t/s. The M2 Pro is a different chip — different generation, different GPU core count, different memory bandwidth — and guessing a number for it from an M4 figure, in either direction, is exactly the mis-attribution this catalogue refuses to do. Cite those figures as the M4's, and treat them only as evidence that the Metal path works and roughly what order of magnitude to expect from Apple Silicon in general. No M2 Pro measurement exists yet. If you take one, please contribute it.

The 262K context headline is throughput-bound, not memory-bound — this is the thing to understand before you commit. Prompt processing on this model is far slower than a conventional Q4_K_M of similar footprint. On the M4, at 50.86 t/s for pp512, a 10K-token prompt takes roughly three minutes just to ingest before a single token comes back; a 262K prompt is an afternoon. The 16 GB of unified memory buys you the room for the full window, not the patience.

Image encoding is slow too. A single image took 32 s to encode on the M4. Again: M4, not M2 Pro.

Quality does not degrade evenly, and the averages hide it. PrismML reports ~89.5% of the FP16 baseline. Independent testing on a Jetson Orin found general knowledge and coding holding at 97–100%, while mathematics fell to ~64% and multilingual performance roughly halved. Treat Bonsai as a strong reader, describer and coder at this size, and reach for a smaller model at Q4 when you need arithmetic or non-English fluency.

Troubleshooting

Ollama can't run it

Expected, on any machine. Ollama's Q1_0 support PR (#15213) was closed without merging, and users report Bonsai failing to load with a 500 error (#15359). On a Mac this is the first thing most people reach for, so: use llama.cpp directly — brew install llama.cpp is all you need.

unknown model architecture or the file won't load

Your llama.cpp is too old. Q1_0 and the qwen35 architecture both need a recent upstream build — use b9960 or newer (brew upgrade llama.cpp). Don't reach for the vendor's fork; it isn't the problem. (Yes, qwen35 — Qwen3.6 declares model_type: qwen3_5 internally. The version number is marketing; the architecture is 3.5.)

The ternary/Q2_0 file won't load, even on a new build

Expected. PrismML ships ternary at group size 128; upstream's Q2_0 is group size 64. The type names collide but the formats differ. Either use the 1-bit build (this recipe) or build the vendor's fork.

llama-cli hangs, printing > forever, pinning a core

You passed -no-cnv and closed stdin. Use -st / --single-turn instead.

ggml_metal_graph_compute: command buffer failed or the process dies once the context grows

You've hit the Metal working-set ceiling, not the 16 GB label. Our 16 GB Apple machine reported ~12.4 GB usable. Lower -c, or add --cache-type-k q4_0 --cache-type-v q4_0 — 4-bit KV takes the full 262K window down to ~9.4 GB per PrismML's table, comfortably under the ceiling.

Image encoding takes ages

Expected — a single image took 32 s on an M4. The find_slot: non-consecutive token position warnings that scroll past during encoding are noisy but did not corrupt output in our testing.