Nanbeige4.2-3B on Apple M2 Pro: the context ladder for 16 GB of unified memory

What You'll Build

A local OpenAI-compatible server running Nanbeige4.2-3B on an Apple M2 Pro, at a 32,768-token context with a quantized KV cache. This is the tightest Apple envelope this site covers, and the recipe is entirely about one trade: the model advertises a 262,144-token window, its 22 decoder blocks execute twice per forward pass so the cache costs double what its config file suggests, and a 16 GB Mac hands its GPU less memory than the number on the box. Those three facts decide which rung of the ladder you take.

Hardware data: Apple M2 Pro (16 GB unified memory, 10.667 GiB GPU-addressable) · Q4_K_M weights 2.398 GiB + q8_0 KV at 32,768 tokens 2.922 GiB + reserved logits 0.317 GiB = 5.637 GiB derived · no Apple benchmark submitted yet · See benchmark data

⚠️ The KV cache is 2× what the layer count suggests. config.json sets num_loops: 2, so the 22 physical blocks run twice and each pass keeps its own KV entries — 44 cache layers, not 22. llama.cpp is explicit about it: src/models/nanbeige.cpp expands the logical layer count "Expand logical layer count before load_tensors() allocates layers / KV." and comments the weight-sharing loop "Share physical weights across loops; each slot still has its own KV index." Budget 176.0 KiB per token at f16, not 88.0 KiB.

Requirements

Component	Minimum	This recipe
GPU	Apple silicon, Metal	Apple M2 Pro, 16 GB unified memory — not measured on this chip; the budget below is derived from file bytes and the runtime's own allocation rule (/contribute)
Unified memory	16 GB — the smallest Apple configuration that clears the lead rung	16 GB, of which 10.667 GiB is GPU-addressable at the macOS default
Storage	2.575 GB / 2.398 GiB (Q4_K_M)	2,574,807,904 bytes on disk; 2.782 GiB if you take Q5_K_M instead
Software	macOS Sonoma 14+, llama.cpp ≥ b10153	Homebrew llama.cpp (currently 10330) or a source build

The number that decides this recipe: 10.667 GiB, not 16

Apple silicon has no dedicated video memory, and the marketing capacity is not what the GPU gets. Metal exposes recommendedMaxWorkingSetSize, and the inference runtimes treat it as a hard ceiling. On a 16 GB Mac that value is 11,453,246,122 bytes = 10.667 GiB — exactly two-thirds of the machine's 16 GiB.

That is not a rule of thumb. Three independent Metal logs land on the same number:

machine	log line	in GiB
Apple M4, 16 GB	"recommendedMaxWorkingSetSize = 11453.25 MB" (whisper.cpp #3493)	10.667
Apple M1, 16 GB	"recommendedMaxWorkingSetSize = 10922.67 MB" (llama-cpp-python #687)	10.667
Apple M2, capacity unstated	same 11453.25 figure (llama.cpp #14527)	10.667

The two figures look different and are the same number twice, which is worth knowing before you compare your own log against someone else's. Current ggml prints this field as max_working_set_size / 1e6 — decimal MB — at ggml-metal-device.m:946. The 2023-era build behind the M1 report divided by 1024.0 / 1024.0 and labelled the result "MB" as well (ggml-metal.m:250 at tag b1180). 11,453.25 decimal MB and 10,922.67 MiB are both 10.667 GiB.

The machines that report it name the capacity themselves: "M4 mac mini with 16gb ram" and "MacBook Pro with 16GB of RAM". Two silicon generations apart, two print conventions apart, one ratio — and it matches the two-thirds figure that Peddals' VRAM-tuning write-up documents for machines under 64 GB, whose own 32 GB log (22906.50 MB, which that author reads as 21.33 GB) is likewise exactly two-thirds of 32 GiB.

Practical consequence: budget against 10.667 GiB, and remember the remaining 5.3 GiB is where macOS, your browser and your editor live. On a 64 GB Mac the addressable share is the only binding constraint; on a 16 GB Mac the operating system's own footprint is co-binding, which is why the comfortable band here is well under the ceiling rather than just under it.

Runtime choice: llama.cpp-Metal leads — and MLX is where the only measurement is

This recipe leads llama.cpp with the Metal backend, and it is worth saying plainly that this is a provenance call rather than a technical one, and that it costs you the only Apple number that exists for this model.

Why llama.cpp. Native support was merged into mainline on 2026-07-27; the architecture is registered as nanbeige, Metal is the default backend on macOS, and you can pin a version. It also gives you the KV-precision flags that this entire recipe turns on.

The MLX shelf is busier than it looks, and mostly unloadable. Searching MLX builds returns 27 repositories (enumerated through the HuggingFace API on 2026-08-09; all 27 declare "model_type": "nanbeige"). Twenty-three of them cannot be loaded by stock mlx-lm at all — it resolves an architecture by importing mlx_lm.models.<model_type>, and there is no nanbeige.py among the 121 files in mlx_lm/models/. The custom_code tag does not rescue them; mlx-lm never reads HuggingFace's auto_map. The remaining four declare a model_file key in their own config.json, which mlx-lm imports under trust_remote_code, so they load on stock mlx-lm with --trust-remote-code: the three jishnuvenugopal/Nanbeige4.2-3B-mlx-{4,6,8}bit repos ship nanbeige.py (14,157 bytes) and WaveCut/Nanbeige4.2-3B-heretic-MLX-DWQ-4bit ships an independently written nanbeige_mlx.py (10,284 bytes).

On this machine the MLX 4-bit build is the one with evidence behind it. Its weights are 2,345,769,894 bytes (2.185 GiB, marginally under the GGUF), and its author published a reproducible agentic-readiness harness run on a 16 GB Apple laptop — see Results. If what you want is the configuration somebody has actually measured on this class of machine, take this path:

pip install mlx-lm
mlx_lm.generate --model jishnuvenugopal/Nanbeige4.2-3B-mlx-4bit \
  --trust-remote-code --prompt "Which number is bigger, 9.11 or 9.8?"

The trade is what you are executing: mainline C++ that upstream reviews and you can pin to a build number, versus one individual's Python model definition that self-pins to mlx-lm internals and whose card does not claim bit-exact parity with the reference. Both are honest options; this recipe documents the first and gives you the second in full.

The memory arithmetic does not change between them. That MLX port's make_cache returns num_loops * num_hidden_layers cache slots — 44 — and its README computes the full-context cost at ~47 GB, which is the same 44.000 GiB the table below states in binary units. Four independent implementations (the HuggingFace reference, llama.cpp, mlx-swift-lm and this port) allocate the same 44 slots, so the budget below is an architecture cost, not a llama.cpp quirk. Only the flag names differ: because the model supplies make_cache, mlx-lm's --max-kv-size is inert, and --kv-bits is the equivalent of llama.cpp's cache-type flags.

Three things you can skip. The model card's llama.cpp section clones the vendor's own fork and configures the build with a CUDA backend flag — mainline superseded that fork and there is no CUDA on a Mac. The card's Ollama section builds a forked Ollama; stock Ollama has no library entry for this model at all, and its MLX engine would not apply here regardless: the Ollama MLX announcement (v0.19, 30 March 2026) says "Please make sure you have a Mac with more than 32GB of unified memory.", so a 16 GB Mac stays on the llama.cpp/Metal backend. And the org publishes -FP8 and -GPTQ-Int8 variants, neither of which has an Apple execution path — as with bitsandbytes, AWQ, Marlin and ExLlamaV2, there is no Metal kernel for any of them. The GGUF K-quants below are the Apple equivalent.

Installation

1. Install llama.cpp with Metal

brew install llama.cpp
llama-cli --version

The Homebrew formula is currently at build 10330; anything at or above b10153 contains the Nanbeige architecture. Metal needs no flags on macOS — it is the default backend. To build from source instead:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

2. Fetch the GGUF

llama.cpp downloads and caches HuggingFace repos itself, so there is no separate step — but on a 16 GB machine you want the flags from the next section, not a bare launch. To hold the file yourself:

pip install -U huggingface_hub

hf download owao/Nanbeige4.2-3B-GGUF Nanbeige4.2-3B-Q4_K_M.gguf \
  --local-dir ~/models/nanbeige4.2-3b

hf is the current name of the HuggingFace CLI — it was renamed from huggingface-cli in huggingface_hub v0.34.0. On an older install the same command is huggingface-cli download, which is why the upgrade flag is there.

There is no first-party GGUF and no first-party MLX build. The Nanbeige org publishes ten repositories and all ten are safetensors-only, so every quant you can run is a community one. owao/Nanbeige4.2-3B-GGUF is the one to use: 13 quants, and the uploader re-uploaded the whole set on 2026-07-28 to pick up the vendor's chat-template fix. That matters because a quant is frozen at its upload timestamp, and this set was first published on release day, before the fix landed.

Running

llama-server \
  -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 32768 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 \
  --temp 0.6 --top-p 0.95 --top-k 20

The server comes up on http://127.0.0.1:8080 with an OpenAI-compatible /v1/chat/completions endpoint and a browser UI at the same address. The KV cache is allocated up front, so an over-ambitious context fails in the first few seconds rather than mid-conversation.

That is 2.398 GiB of weights plus 2.922 GiB of quantized cache plus 0.317 GiB of reserved logits = 5.637 GiB, a little over half the 10.667 GiB this Mac addresses — which leaves the rest of the compute buffer its room and leaves the machine usable while the model is loaded. The logits reservation is the compute buffer's dominant term on this model and it is easy to miss, because it is set by --ubatch-size rather than by --ctx-size: see the memory-budget section below. The sampler values are the model card's own defaults for reasoning and chat; for agentic and tool-use work the card recommends --temp 1.0, but read the tool-call entry under Troubleshooting first.

--cache-type-k / --cache-type-v are the long forms of -ctk / -ctv. Keep them equal — see Troubleshooting.

The other two rungs worth knowing

Twice the context, at the same cache precision:

llama-server -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M --host 127.0.0.1 --port 8080 \
  --ctx-size 65536 --cache-type-k q8_0 --cache-type-v q8_0 -ngl 99

That is 8.559 GiB — inside the pool, past the comfortable band. Or four times the context for almost the same money, by halving the cache precision instead:

llama-server -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M --host 127.0.0.1 --port 8080 \
  --ctx-size 131072 --cache-type-k q4_0 --cache-type-v q4_0 -ngl 99

8.902 GiB — a 131,072-token window for 0.344 GiB more than the 65,536-token one above. Both cache types are supported by the Metal attention kernel; what nobody has published is what q4_0 costs this model in output quality, so treat that rung as a capability to evaluate on your own prompts rather than a default. If you measure it, contribute the result.

The memory budget — the loop tax on a small Mac

Where the parameters are. config.json describes 22 layers of hidden_size 3072 with 48 query heads over 8 KV heads at head_dim 128, an intermediate size of 10752, an untied 166,144-token vocabulary, and num_loops: 2. That sums to 4,169,800,704 parameters in total, of which 3,149,011,968 are non-embedding — which closes exactly against the parameter count HuggingFace reports for the safetensors and settles the naming: the "3B" counts non-embedding parameters. The exact closure with one copy of the stack is itself proof that the second loop costs nothing in weights, and the Q4_K_M file carries 201 tensors (22 blocks × 9, plus embedding, output norm and LM head) rather than 44 blocks' worth.

Where the memory goes. The loop is not free in cache terms. With 8 KV heads at 128 dimensions, one layer stores 2,048 K+V elements per token. load_arch_hparams() sets hparams.n_layer_all = n_layer_phys * n_loops = 44, and the allocator in src/llama-kv-cache.cpp reads exactly that field — const uint32_t n_layer = hparams.n_layer_all; — creating one K and one V tensor per slot. So per token:

44 × 2048 × 2 bytes = 180,224 bytes = 176.0 KiB at f16, against 88.0 KiB for the same stack run once.

cache type	bytes/token	@32,768	@65,536	@131,072	@262,144
f16 (default)	180,224	5.500 GiB	11.000 GiB	22.000 GiB	44.000 GiB
q8_0	95,744	2.922 GiB	5.844 GiB	11.688 GiB	23.375 GiB
q4_0	50,688	1.547 GiB	3.094 GiB	6.188 GiB	12.375 GiB

The third term, and it is not the cache. Weights and KV are not the whole Metal bill. The compute buffer holds one reserved tensor that is large on this model specifically: the logits. llama.cpp sizes its worst-case graph once at startup, and src/llama-context.cpp asks for n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max) rows of it at line 625, then reserves the prompt-processing graph with that number at line 629. n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch, so with stock settings it is min(min(32768, 512), 2048) = 512 rows, each a full f32 distribution over the vocabulary. The graph's result_output is the LM-head matmul, so it is [n_vocab, n_outputs] in f32, and this model's vocabulary is 166,144 tokens:

512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB

Two properties make it worth a column of its own. It is thirteen times the attention scratch discussed below, and --ctx-size does not move it — --ubatch-size does. Halving -ub halves it; going from a 32K window to a 128K one leaves it untouched. It is also unusually large here: a model with a 32,000-token vocabulary would reserve 62.5 MiB for the same graph. The oversized vocabulary that explains this model's parameter count also explains its compute buffer.

Adding weights and logits, against the 10.667 GiB this Mac addresses:

configuration	weights	KV	logits	total	verdict
Q4_K_M + f16 @ 32K	2.398	5.500	0.317	8.215 GiB	Fits, but spends 77% of the pool, most of it on a cache you can halve
Q4_K_M + q8_0 @ 32K	2.398	2.922	0.317	5.637 GiB	Fits comfortably — this recipe
Q5_K_M + q8_0 @ 32K	2.782	2.922	0.317	6.021 GiB	Fits — spend the headroom on weights
Q8_0 + q8_0 @ 32K	4.130	2.922	0.317	7.369 GiB	Fits, but see the bandwidth note below
Q4_K_M + q8_0 @ 64K	2.398	5.844	0.317	8.559 GiB	Inside the pool; you will feel it
Q4_K_M + q4_0 @ 128K	2.398	6.188	0.317	8.902 GiB	Inside the pool; coarser cache
Q4_K_M + q8_0 @ 128K	2.398	11.688	0.317	14.402 GiB	Over
Q4_K_M + q4_0 @ 256K	2.398	12.375	0.317	15.090 GiB	Over — at every cache precision

This is also what settles the 16 GB floor rather than assuming it. Below 64 GB the addressable share is exactly two thirds — the rule established earlier in this recipe, which every capacity logged here is consistent with — so an 8 GB Apple machine addresses exactly 5.333 GiB. On weights and cache alone the lead rung is 5.320 GiB, which clears that by 13.8 MiB — a quarter of one percent of the budget, which is less a margin than an artefact of what got counted. With the logits reservation counted the same rung is 5.637 GiB, which is 311 MiB over what such a machine can address, before a single byte of attention scratch. That turns an undecided into a decision, and it is why this recipe's floor is 16 GB rather than 8.

The last row is the honest headline of this card. The model card states that "The model supports a context length of up to 262,144 tokens (256K).", and on this machine that window is unreachable: 12.375 GiB of q4_0 cache plus 2.398 GiB of weights plus 0.317 GiB of logits is 15.090 GiB against a 10.667 GiB pool, and raising the wired limit that far on a 16 GiB machine would leave macOS about 0.9 GiB. The author of the MLX port reached the same conclusion from the other direction, calling the full context "unreachable on a 16 GB machine". The escape hatch that a discrete-GPU owner reaches for here — keep the weights on the GPU and push the cache into system RAM — is not a capacity win on Apple, because there is no second pool: the same DRAM backs both, and the only thing that changes is that the cache leaves the Metal backend and the attention reading it leaves the GPU with it.

The last term on top of the 5.637 GiB — and two things Metal does not charge you for. That figure is weights, cache and the logits reservation. One allocation is left, and it is the small one: the attention kernel's own scratch, which shares the compute buffer with the logits and adds roughly 24.6 MiB to it. On this backend that scratch is small and, the part worth knowing, almost entirely independent of --ctx-size. ggml_backend_metal_buffer_type_get_alloc_size appends exactly three regions to each flash-attention output — "some operations require additional memory for fleeting data" — and ggml-metal-ops.cpp sizes all three: a padding region of 64 cache positions' worth of K and V rows, with no context term; a mask block-skip bitmap at one byte per 64 cache positions per 8 query rows, the only context-linear term, which is 32 KiB at 32,768 tokens and 128 KiB at 131,072; and a per-workgroup result buffer of 4 × 32 × n_head × 32 × (head_dim + 2) bytes — 24.375 MiB for this model's 48 heads — again with no context term.

Neither of the two things a reader coming from a discrete card would brace for applies here. Quantizing the cache costs no dequantisation copy on Metal. The CUDA flash-attention path stages an extra f16 buffer of ggml_nelements(K) × 2 bytes whenever the K cache is not already f16 (fattn-common.cuh), so on that backend a quantized cache hands part of its saving back, in proportion to the context. The Metal encoder has no equivalent — the three regions above are its complete extra allocation set for this op and none of them is a copy of K or V. The q8_0 and q4_0 rows in the table above are therefore the whole cache cost on this machine, which is what lets the lead rung be quoted as a flat 5.637 GiB rather than a figure that grows with the window. And the attention mask is not in the Metal budget at all. llm_graph_input_attn_kv::set_input fills it by writing straight to tensor->data, so it asserts ggml_backend_buffer_is_host(self_kq_mask->buffer) (llama-graph.cpp:450) — and ggml_backend_metal_buffer_type_shared_is_host() returns a hard-coded false (ggml-metal.cpp:275), so the mask lands on the CPU backend instead. On Apple that does not mean "somewhere else in the machine" — there is one pool of DRAM and the mask's bytes still come out of your 16 GB. It means they do not count against the 10.667 GiB Metal ceiling. This is the one place where unified memory makes the usual host-versus-device accounting misleading, and it happens to be misleading in your favour.

This is a design decision, not an implementation artifact. The team tested KV sharing across loop passes and rejected it — "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2." (discussion #18, Nanbeige team member leran1995). Sharing would have halved the cache; they kept the model quality. No runtime flag can undo that — only cache quantization moves this number.

What the loop costs in bandwidth. Token generation streams weights, and the loop makes the decoder stack stream twice. From the Q4_K_M tensor table the 22 blocks are 1,865,048,064 bytes and the Q6_K LM head is 418,682,880, so a generated token moves 2 × 1,865,048,064 + 418,682,880 = 4,148,779,008 bytes ≈ 3.864 GiB — about 1.82× what a same-sized model without the loop would move. On a bandwidth-bound machine this is close to a throughput ratio. It is also why the Q8_0 row above, while it fits, is the wrong trade here: at this tier bytes-per-token is the scarce resource, not gigabytes-on-disk — and the one measured quant ladder that exists for this model, quoted in Results, puts its 4-bit rung well ahead of its 8-bit one.

Results

• Speed: no measurement exists for this model on the llama.cpp-Metal path on any Apple chip — not the model card, not the technical report (neither publishes a tokens-per-second figure at all), not the 27 discussions on the canonical repo, not the three on the GGUF repo, and not two targeted web searches for M-series numbers. One Apple measurement does exist, on the other runtime and a different chip, and it is close enough to be worth quoting with its labels attached: the author of the MLX port published a 30-case agentic eval on a machine he records as "Apple M1 Pro, 10-core, 16 GB unified memory", greedy decoding, one warm-up pass, with the raw per-case results committed alongside (nanbeige-mlx-eval). At MLX 4-bit he reports 35.1 tok/s aggregate decode, ~2.2 s time-to-first-token with tools, and a 27/30 pass rate; at 8-bit, 20.7 tok/s. Four things bound how far that travels: it is a single community source, it is the MLX 4-bit build rather than this recipe's GGUF, it is short tool-use contexts rather than the 32K window above, and the chip is an M1 Pro — one generation older than the M2 Pro at the same 16 GB, so for a bandwidth-bound decode it is a floor this machine should beat, not an estimate of it. Treat it as community-reported, single-source. If you measure this pair, please contribute the numbers so /check/nanbeige4-2-3b/m2-pro stops being empty.
• Unified memory usage: 5.637 GiB derived for the configuration above — 2,574,807,904 bytes of Q4_K_M weights, 3,137,339,392 bytes of q8_0 KV cache and 340,262,912 bytes of reserved logits — against 10.667 GiB addressable, a 5.030 GiB surplus. That is a derivation from file bytes, the runtime's 44-layer allocation rule and its worst-case graph reservation, not a measured peak; one term sits on top of it, the attention scratch, sized in the section above at roughly 24.6 MiB and near-flat in context. The nearest measured Apple figure is again from the MLX eval, whose harness records mx.get_peak_memory() per case and reports 2,850.9 MiB of peak Metal allocator usage at 4-bit on short tool-use prompts. Note its results table labels that column "peak RSS" while the per-run report calls it peak allocator memory; the code populates it from the Metal allocator, which is the meaningful one. See /check/nanbeige4-2-3b/m2-pro.
• Quality notes: the same eval found tool selection and argument extraction essentially flat from 4-bit to 8-bit — "Lower quants are faster, not slower." — which on a bandwidth-bound 16 GB machine is the argument for staying at the Q4_K_M rung and spending the saved memory on context. Two caveats from the vendor's own card: its published benchmark tables were produced in thinking mode with preserve_thinking=true, i.e. long reasoning traces, so the KV table above is not a worst case you can ignore; and the card recommends preserve_thinking=False for general chat and True for multi-turn tool use, office tasks and code-agent workflows.

For the full benchmark data, see /check/nanbeige4-2-3b/m2-pro.

Troubleshooting

error loading model: unknown model architecture: 'nanbeige'

Your runtime predates 2026-07-27. Mainline llama.cpp merged the architecture in PR #25994 and the first release carrying it is b10153; brew upgrade llama.cpp is the fix. LM Studio shows the same message when its bundled backend is older than that — the model card's instruction to hand-copy a fork's binaries into the backend directory was written before mainline merged and can be ignored once your backend is current.

Keep --cache-type-k and --cache-type-v the same type

On Metal this is not a style preference. ggml_metal_device_supports_op rejects the fused-attention op outright when the K and V cache types differ — ggml-metal-device.m:1291 is a bare if (op->src[1]->type != op->src[2]->type) { return false; } — so a mismatched pair takes your attention off the GPU rather than erroring. The same function whitelists the usable cache types: f32, f16, q8_0, q4_0, q4_1, q5_0, q5_1 (plus bf16 where the device supports it). Anything outside that list, iq4_nl included, is refused the same way. This model's head_dim of 128 is on the kernel's supported head-size list, so it is only the type pairing you have to get right.

Related: a quantized V cache requires the fused-attention path. llama.cpp turns it on for you — the log line is "enabling flash_attn since it is required for quantized V cache" — but if you have explicitly passed -fa off, context creation fails with quantized V cache requires flash_attn to be enabled (llama-context.cpp:3567 and :3571). This is not FlashAttention. -fa selects ggml's own fused-attention kernel, which has a Metal implementation compiled into the binary you just installed; the CUDA flash-attn package has no Apple build, is not a dependency of anything here, and must not be installed.

--ctx-size 0 allocates 44 GiB and fails instantly

0 does not mean "use the default". common/arg.cpp sets params.fit_params_min_ctx = UINT32_MAX when you pass it, which disables the automatic context reduction and asks for the model's full 262,144 tokens — 44.000 GiB of f16 cache on a machine with 10.667 GiB addressable. Always name the context explicitly on this tier.

For the same reason, do not assume llama-server's four slots multiply your cache. The server sets n_parallel = 4 and kv_unified = true in the same branch, and under a unified cache n_ctx_seq = n_ctx — so --ctx-size 32768 buys one conversation the full 32,768 tokens out of a single shared pool, and the 2.922 GiB above is the whole cost, not a quarter of it.

ValueError: Model type nanbeige not supported. from mlx-lm

Expected on 23 of the 27 MLX builds, for the reason in the runtime-choice section: stock mlx-lm has no nanbeige architecture. Nothing about the repo you picked is wrong — including the one under mlx-community, which is in the failing group. Either use the GGUF path, or switch to one of the four repos that ship their own model_file implementation and pass --trust-remote-code.

One of those four is a trap. WaveCut/Nanbeige4.2-3B-heretic-MLX-DWQ-4bit loads, and it is an abliterated derivative rather than a quantization of the base model — "it worked when the others didn't" is the worst possible reason to settle on it. It is not a like-for-like substitute for anything else in this recipe.

Tool calls come back as plain text in message.content

A live parser bug in llama.cpp, not a model failure. The model emits <tool_call> followed by a space where the auto-derived marker expects a newline, and the call is dropped — the open fix, PR #26324, puts the incidence at roughly 25% of calls and describes the consequence as "All such tool calls currently fail and are displayed verbatim to the user instead of being executed." Until it merges, lower the temperature for tool-use runs or build llama.cpp from the PR branch.

Failed to initialize samplers: Unexpected empty grammar stack

Reported when using response_format: json_schema against llama-server on the default jinja chat path; the reporter's workaround is --no-jinja (discussion #6). Both reports in that thread are community, and the second is flagged by its own author as written by an AI agent, so treat the root-cause analysis as unconfirmed. Note the trade: --no-jinja drops the chat template, and the template is what tool calling is parsed from — so this workaround and the entry above are mutually exclusive.

Memory pressure, beachballing, or a failed allocation at startup

If you are only a couple of hundred megabytes short, reach for --ubatch-size 256 first: it takes the logits reservation from 324.5 MiB to 162.25 MiB, costs prompt-processing throughput and nothing else, and leaves both the context window and the decode speed untouched. Past that, drop --ctx-size one rung down the ladder, or halve the cache with --cache-type-k q4_0 --cache-type-v q4_0, before reaching for sudo sysctl iogpu.wired_limit_mb=<MB>. On a 16 GB machine the raise is almost never the right answer: the ceiling is two-thirds of the machine precisely so the other third can run macOS, and there is no configuration in the table above that a raise unlocks and cache quantization does not unlock more cheaply. If you do raise it, leave 8–16 GB for the OS — which on this machine means there is nothing to give — watch Activity Monitor's memory-pressure gauge, and remember the setting is temporary and resets on reboot, with 0 restoring the default.

Do not install the Python path just to try it

The card's HuggingFace quickstart needs trust_remote_code=True and a very specific dependency set; a community write-up got it working only after pinning Python 3.11, PyTorch 2.8.0, Transformers 4.42.4 and several more, having first produced gibberish on a mismatched configuration (discussion #15). That path also pulls CUDA-only attention packages with no Apple equivalent. Neither route above has any of those dependencies.

Nothing else widely reported. Problems, or a throughput number for this chip, go to the submission form.