Nanbeige4.2-3B on Apple M3 Max: the full 262,144-token context in 36 GiB

What You'll Build

A local OpenAI-compatible server running Nanbeige4.2-3B at its full advertised 262,144-token context on an Apple M3 Max. The install is mainline llama.cpp with the Metal backend and a community GGUF; what makes this machine the right one for this model is that 48 GB of unified memory is the first Apple configuration where the whole window is reachable — the model's 22 decoder blocks execute twice per forward pass, so the cache costs double what its config file suggests, and that doubling is what puts the top of the context ladder out of reach on smaller Macs.

Hardware data: Apple M3 Max (48 GB unified memory, 36.000 GiB GPU-addressable) · Q4_K_M weights 2.398 GiB + q8_0 KV at 262,144 tokens 23.375 GiB + reserved logits 0.317 GiB = 26.090 GiB derived · no Apple benchmark submitted yet · See benchmark data

⚠️ The KV cache is 2× what the layer count suggests. config.json sets num_loops: 2, so the 22 physical blocks run twice and each pass keeps its own KV entries — 44 cache layers, not 22. llama.cpp is explicit about it: src/models/nanbeige.cpp widens the logical layer count — "Expand logical layer count before load_tensors() allocates layers / KV." — and comments the weight-sharing loop "Share physical weights across loops; each slot still has its own KV index." Budget 176.0 KiB per token at f16, not 88.0 KiB.

Requirements

Component	Minimum	This recipe
GPU	Apple silicon, Metal	Apple M3 Max, 48 GB unified memory — not measured on this chip; the budget below is derived from file bytes and the runtime's own allocation rules (/contribute)
Unified memory	36 GB — the smallest Apple capacity whose default share clears the lead configuration	48 GB, of which 36.000 GiB is GPU-addressable at the macOS default
Storage	2.575 GB / 2.398 GiB (Q4_K_M)	2,574,807,904 bytes on disk; 4.130 GiB if you take Q8_0 instead
Software	macOS Sonoma 14+, llama.cpp ≥ b10153	Homebrew llama.cpp (currently 10330) or a source build

The pool is 36.000 GiB, not 48

Apple silicon has no dedicated video memory, and the number on the box is not what the GPU gets. Metal publishes a ceiling in recommendedMaxWorkingSetSize, and MLX, llama.cpp-Metal and Ollama all treat it as a hard cap. On a 48 GB Mac that ceiling is 38,654,705,664 bytes = 36.000 GiB — exactly three-quarters of the machine's 48 GiB.

That is a logged figure, not a rule of thumb. A report against LM Studio's tracker from an owner who describes his machine as a "MacBook Pro with 48 GiB memory" pastes a llama.cpp startup log containing "recommendedMaxWorkingSetSize = 38654.71 MB" (lmstudio-bug-tracker #271); the identical value appears in a separate report on ollama #12012.

Read the units before you convert. Current ggml prints this field as max_working_set_size / 1e6 — decimal MB — at ggml-metal-device.m:946, and three-quarters of 48 GiB is 38,654,705,664 bytes, which that line renders as 38654.71. Reading the same figure as mebibytes instead would give 37.749 GiB — 78.6% of the machine, a fraction of nothing in particular. The log settles its own ambiguity two lines earlier, in a field printed in unambiguous MiB: "using device Metal (Apple M4 Pro) - 36863 MiB free", i.e. a pool of 36,864 MiB = 36.000 GiB. Note the label is not a reliable tell on its own — the 2023-era build at tag b1180 printed the same MB suffix while dividing by 1024², so an older log's MB means MiB and a current one's means MB.

One honest caveat about the chip. Every 48 GB log we have is from an Apple M4 Pro, not an M3 or M4 Max. The share appears to be an OS policy keyed on installed memory rather than on GPU core count — every other capacity we have logs for lands on an exact fraction of its own GiB capacity, regardless of chip — but that inference is ours, not a statement from Apple. It is also the direction that would overpromise if wrong, since 36.000 GiB is more generous than the two-thirds share smaller Macs get. The recommendation on this page has 9.910 GiB of headroom against it, so it survives a materially smaller ceiling; the top of the ladder does not.

Below 36 GB the default share is exactly two-thirds instead — a 32 GB Mac addresses 21.333 GiB — and the break between the two regimes sits somewhere above 32 GB and at or below 36 GB, where a 36 GB machine logs 28991.03 MB = 27.000 GiB, three-quarters again.

Runtime choice: llama.cpp with Metal

Why llama.cpp. Native support for this architecture was merged into mainline on 2026-07-27; it is registered as nanbeige, Metal is the default backend on macOS, and you can pin a build number. It also gives you the KV-precision flags that the lead configuration on this page depends on entirely.

The MLX shelf is large and mostly unloadable. A HuggingFace search returns 27 MLX-tagged repositories for this model (re-enumerated, not sampled, on 2026-08-10 — 27 of 78 hits, unchanged in count from the previous week). Twenty-three of them cannot be loaded by stock mlx-lm at all: it resolves an architecture by importing mlx_lm.models.<model_type>, and there is no nanbeige.py among the 121 files in mlx_lm/models/. HuggingFace's custom_code tag does not rescue them, because mlx-lm never reads auto_map. The remaining four declare a model_file key in their own config.json, which mlx-lm imports under trust_remote_code, so they load on stock mlx-lm with --trust-remote-code: three jishnuvenugopal/Nanbeige4.2-3B-mlx-{4,6,8}bit repos shipping nanbeige.py, and WaveCut/Nanbeige4.2-3B-heretic-MLX-DWQ-4bit shipping an independently written nanbeige_mlx.py. That is a provenance call rather than a technical verdict: leading llama.cpp means running reviewed upstream C++ you can pin, instead of one individual's Python model definition that self-pins to mlx-lm internals.

Whichever you pick, the memory arithmetic is the same. That community MLX port's make_cache returns num_loops * num_hidden_layers cache slots — 44 — matching llama.cpp, the HuggingFace reference and the Swift port. The 44 slots are an architecture cost, not a llama.cpp quirk; only the flag names differ.

Two things on the model card to skip. Its llama.cpp section clones the vendor's own fork and configures the build with a CUDA backend flag — mainline superseded that fork and there is no CUDA on a Mac. And the org publishes -FP8 and -GPTQ-Int8 variants; neither has an Apple execution path, any more than bitsandbytes, AWQ, Marlin or ExLlamaV2 do, because no Metal kernel exists for any of them. The GGUF K-quants below are the Apple equivalent.

Installation

1. Install llama.cpp with Metal

brew install llama.cpp
llama-cli --version

The Homebrew formula is currently at build 10330; anything at or above b10153 contains the Nanbeige architecture. Metal needs no flags on macOS — it is the default backend. To build from source instead:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

2. Fetch the GGUF

llama.cpp downloads and caches HuggingFace repos itself, so the run command below needs no separate download step. To hold the file yourself:

pip install -U huggingface_hub

hf download owao/Nanbeige4.2-3B-GGUF Nanbeige4.2-3B-Q4_K_M.gguf \
  --local-dir ~/models/nanbeige4.2-3b

hf is the current name of the HuggingFace CLI — it was renamed from huggingface-cli in huggingface_hub v0.34.0, so on an older install the same command is huggingface-cli download.

There is no first-party GGUF and no first-party MLX build. The Nanbeige org publishes ten repositories and every one is safetensors-only, so every quant you can run is a community one. owao/Nanbeige4.2-3B-GGUF is the set to use: 13 rungs from Q2_K to Q8_0, and the uploader re-uploaded all of them on 2026-07-28 to pick up the vendor's chat-template fix. That matters because a quant is frozen at its upload timestamp, and this set was first published on release day, before the fix landed.

Running

llama-server \
  -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M \
  --host 127.0.0.1 --port 8080 \
  --ctx-size 262144 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 \
  --temp 0.6 --top-p 0.95 --top-k 20

The server comes up on http://127.0.0.1:8080 with an OpenAI-compatible /v1/chat/completions endpoint and a browser UI at the same address. The KV cache is allocated up front, so an over-ambitious context fails within the first few seconds rather than mid-conversation.

That is 2.398 GiB of weights plus 23.375 GiB of quantized cache plus 0.317 GiB of reserved logits = 26.090 GiB, against the 36.000 GiB this Mac addresses — a 9.910 GiB surplus. The model card states that "The model supports a context length of up to 262,144 tokens (256K).", and --ctx-size is llama.cpp's total context, prompt plus generation, so this configuration is the whole advertised window in one pool.

Spending 72% of the addressable pool would be an uncomfortable lead on a 16 GB machine, where the remaining third of the memory is where macOS lives. It is not uncomfortable here: 26.090 GiB of a 48 GB machine leaves roughly 20 GB outside the Metal ceiling for the operating system and everything else you have open.

--cache-type-k / --cache-type-v are the long forms of -ctk / -ctv. Keep them equal — see Troubleshooting. The sampler values are the model card's own defaults for reasoning and chat; for agentic and tool-use work the card recommends --temp 1.0, but read the tool-call entry under Troubleshooting first.

The exact-cache alternative

If you would rather not quantize the cache at all, halve the window instead:

llama-server -hf owao/Nanbeige4.2-3B-GGUF:Q4_K_M --host 127.0.0.1 --port 8080 \
  --ctx-size 131072 -ngl 99 --temp 0.6 --top-p 0.95 --top-k 20

That is 24.715 GiB with an f16 cache — 1.375 GiB less than the lead configuration, for half the context. Which trade is better depends on your work, and nobody has published what q8_0 costs this model in output quality, so treat the choice as one to evaluate on your own prompts. What is not available at any setting is the full window with an f16 cache: that is 46.715 GiB against a 36.000 GiB pool, over by 10.715 GiB, and it is the one configuration a 64 GB Mac has that this one does not.

The memory budget — the loop tax

Where the parameters are. config.json describes 22 layers of hidden_size 3072 with 48 query heads over 8 KV heads at head_dim 128, an intermediate size of 10752, an untied 166,144-token vocabulary, and num_loops: 2. That sums to 4,169,800,704 parameters, of which 3,149,011,968 are non-embedding — closing exactly against the count HuggingFace reports for the safetensors and settling the naming: the "3B" counts non-embedding parameters. The census closing with one copy of the stack is itself the proof that the second loop is free in weights, and the Q4_K_M file carries 201 tensors — 22 blocks × 9, plus embedding, output norm and LM head — rather than 44 blocks' worth.

Where the memory goes. The loop is not free in cache terms. With 8 KV heads at 128 dimensions, one layer stores 2,048 K+V elements per token. load_arch_hparams() sets hparams.n_layer_all = n_layer_phys * n_loops = 44, and the allocator in src/llama-kv-cache.cpp reads exactly that field — const uint32_t n_layer = hparams.n_layer_all; — creating one K and one V tensor per slot. So per token:

44 × 2048 × 2 bytes = 180,224 bytes = 176.0 KiB at f16, against 88.0 KiB for the same stack run once.

cache type	bytes/token	@32,768	@65,536	@131,072	@262,144
f16 (default)	180,224	5.500 GiB	11.000 GiB	22.000 GiB	44.000 GiB
q8_0	95,744	2.922 GiB	5.844 GiB	11.688 GiB	23.375 GiB
q4_0	50,688	1.547 GiB	3.094 GiB	6.188 GiB	12.375 GiB

The third term, and it is not the cache. The Metal compute buffer holds one reserved tensor that is large on this model specifically: the logits. llama.cpp sizes its worst-case graph once at startup, and src/llama-context.cpp asks for n_outputs_pp = std::min(n_tokens, cparams.n_outputs_max) rows of it, where n_tokens is itself std::min(cparams.n_ctx, cparams.n_ubatch) and n_outputs_max defaults to n_batch. At stock settings that is 512 rows whatever --ctx-size you pass, each a full f32 distribution over the vocabulary:

512 rows × 166,144 vocab × 4 B = 340,262,912 B = 324.5 MiB = 0.317 GiB

Two properties earn it a column. --ctx-size does not move it — --ubatch-size does, linearly, which makes -ub the first lever when you are a couple of hundred mebibytes short of a rung. And it is large only because this vocabulary is: a 32,000-token vocabulary would reserve 62.5 MiB for the same graph.

Adding weights and logits, against the 36.000 GiB this Mac addresses:

configuration	weights	KV	logits	total	verdict
Q4_K_M + q4_0 @ 256K	2.398	12.375	0.317	15.090 GiB	Fits with room to spare; coarsest cache
Q4_K_M + q8_0 @ 128K	2.398	11.688	0.317	14.402 GiB	Fits; half the window for half the cache
Q4_K_M + f16 @ 128K	2.398	22.000	0.317	24.715 GiB	Fits — the exact-cache alternative above
Q4_K_M + q8_0 @ 256K	2.398	23.375	0.317	26.090 GiB	Fits — this recipe
Q5_K_M + q8_0 @ 256K	2.782	23.375	0.317	26.474 GiB	Fits — spend the headroom on weights
Q8_0 + q8_0 @ 256K	4.130	23.375	0.317	27.822 GiB	Fits, but see the bandwidth note below
Q4_K_M + f16 @ 256K	2.398	44.000	0.317	46.715 GiB	Over by 10.715 GiB

This is also what settles the 36 GB floor rather than assuming it. The share is an exact fraction of capacity rather than a range, so the comparison is decidable in both directions: a 36 GB Mac addresses exactly 27.000 GiB and clears the lead configuration by 0.910 GiB — before roughly 25 MiB of attention scratch, and comfortably outside rounding — while a 32 GB Mac falls under the two-thirds regime at 21.333 GiB and is 4.757 GiB short. That is not a margin call in either direction, which is why this recipe's floor is 36 rather than a hedge.

Two things Metal does not charge you for. Quantizing the cache costs no dequantisation copy here. The CUDA flash-attention path stages an extra f16 buffer whenever the K cache is not already f16, sized from ggml_nelements(K) in fattn-common.cuh, so on that backend a quantized cache hands part of its saving back in proportion to the context. The Metal encoder has no analogue — ggml_backend_metal_buffer_type_get_alloc_size appends exactly three regions to a flash-attention output, and none of them is a copy of K or V. Sized from ggml-metal-ops.cpp for this model's 48 heads and 128-wide value head, the largest is the per-workgroup result buffer at 4 × 32 × 48 × 32 × 130 bytes = 24.375 MiB with no context term at all; the only context-linear region is a block-skip bitmap of one byte per 64 cache positions per 8 query rows, which is 256 KiB even at 262,144 tokens. So the q8_0 row above is the whole cache cost on this machine, which is what lets the lead configuration be quoted as a flat 26.090 GiB.

And the attention mask is not inside the Metal ceiling at all. llm_graph_input_attn_kv::set_input fills it by writing straight to tensor->data, so it asserts ggml_backend_buffer_is_host(self_kq_mask->buffer) (llama-graph.cpp:450) — and both Metal buffer types return a hard-coded false from their is_host callback (ggml-metal.cpp:275), so the scheduler places the mask on the CPU backend. On Apple that does not mean "somewhere else in the machine" — one pool of DRAM backs both, and those bytes still come out of your 48 GB. It means they do not count against the 36.000 GiB figure every budget on this page is measured against.

The loop is a design decision, not an implementation artifact. The team tested KV sharing across loop passes and rejected it — "we did try sharing the KV cache across loop passes, but it noticeably hurt performance, so we kept the full cache in Nanbeige4.2." (discussion #18, Nanbeige team member leran1995). Sharing would have halved the cache; they kept the model quality. No runtime flag undoes that — only cache quantization moves this number.

What the M3 Max brings

This chip is the 16-core-CPU / 40-core-GPU bin, and Apple's own specification page is the reason there is no ambiguity: the M3 Max ships in two bins — "M3 Max with 16-core CPU and 40-core GPU (400GB/s memory bandwidth)" against a 300 GB/s 14-core part — and 48 GB is sold only on the faster one, listed as "48GB unified memory (M3 Max with 16-core CPU)" (Apple, MacBook Pro 14-inch M3 specifications). The 14-core bin is offered at 36 GB and 96 GB instead, so on this model of Mac the capacity identifies the bin. Quote a bandwidth figure only with the bin it belongs to.

That 400 GB/s is what the loop actually spends. Token generation streams weights, and the loop makes the decoder stack stream twice. Reading the Q4_K_M tensor table out of the GGUF header: the 22 blocks are 1,865,048,064 bytes and the Q6_K LM head is 418,682,880, so a generated token moves 2 × 1,865,048,064 + 418,682,880 = 4,148,779,008 bytes ≈ 3.864 GiB — about 1.82× what a same-sized model without the loop would move. On a bandwidth-bound machine that ratio is close to a throughput ratio, and it is the reason the Q8_0 row in the table above, while it fits, is the wrong trade: at 4.130 GiB of weights the per-token traffic rises with it, and bytes moved is the scarce resource here, not gigabytes on disk.

It is also why no speed number on this page is borrowed from another chip. The M3 Max shares its 400 GB/s class with the M2 Max, but not with the M4 Max, whose 48 GB bin runs 546 GB/s (Apple's own spec page) — roughly 1.37× as much — so an M4 Max figure would not transfer here even if one existed.

Results

• Speed: omitted. No throughput measurement exists for this model on the llama.cpp-Metal path on any Apple chip, and none on an M3 Max on any runtime — not the model card, not the technical report (neither publishes a tokens-per-second figure at all), not the 28 discussions on the canonical repo, not the three on the GGUF repo, and not a targeted web search naming this chip. One Apple measurement does exist, and it is worth naming precisely so nobody mistakes its scope: the author of the community MLX port published a 30-case agentic eval on a machine he records as "Apple M1 Pro, 10-core, 16 GB unified memory", greedy decoding, reporting 35.1 tok/s aggregate decode at MLX 4-bit with the raw per-case results committed alongside (nanbeige-mlx-eval). It is a single community source, on the other runtime, on short tool-use prompts rather than a long window, and on an entry-tier chip several generations and a large bandwidth gap away from this one — so it is a floor this machine should beat by a wide and unquantified margin, not an estimate of it. This page therefore quotes no figure for the M3 Max. If you run it, please contribute the numbers so /check/nanbeige4-2-3b/m3-max stops being empty.
• Unified memory usage: 26.090 GiB derived for the configuration above — 2,574,807,904 bytes of Q4_K_M weights, 25,098,715,136 bytes of q8_0 KV cache at 262,144 tokens and 340,262,912 bytes of reserved logits — against 36.000 GiB addressable, a 9.910 GiB surplus. That is a derivation from measured file bytes, the runtime's 44-layer allocation rule and its worst-case graph reservation, not a measured peak; one term sits on top of it, the flash-attention scratch sized above at roughly 25 MiB and near-flat in context. See /check/nanbeige4-2-3b/m3-max.
• Quality notes: the card's published benchmark tables were produced in thinking mode with preserve_thinking=true — long reasoning traces — so the cache table above is not a worst case you can ignore, and it is a good part of the argument for taking the full window here. The card recommends preserve_thinking=False for general chat and True for multi-turn tool use, office tasks and code-agent workflows. Nobody has published a quality delta between f16, q8_0 and q4_0 caches on this model, so no such claim is made on this page.

For the full benchmark data, see /check/nanbeige4-2-3b/m3-max.

Troubleshooting

error loading model: unknown model architecture: 'nanbeige'

Your runtime predates 2026-07-27. Mainline llama.cpp merged the architecture in PR #25994 and the first release carrying it is b10153 — src/models/nanbeige.cpp returns 404 at b10152 and 200 at b10153; brew upgrade llama.cpp is the fix. LM Studio shows the same message when its bundled backend is older than that — the model card's instruction to hand-copy a fork's binaries into the backend directory was written before mainline merged and can be ignored once your backend is current. Stock Ollama has no library entry for this model at all: ollama.com/library/nanbeige, /nanbeige4.2, /nanbeige4-2 and /nanbeige4.2-3b all return 404, and the nanbeige namespace the card's ollama run command assumes does not exist either.

Keep --cache-type-k and --cache-type-v the same type

On Metal this is not a style preference. ggml_metal_device_supports_op rejects the fused-attention op outright when the K and V cache types differ — ggml-metal-device.m:1291 is a bare if (op->src[1]->type != op->src[2]->type) { return false; } — so a mismatched pair takes your attention off the GPU rather than erroring. The same function whitelists the usable cache types: f32, f16, q8_0, q4_0, q4_1, q5_0, q5_1, plus bf16 where the device supports it. Anything outside that list, iq4_nl included, is refused the same way. This model's head_dim of 128 is on the kernel's supported head-size list, so the type pairing is the only thing you have to get right.

Related: a quantized V cache requires the fused-attention path, and llama.cpp turns it on for you at the default -fa auto. This is not FlashAttention. -fa selects ggml's own fused-attention kernel, which has a Metal implementation compiled into the binary you just installed; the CUDA flash-attn package has no Apple build, is not a dependency of anything here, and must not be installed.

--ctx-size 0 allocates 44 GiB and fails instantly

0 does not mean "use the default" — common/arg.cpp sets params.fit_params_min_ctx = UINT32_MAX when you pass it, which disables automatic context reduction and asks for the model's full 262,144 tokens at the default f16 cache: 44.000 GiB, well past this pool. Name the context and the cache types explicitly, as the run command does.

For the same reason, do not assume llama-server's four slots divide your cache. The server sets n_parallel = 4 and kv_unified = true in the same branch, and under a unified cache n_ctx_seq = n_ctx — so --ctx-size 262144 buys one conversation the whole window out of a single shared pool, and the 23.375 GiB above is the entire cost, not a quarter of it.

ValueError: Model type nanbeige not supported. from mlx-lm

Expected on 23 of the 27 MLX builds, for the reason in the runtime-choice section: stock mlx-lm has no nanbeige architecture. Nothing about the repo you picked is wrong — including the one under mlx-community, which is in the failing group. Either use the GGUF path, or switch to one of the four repos that ship their own model_file implementation and pass --trust-remote-code.

One of those four is a trap. WaveCut/Nanbeige4.2-3B-heretic-MLX-DWQ-4bit loads, and it is an abliterated derivative rather than a quantization of the base model — "it worked when the others didn't" is the worst possible reason to settle on it. It is not a like-for-like substitute for anything else in this recipe.

Tool calls come back as plain text in message.content

This was fixed upstream on 2026-08-02, and the fix is in release tag b10227. The defect was llama.cpp's, not the model's: the model sometimes emits <tool_call> followed by a space where the auto-derived marker expected a newline, the parser failed to match, and the whole reply came back as content. The proposed patch, PR #26324, put the loss at roughly a quarter of calls and never merged — a maintainer objected that trimming trailing whitespace from markers had already been reverted once for degrading other models' output. A contributor then pointed at the actual mechanism — "The grammar sampler should constrain the output to force <tool_call>\n. I suspect the grammar trigger contains the trailing \n, preventing the grammar sampler from activating." — and at PR #26252, a specialized parser that merged on 2026-08-02 and shipped as b10227. #26324's author tested a later build, reported the calls working every time, and closed his own pull request on 2026-08-10.

That makes this a version check rather than a workaround. The Homebrew formula named under Installation is build 10330, past b10227, so a current brew install llama.cpp already has the fix and the card's --temp 1.0 for agentic work needs no hedging. If calls still come back as text, your binary is older than b10227 — brew upgrade llama.cpp.

Is a quantized cache slow at this context?

A community report opened on 2026-08-10 claims that implementations with a quantized KV cache read it as f16, multiplying DRAM traffic and making long-context inference "incredibly slow". Treat it as unverified: it has no maintainer response, it names no backend, and the same reporter's other posts in that repo's discussions describe an 8 GB discrete NVIDIA card (discussion #18) — which is the backend whose flash-attention path genuinely does stage an f16 copy, as the CUDA source cited above shows and the Metal source does not. It is in any case a throughput claim, not a memory one: nothing in the budget on this page changes either way. If you can measure it on this chip, that measurement would settle it — send it in.

Memory pressure or a failed allocation at startup

Reach for --ubatch-size 256 first if you are only a couple of hundred mebibytes short: it halves the logits reservation to 162.25 MiB, costs prompt-processing throughput and nothing else, and leaves both the window and the decode speed untouched. Past that, drop --ctx-size one rung, or take --cache-type-k q4_0 --cache-type-v q4_0, before reaching for sudo sysctl iogpu.wired_limit_mb=<MB>. On this model the raise is rarely the right answer — cache quantization buys back more, more safely — but if you do raise it, leave 8–16 GB for macOS, watch Activity Monitor's memory-pressure gauge, and remember the setting is temporary and resets on reboot, with 0 restoring the default.

Do not install the Python path just to try it

The card's HuggingFace quickstart needs trust_remote_code=True and a very specific dependency set; a community write-up got it working only after pinning Python 3.11, PyTorch 2.8.0, Transformers 4.42.4 and several more, having first produced gibberish on a mismatched configuration (discussion #15). That path also pulls CUDA-only attention packages with no Apple equivalent. The GGUF route above has none of those dependencies.

Nothing else widely reported. Problems, or a throughput number for this chip, go to the submission form.