Muse Glimmer 30B on Apple M3 Max: ExecuTorch Metal agent server with vision and DFlash

What You'll Build

An OpenAI-compatible agent server running Meta's Muse Glimmer 30B entirely on a MacBook Pro M3 Max, with image input and speculative decoding both switched on, and enough unified memory left over that you never touch a wired-limit override. The interesting part is the runtime: Meta measured this model on Apple Silicon with ExecuTorch, not with llama.cpp and not with MLX-LM, and ships pre-exported Metal artifacts for it.

Hardware data: Apple M3 Max (48 GB unified memory, 400 GB/s) · working set 21.316 GiB against a 36.000 GiB addressable pool · See benchmark data

ℹ️ Three Apple paths exist, and they are not interchangeable evidence. ExecuTorch, llama.cpp-Metal and MLX all run this model today, and this recipe verified each one separately. Meta's published Apple tok/s figures were produced only on ExecuTorch — the model card states "M4/M5 measurements were done using ExecuTorch, and RTX using llama.cpp." A number measured on ExecuTorch says nothing about MLX or llama.cpp throughput, and on this model the gap is not academic: the only independent Apple llama.cpp measurement this round found puts the DFlash drafter at parity, where Meta's ExecuTorch table puts it at 1.5×. Both figures are below, each attached to the runtime that produced it.

Requirements

Which M3 Max this is, and why the memory size settles it. Apple sells the M3 Max in two bins with different memory buses, and the RAM option identifies the bin. Apple's MacBook Pro specifications list M3 Max with 14-core CPU and 30-core GPU (300GB/s memory bandwidth), or M3 Max with 16-core CPU and 40-core GPU (400GB/s memory bandwidth) — and, in the same document, 48GB unified memory (M3 Max with 16-core CPU). The 14-core part is sold at 36 GB and 96 GB only. A 48 GB M3 Max is therefore unambiguously the 16-core-CPU / 40-core-GPU part at 400 GB/s. This matters more than anything else on this page, because token generation is memory-bandwidth-bound and every published Apple figure for this model was measured on a different bin.

The memory budget (derived, not measured)

Apple has no dedicated VRAM. Our catalogue stores full unified memory in the GPU row, but Metal's recommendedMaxWorkingSetSize caps what the GPU can actually address: exactly 3/4 on machines of 36 GB and above, so a 48 GB Mac addresses 36.000 GiB. Everything below is framed against that number, not against 48.

The KV cache is derived from the model's own config.json, not from prose. That file declares num_key_value_heads: 2, head_dim: 128, sliding_window: 2048, and a layer_types array containing 39 sliding_attention and 13 full_attention entries — the [Local, Local, Local, Global] repeating pattern from the model card, counted out of the artifact.

Per-token KV is 2 (K+V) × 2 KV heads × 128 head dim × 2 bytes = 1024 B per layer — 1 KiB, which is why a 131K context costs under 2 GiB here. The first-party GGUF card makes the same point in prose: "KV cache stays cheap".

Consequence: no sudo sysctl iogpu.wired_limit_mb raise is needed on this machine, and you should not add one. The full multimodal-plus-drafter stack sits 14.684 GiB inside the default pool.

Why 36 GB is the floor. A 32 GB Mac addresses exactly 2/3 of 32 GiB = 21.333 GiB, against this build's 21.316 GiB working set — a 0.017 GiB margin with macOS still to house. A 36 GB Mac addresses 27.000 GiB and clears it comfortably, and 36 GB is a real shipping configuration rather than a rounded-off number: the same Apple specification page lists 36GB unified memory as a sold SKU of the 14-core-CPU M3 Max, alongside 48 GB, 64 GB and 128 GB on the 16-core part. That is where min_vram_gb: 36 comes from — it is a filter floor naming a machine you can actually buy, not this recipe's measured peak.

Installation

1. Download one ExecuTorch variant

The repo publishes 16 pre-exported variants and is 372 GB in total, so --include is mandatory. The metal builds are self-contained: unlike the CUDA builds they carry their weights inside the .pte, with no separate blob to fetch.

pip install huggingface_hub

REPO=meta-models/Muse-Glimmer-30B-ExecuTorch-PTE
VARIANT=muse-glimmer-k-quant-17G-128K-text-image-dflash-metal
LOCAL_DIR=./muse-glimmer-pte

hf download "$REPO" \
  --include "$VARIANT/*" \
  --include "tokenizer.json" \
  --include "tokenizer_config.json" \
  --include "chat_template.jinja" \
  --local-dir "$LOCAL_DIR"

⚠️ Re-pull chat_template.jinja if you downloaded it before 2026-08-15. The ExecuTorch repo keeps its own copy of the template, and until commit fc6fa93c that copy was a stale one — it named a different model family in its raise_exception string and lacked the system-prompt normalisation the base repo's template had gained. Verify in one line: shasum -a 256 "$LOCAL_DIR/chat_template.jinja" must print cfc67e5f349f37690dfd31ed1f18bc4442a9dd32fe39a648f993cb4eb3cae678, and the file must be 9,992 bytes. That is byte-for-byte the base repo's template — confirmed identical on 2026-08-21 — against the old file's 7,167 bytes. What the current logic does: it normalises four casings of "Reasoning effort" to "Reasoning strength", then emits the kwarg-driven directive only {%- if 'reasoning strength' not in (sys_text | lower) -%}. Under the old template a system prompt that already carried a reasoning line got a second, conflicting one appended — the same defect the GGUF card describes for the pre-2026-08-12 GGUFs, which survived three days longer here.

The variant name is a fixed scheme — muse-glimmer-<quant>-128K-<modality>-<decoding>-<backend>. k-quant-17G is the ~4-bit build Meta targets at a 24 GB envelope, text-image adds the perception encoder, dflash bundles the speculative-decoding drafter, and metal is the Apple Silicon backend. The repo publishes no CPU build: the card states "There is no CPU variant."

Two byte-level cross-checks on that artifact, both repeatable against the HF tree API and both done on the .pte files alone, since pos_embed.bin is a separate 6,291,456 B object. Subtracting the text-solo build from the text-image-solo build gives a perception encoder of 1,413,208,832 B; doing the same subtraction inside the dflash pair gives 1,413,208,704 B — the same figure to within 128 bytes of serialization padding, and within 1% of the 1,400,328,928 B mmproj-Muse-Glimmer-30B-Q4_K_M.gguf that the GGUF repo publishes separately. The drafter subtracts out at 1,695,925,376 B and 1,695,925,248 B by the same two routes.

2. Build the ExecuTorch runner

A download is not enough — the pre-exported artifact skips the export step, not the native runtime. Follow the ExecuTorch build-from-source guide to get a checkout configured, then build the model preset from the repo root. The Apple preset is muse-glimmer-mlx, and it is gated to Darwin in CMakePresets.json — there is no chip-generation condition anywhere in it:

(cd examples/models/muse-glimmer && cmake --workflow --preset muse-glimmer-mlx)
pip install -r examples/llm_server/python/requirements.txt

⚠️ Command Line Tools are not enough — you need Xcode itself. The delegate this preset builds is ExecuTorch's MLX backend, and its README says the Metal compiler is the thing "which ships with Xcode (not the standalone Command Line Tools)". Check with xcrun -sdk macosx --find metal; if it errors while Xcode is installed, point the developer directory at it with sudo xcode-select -s /Applications/Xcode.app/Contents/Developer. The same README opens with "The MLX delegate is experimental and under active development." — worth knowing before you make this your only path.

Binaries land in cmake-out/examples/models/muse-glimmer/: solo_runner, dflash_runner, and muse_glimmer_worker. The server needs the last one.

This chip is old enough to be a fair question, and the answer is yes. The pre-exported metal artifacts are not gated to a newer silicon generation: the MLX delegate's stated hardware requirement is Apple Silicon M1 or later, the build preset tests only for Darwin, and the sole MTLGPUFamily feature-gate anywhere in the ExecuTorch tree lives in the unrelated MPS backend, not in MLX. What no source establishes is speed on this generation — see Results.

This machine has no CUDA and none of the NVIDIA escape hatches apply: skip the muse-glimmer-cuda preset, skip every sm80+ptx variant in the repo, and ignore the FP8 / NVFP4 tensor-core paths the vendor's RTX numbers rely on — Apple Silicon has no such hardware. Skip pip install flash-attn as well; on Metal the attention path is MLX-native. Under the MLX heading of PyTorch's announcement, the first bullet reads "RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations are lowered to MLX-native or custom Metal implementations."

Running

Start the OpenAI-compatible server from the ExecuTorch repository root. The Metal build takes no --data-path; text-image needs --pos-embed-path, and dflash needs no flag at all because the exported method contract is auto-detected.

VARIANT=muse-glimmer-k-quant-17G-128K-text-image-dflash-metal
LOCAL_DIR=./muse-glimmer-pte

python -m executorch.examples.models.muse_glimmer.serving.serve \
  --model-path "$LOCAL_DIR/$VARIANT/$VARIANT.pte" \
  --pos-embed-path "$LOCAL_DIR/$VARIANT/pos_embed.bin" \
  --tokenizer-path "$LOCAL_DIR/tokenizer.json" \
  --hf-tokenizer "$LOCAL_DIR" \
  --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
  --model-id muse-glimmer-30B \
  --tool-parser atem \
  --host 127.0.0.1 --port 8000

Smoke-test it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"muse-glimmer-30B","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":32,"temperature":0}'

--tool-parser atem is what converts Muse Glimmer's native tool output into OpenAI tool_calls; the default is none, so an agent harness needs it passed explicitly. Reasoning depth is a system-prompt setting (Reasoning strength: low|medium|high|xhigh), not the OpenAI reasoning_effort parameter, which this server rejects with a structured 400.

One Apple-specific bonus worth knowing. The DFlash block length is exported dynamically and the accepted range differs by backend: the Metal build accepts --dflash-block-length in [2, 16] while the CUDA build accepts only [2, 4]. The Metal path is the one that can run the drafter at its full trained block size of 16.

⚠️ Two different DFlash mechanisms share a name — do not mix their flags. On this ExecuTorch path the drafter is baked into the exported .pte method contract: the text-image-dflash variant is the drafter, --artifact-mode defaults to auto and detects it, and --dflash-block-length is an ExecuTorch server flag. On the llama.cpp path below the drafter is a separate GGUF file bound with -md / -ngld and driven by llama.cpp's own draft-dflash speculative type. Neither set of flags exists in the other runtime.

Alternative: llama.cpp-Metal with the first-party GGUF

Mainline llama.cpp registers this architecture — LLM_ARCH_MUSE_GLIMMER is at line 74 of src/llama-arch.cpp, added by PR #26841 (merged 2026-08-10, commit 62bf73d) and first released in build b10353. Metal is on by default on macOS, so omit any GPU backend flag:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DBUILD_SHARED_LIBS=OFF
cmake --build build --config Release -j --target llama-cli llama-mtmd-cli llama-server

hf download meta-models/Muse-Glimmer-30B-GGUF \
    --local-dir Muse-Glimmer-30B-GGUF \
    --include "Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf" \
    --include "mmproj-Muse-Glimmer-30B-Q4_K_M.gguf" \
    --include "dflash-Muse-Glimmer-30B-Q4_K_M.gguf"

./build/bin/llama-server \
    -m       Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
    --mmproj Muse-Glimmer-30B-GGUF/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
    -md      Muse-Glimmer-30B-GGUF/dflash-Muse-Glimmer-30B-Q4_K_M.gguf -ngld 99 \
    --spec-type draft-dflash \
    -a muse-glimmer-30B \
    -ngl 99 -c 131072 -np 1 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64

Image input on this path goes through llama-mtmd-cli rather than llama-cli, and vision is genuinely implemented rather than merely implied by the presence of an mmproj file: llama.cpp's multimodal layer carries a dedicated PROJECTOR_TYPE_MUSE_GLIMMER, a graph builder at tools/mtmd/models/muse-glimmer.cpp, and its own image preprocessor.

Speculative decoding is a first-class type here too, not an improvisation: COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH has a dedicated implementation in common/speculative.cpp that reads the trained block size from the dflash.block_size GGUF metadata key, and LLM_ARCH_DFLASH is a registered architecture.

⚠️ --spec-type without -md silently does nothing — and the reverse used to be true, but stopped being true on 2026-08-13. Both directions are worth the paragraph, because most of what is written about this flag pair describes the older behaviour.

-md alone is enough on a current build. PR #26814 ("common : auto-detect spec type from draft GGUF metadata", merged 2026-08-13, first release b10413) added a second inference path in common/arg.cpp: when no --spec-type was passed and a local draft path is set, llama.cpp opens the drafter, reads its general.architecture, and picks the type itself. For this drafter that resolves to draft-dflash specifically, and the check is finer than the architecture string alone — common_speculative_types_from_gguf returns draft-dspark instead when the file carries a markov_w1.weight tensor. Meta's dflash-Muse-Glimmer-30B-Q4_K_M.gguf declares general.architecture = dflash and carries 58 tensors, none of them a Markov head, so the inference lands on draft-dflash. llama-server reaches this code: it calls the same handler itself in single-model mode.

Pass --spec-type draft-dflash anyway — that is why it is in the command above. It is version-independent, so the command works for a reader on an older binary as well as on master, and the auto-detect carries a documented limit in its own source comment: it "reads only the first split - sharded drafts need an explicit --spec-type".

What was true before b10413, and is still true for anyone on such a build: common_params_speculative::types defaults to { COMMON_SPECULATIVE_TYPE_NONE }, and -md never writes to it. NONE is enum value 0 and DRAFT_DFLASH is 4, so the default produces the bitmask 1, which matches nothing — the drafter file loads into memory and is never used. This is exactly the behaviour the community write-up cited in Results measured against, three days before the change landed, and it is why the claim is still repeated.

--spec-type without a draft model does nothing, then and now. common_speculative_init adds the DFlash config through add_config_if_enabled(COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH, params.draft.ctx_dft != nullptr), so with no draft path there is no context to bind and the implementation is skipped before it can assert. You get no error, no warning and no speculation — the server starts and serves normally, just without the drafter. It is attested in the wild: dmpr, in GGUF repo discussion #1, posts a command carrying --spec-type draft-dflash and three --spec-draft-* flags but no draft file of any kind.

How to check that speculation is actually running. Two log lines, both real. If the auto-detect fired you get auto-detected speculative type 'draft-dflash' from the draft model metadata; when the implementation is actually constructed you get adding speculative implementation 'draft-dflash'. A non-zero acceptance rate is the other signal. A drafter that never bound reports none of them.

⚠️ Use these filenames, not the ones you may have seen earlier. Meta republished the GGUFs on 12 Aug 2026 under canonical Q4_K names with a corrected embedded chat template; the card says the old one "Earlier GGUFs shipped a template that skipped both" normalisation steps and tells you to "re-download if you pulled before this fix". Since --jinja above uses the embedded template, this path is affected. Meta then deleted the superseded muse-glimmer-30B-kquant-17gb.gguf / dflash-kquant.gguf names on 2026-08-18 (commit 70bf1b61), so those names now 404; the new files are only ~2,800 bytes larger, so no memory figure in this recipe moves. mmproj-Muse-Glimmer-30B-Q4_K_M.gguf is byte-identical to the old mmproj-kquant.gguf — same object, new name.

Weights on this path total 18.429 GiB (15.606 + 1.304 + 1.519), so with the same 1.701 GiB KV the working set is 20.130 GiB — comfortably inside the pool. If you disable the sliding-window crop with --swa-full the KV rises to 6.500 GiB and the total to 24.929 GiB, which still fits.

Alternative: MLX via mlx-vlm

mlx-lm does not implement this architecture — there is no muse_glimmer.py under mlx_lm/models/, and a community port is still open as a PR (last touched 2026-08-20). mlx-vlm does, with a full muse_glimmer module including a real vision.py. It landed in v0.6.12: the module is absent at tag v0.6.10 and present at v0.6.12 (verified against a control file that resolves at both tags, so the 404 is absence rather than a missing ref). The current release is v0.6.15.

pip install -U "mlx-vlm>=0.6.12"
mlx_vlm.generate --model mlx-community/Muse-Glimmer-30B-4bit \
  --image screenshot.png --prompt "What does this dialog ask the user to do?"

That build is clean by the test that matters: its 815 vision tensors resolve through model.safetensors.index.json into model-00003/4-of-00004.safetensors — the same shards as the language weights — rather than into a vendor-specific side artifact that stock loaders never open. Its 19,414,804,113 B of safetensors (18.081 GiB) plus KV leaves 16.217 GiB spare.

This is the one path where the 48 GB pool buys you real fidelity. mlx-community publishes a full ladder and the whole middle of it fits: 6-bit is 24.583 GiB of weights (26.284 GiB with KV) and 8-bit is 31.084 GiB (32.785 GiB with KV, still 3.215 GiB inside the pool). Remember what you are trading, though — generation is bandwidth-bound, so on a 400 GB/s part the 8-bit build streams 1.72× the bytes per token that the 4-bit build does.

Alternative: Ollama, for a one-liner

ollama run muse-glimmer:30b

What that tag actually is, checked rather than assumed — and the check has three separate answers. Its registry manifest carries two content layers. The projector layer, 1,400,328,928 B at sha256:f48b452316f9…, is bit-identical to Meta's mmproj-kquant.gguf, whose HF LFS oid is the same hash. The model layer, 16,756,681,056 B at sha256:71b5c9c9abbc…, matches the byte count of Meta's superseded muse-glimmer-30B-kquant-17gb.gguf exactly but not its digest — the two files carry the same 731 tensors and the same kv_count of 32 in a different metadata key order, which changes the hash at identical length. So muse-glimmer:30b is Meta's K-Quant-17GB build repackaged, not a different quantisation and not the 19.65 GB K-Quant-Dynamic build; the 18 GB the library page displays is the sum of the two layers, not a third artifact's size. muse-glimmer:latest and muse-glimmer:30b-q4_K_M resolve to the same manifest. muse-glimmer:30b-q4_K_M-dflash is that same pair plus a third layer of 1,631,205,312 B at sha256:27d9a805fa29…, bit-identical to Meta's dflash-kquant.gguf — a superset, again not a different quant. All three digests were unchanged on 2026-08-21.

The tag actually aimed at this machine is muse-glimmer:30b-mlx, and it is not what its name suggests. Its manifest is byte-identical to muse-glimmer:30b-nvfp4-dflash — same manifest digest ef32a55b4976, same 1,503 layers, same 21,232,870,780 B total — so the "MLX" tag is that image under a second name. It is not the plain 30b-nvfp4 tag plus a drafter, and the difference matters if you use images: comparing the two manifests by tensor name, 1,443 layer names are shared and 555 of them carry different digests — 553 vision tensors plus config.json and hf_quant_config.json. The perception encoder is 3.74 GB here against 1.56 GB in 30b-nvfp4. The 59 layers present only in this tag are the drafter (draft.*, 58 tensors plus its own config.json). So choosing 30b-nvfp4 to save the difference buys you a coarser vision tower, quietly. NVFP4 here is an MLX weight format, not an NVIDIA hardware requirement; it runs on this GPU. At 19.775 GiB it sits well inside the 36.000 GiB pool. Ignore the bf16 tags at 57–65 GB on a 48 GB machine.

ollama run muse-glimmer:30b-mlx

ℹ️ Ollama's blobs predate the chat-template fix, and it does not matter — because Ollama never reads the template. The :30b and :30b-q4_K_M-dflash layer digests still match the superseded GGUFs, which makes them the last copies of those objects in public circulation now that Meta has removed the old names from the Hub. The obvious inference — that this route therefore serves the defective prompt — is wrong, and the manifests say so themselves: every muse-glimmer tag's config blob declares "renderer": "glimmer" and "parser": "glimmer", GGUF and MLX alike. In server/prompt.go renderPrompt returns the named renderer's output and never reaches m.Template.Execute, so the GGUF's embedded Jinja is not consulted at all; server/renderer_resolution.go passes glimmer through unchanged, and rendererForName resolves it to a Go-native GlimmerRenderer. That renderer carries both of the fixes the corrected Jinja gained: a replacer mapping four casings of Reasoning effort to Reasoning strength, and a glimmerHasSystemReasoning guard that suppresses the second directive when your system prompt already sets one. Ollama pins it, too — TestGlimmerRendererMatchesJinja2Reference checks the Go output against a byte-for-byte copy of Meta's template at revision a4e59da5, SHA-256 cfc67e5f349f…, the same hash this recipe prints as your check in step 1. So on the Ollama routes the template defect does not apply, whatever the blob age. One real version floor does apply instead: the GGUF tags declare "requires": "0.32.8" and the MLX tags "requires": "0.32.7".

Results

• Speed (ExecuTorch): no measurement exists for this chip, and the vendor's Apple figures run the wrong way to substitute for one. Meta's model card reports 23.7 tok/s without speculation and 37.8 with the DFlash drafter on an Apple M4 Max, and 26.6 / 50.2 on an M5 Max — batch size 1, greedy decoding, K-Quant-17GB. Both are newer and faster parts: the card names "MacBook M4-Max" with no memory configuration, and per Apple's M4 MacBook Pro specifications every M4 Max bin runs at 410 or 546 GB/s against this machine's 400. So those numbers are a ceiling this M3 Max will not reach, not a floor it will beat, and this recipe does not scale them into an estimate for you. Ran it? Please contribute the figure.
• Speed (llama.cpp-Metal): no M3 Max figure exists either, but the nearest Apple datapoint is unusually close in the only dimension that governs generation. CogniTechSystems' firsthand write-up measured a 4-bit Muse Glimmer on a MacBook Pro M4 Max 36 GB — a bin Apple sells only with the 14-core CPU, i.e. 410 GB/s, within 2.5% of this machine — and reports ~22 tok/s generation on a fresh context, ~7 tok/s about 29K tokens deep, and ~110 tok/s prefill, with the raw per-case JSON committed in the repo. Two caveats that are part of the number: they ran the unsloth UD-Q4_K_XL conversion rather than Meta's own build, and the M4 generation is one step newer than this one, so treat it as a close bound rather than as this card's value. /contribute a real M3 Max run and /check/muse-glimmer-30b/m3-max stops being empty.
• Speculation, and why the two runtimes disagree: the same write-up measured the DFlash drafter on llama.cpp-Metal at 0.9× to 1.0× — i.e. no gain — against Meta's 1.5× on ExecuTorch. The measurement stands: their command passes --spec-type draft-dflash explicitly, so the drafter was genuinely bound. Their explanation of the flag has since gone stale, which is the subject of the callout in Running. That is not a contradiction to resolve, it is the reason this page attaches every figure to a runtime. If speculative decoding is what you came for, the ExecuTorch path is the one with a vendor measurement behind it on Apple.
• Speed (MLX): none found across the space searched. Omitted rather than estimated — /contribute.
• Unified memory usage: 21.316 GiB working set for the ExecuTorch multimodal-plus-drafter build at the full 131,072-token context, against a 36.000 GiB addressable pool. Derived from published artifact bytes and the architecture's own config.json, not observed — see /check/muse-glimmer-30b/m3-max for live data as it lands.
• Quality notes: Meta's model card puts K-Quant-17GB at 1.0% average degradation across 15 benchmarks and K-Quant-Dynamic at 0.2%. K-Quant-Dynamic's text-image-dflash-metal build is 22.211 GiB and lands at 23.912 GiB with KV, leaving 12.088 GiB spare — so it fits this machine easily. It is not a free upgrade here the way it is on faster silicon, though: it streams about 13% more bytes per token, and on a 400 GB/s part that comes straight off the token rate. Take it when accuracy matters more than latency.

For the full benchmark data, see /check/muse-glimmer-30b/m3-max.

Troubleshooting

unknown model architecture: 'muse-glimmer'

Your llama.cpp predates the architecture. Releases b10344 and older do not register it and reject the file in about a fifth of a second, before allocating any memory — which is why changing context length or offload settings appears to do nothing. Use b10353 or newer, or confirm a source checkout with grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp. A source build from before that release is fine as long as it is after the merge: a user in GGUF repo discussion #1 answers this exact error with "Works fine if you build latest from source," and shows version: 10352.

In LM Studio the same failure surfaces as Engine protocol runtime llama-server exited before becoming healthy. exitCode=1, with the real message only in ~/.lmstudio/server-logs/. A community reporter who diagnosed it on an M5 Pro documents the fix in discussion #42: update the llama.cpp runtime to llama.cpp-mac-arm64-apple-metal-advsimd 2.28.2 or newer via lms runtime update. They note that "any Apple-silicon machine needs runtime 2.28.2 or newer to load Muse Glimmer at all."

Throughput collapses when the perception encoder is loaded

This is a memory-fit failure, and on a 48 GB M3 Max it should not happen — but it is worth understanding, because it is the reason this recipe specifies a 36 GB floor. A firsthand M4 Pro 24 GB run dropped from a median 14.40 tok/s text-only to 3.67 tok/s once the 1.40 GB projector was loaded. The author reports the measurement without diagnosing it; the arithmetic is ours, and it lands exactly on the boundary. A 24 GB Mac addresses 2/3 of 24 GiB = 16.000 GiB. The text weights alone are 15.606 GiB — inside the pool by 0.394 GiB. Adding the 1.304 GiB projector takes the total to 16.910 GiB, which overshoots by 0.910 GiB, before any KV cache. A 48 GB M3 Max addresses 36.000 GiB and never approaches that wall.

The same run also found the DFlash drafter reduced median decode speed by 27.6% when it had to be CPU-offloaded. That is a symptom of the same shortage, not a property of the drafter — though note that the parity result on the M4 Max 36 GB cited in Results was not memory-starved, so on llama.cpp-Metal the drafter's ceiling is genuinely lower than on ExecuTorch.

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

Scope: the llama.cpp-Metal alternative only. The ExecuTorch path above never reaches this code, because the drafter is part of the exported artifact rather than a separately bound model.

The message is an out-of-memory condition wearing a bounds-check error's clothes. llama.cpp prints it when every visible device reports zero free memory as a model loads — nothing to do with this model, this drafter, or Metal.

Issue #26894 reads as though it were. It was filed against Meta's own GGUF and traced the crash to one metadata key, muse-glimmer.attention.sliding_window_pattern, which Meta's builds encode as a 52-entry boolean array where third-party conversions write the scalar 4 — an encoding difference that is genuinely there. The reporter withdrew the diagnosis on 2026-08-13 — "My original diagnosis was wrong" — and named the real mechanism instead: "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the file the issue names, verified its SHA-256 against the Hub, and bound the drafter 10 times out of 10, concluding that the published file is fine as it stands. The encoding difference was real and irrelevant: he had tested the rewritten file later, on a machine that happened to have memory free. The issue is still open as of 2026-08-21, with no further comments since the retraction — but its title still carries the withdrawn diagnosis, which is why most of the web still repeats it.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum.
• The zero guard there catches only free == 0 and total == 0 — a device with nothing at all to report (#18577). A device that is merely full reports free == 0 against a real total and falls straight through it.
• Every splits entry is then 0, so split_sum == 0 and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs matches nothing (x < NaN is false), returns end(), and devices.at(n_devices()) throws.

What that means on Metal specifically, which no report covers. ggml's Metal device reports total = recommendedMaxWorkingSetSize and free = total − currentAllocatedSize (ggml-metal-device.m), and on any macOS from 10.12 onward total is never zero — so the free == 0 && total == 0 guard cannot fire here. free reaches zero when allocations have filled the 36.000 GiB pool exactly — and exactly is the word, because both operands are unsigned: macOS lets a process allocate past recommendedMaxWorkingSetSize, and one byte beyond it the subtraction wraps to a huge value instead of saturating at zero. So this is a knife-edge rather than a regime, and the crash is correspondingly harder to reach here than on a discrete card. When it does fire the shape is the one in the original report: the ~20 GB target loads first, and the drafter is what falls over. Quit whatever else is holding the GPU, check with sudo powermetrics --samplers gpu_power or Activity Monitor, and re-run. Unless your build defines GGML_METAL_NDEBUG, ggml also logs a warning the moment currentAllocatedSize passes the working-set size — that is the early signal, ahead of the crash. If you still cannot fit, lower -c or drop -md … --spec-type draft-dflash -ngld 99.

Do not rewrite the GGUF's metadata. The scalar-sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and leaves you running a file whose checksum no longer matches the Hub.

Two things about the surrounding evidence are worth keeping, because they were true before the retraction and remain true after it. Three independent users bind this drafter successfully — on b10358 against this recipe's exact kquant-17gb file on a 24 GB RX 7900 XTX under ROCm (GGUF repo discussion #2, "now working well for me after I built the latest release of llama.cpp"), and twice in discussion #34, once with a startup log containing "adding speculative implementation 'draft-dflash'". And every report either way is CUDA or ROCm: nobody has exercised this path on Metal in either direction, so if you run the llama.cpp alternative here you are the first datapoint — please contribute it.

One genuinely separate change did land: PR #26900 — "model : disallow integer dflash sliding_window_pattern", merged 2026-08-12 — which is a single-line swap of get_key_or_arr for get_arr in the drafter's own hparams read in src/models/dflash.cpp. Its author wrapped the Nixes #26894 line of the PR description in a strikethrough, and #26894 did not close when it merged — correctly, as it turns out. The issue remains open, now in effect as a request for a clear "insufficient device memory" message in place of the out_of_range that sent the original investigation down the wrong path.

The newer DFlash2 drafter is not a path on this machine yet

A DFlash 2 drafter for this model appeared on 2026-08-18/19, published as byte-identical mirrors: the GGUF conversions at z-lab/Muse-Glimmer-30B-DFlash2-GGUF and incoai/Muse-Glimmer-30B-DFlash2-GGUF, and the safetensors original at incoai/Muse-Glimmer-30B-DFlash2, whose card reports longer accepted blocks than Meta's own drafter. Do not reach for them here yet, for a mechanical reason: mainline llama.cpp registers dflash and has no dflash2 architecture at all, and the drafter's own card tells you to build PR #27342, which was still open and blocked on 2026-08-21. The throughput comparison behind that claim is, in that card's own words, "SGLang on one NVIDIA H200, with FlashAttention 3 for target and draft attention" — a stack with no Apple counterpart — and the ExecuTorch path on this page has no slot for a swapped drafter at all, because its drafter is compiled into the .pte. Revisit when that PR merges.

The M5 tensor-API warning does not apply to this machine

If you read reports of the tensor API is not supported in this environment - disabling costing 2–3× on prompt processing, that is M5-only. The same discussion is explicit that "the tensor API targets the GPU Neural Accelerators introduced with the M5 generation" and that "llama.cpp explicitly disables it for pre-M5 devices because the hardware is not there". An M3 Max has no such accelerators to leave idle.

Nothing comes back on a tool call

Never treat <|eom|> as a stop token. The stop tokens are <|end_of_text|> (200001) and <|eot|> (200008); <|eom|> ends a single message and the turn continues past it. On a tool call the model emits a private reasoning message closed by <|eom|> first, so stopping there means the tool call is never generated. The bundled runtimes handle this; it bites only if you drive the raw runner from your own client.

Reasoning cannot be turned off, and long generations can silently return nothing

Scope: the llama.cpp paths. On the ExecuTorch server above, reasoning_effort is rejected with a structured 400 rather than ignored, and --reasoning off / --reasoning-budget do not exist at all.

The chat template opens the thinking channel unconditionally, so --reasoning off and "reasoning_effort": "none" have no effect — control the depth instead with reasoning_strength (low/medium/high/xhigh, default high), or hard-cap it with --reasoning-budget N. On llama-server, also remember that -c is divided across -np slots, so "a single request gets" -c / -np; check n_ctx_slot in the startup log. Meta's own published server command uses -np 4, so a copied invocation quietly gets a quarter of the context you asked for. A generation that exhausts its slot context produces no answer and logs no error, which reads as a wrong result rather than a failure.

One request at a time

The ExecuTorch server is deliberately serial: --num-runners must be 1, every exported method is batch-1, and "One request executes at a time." Named sessions give isolation between conversations, not concurrency. Video is not supported on this path either — "Text and images only, one image per request, JPEG or PNG."

Report anything else via the submission form.