Muse Glimmer 30B on Apple M4 Max: ExecuTorch Metal agent server with vision and DFlash

What You'll Build

An OpenAI-compatible agent server running Meta's Muse Glimmer 30B entirely on a MacBook Pro M4 Max, with image input and speculative decoding both switched on — and enough unified memory left over that you never touch a wired-limit override. The interesting part is the runtime: Meta measured this model on Apple Silicon with ExecuTorch, not with llama.cpp and not with MLX-LM, and ships pre-exported Metal artifacts for it.

Hardware data: Apple M4 Max (48 GB unified memory, 546 GB/s) · working set 21.316 GiB against a 36.000 GiB addressable pool · See benchmark data

ℹ️ Three Apple paths exist, and they are not interchangeable evidence. ExecuTorch, llama.cpp-Metal and MLX all run this model today, and this recipe verified each one separately. Meta's published Apple tok/s figures were produced only on ExecuTorch — the model card states "M4/M5 measurements were done using ExecuTorch, and RTX using llama.cpp." A number measured on ExecuTorch says nothing about MLX or llama.cpp throughput, so this recipe attaches every figure to the runtime that produced it and omits figures for the runtimes where this round found no measurement.

Requirements

Which M4 Max this is. Apple sells the M4 Max in two bins and the memory size identifies the bin: per Apple's MacBook Pro specifications, 36 GB unified memory goes with the 14-core-CPU M4 Max at 410 GB/s, and 48/64/128 GB only with the 16-core-CPU / 40-core-GPU part at 546 GB/s. A 48 GB M4 Max is therefore unambiguously the 546 GB/s bin. This matters because token generation is memory-bandwidth-bound.

The memory budget (derived, not measured)

Apple has no dedicated VRAM. Our catalogue stores full unified memory in the GPU row, but Metal's recommendedMaxWorkingSetSize caps what the GPU can actually address: exactly 3/4 on machines of 36 GB and above, so a 48 GB Mac addresses 36.000 GiB. Everything below is framed against that number, not against 48.

The KV cache is derived from the model's own config.json, not from prose. That file declares num_key_value_heads: 2, head_dim: 128, sliding_window: 2048, and a layer_types array containing 39 sliding_attention and 13 full_attention entries — the [Local, Local, Local, Global] repeating pattern from the model card, counted out of the artifact.

Per-token KV is 2 (K+V) × 2 KV heads × 128 head dim × 2 bytes = 1024 B per layer — 1 KiB, which is why a 131K context costs under 2 GiB here. The first-party GGUF card makes the same point in prose: "KV cache stays cheap".

Consequence: no sudo sysctl iogpu.wired_limit_mb raise is needed on this machine, and you should not add one. The full multimodal-plus-drafter stack sits 14.684 GiB inside the default pool.

Why 36 GB is the floor. A 32 GB Mac addresses exactly 2/3 of 32 GiB = 21.333 GiB, against this build's 21.316 GiB working set — a 0.017 GiB margin with macOS still to house. A 36 GB Mac addresses 27.000 GiB and clears it comfortably. And 36 GB is a real shipping configuration, not a rounded-off number: Apple's specification page lists 36 GB unified memory as a sold SKU of the 14-core-CPU M4 Max, alongside 48 GB, 64 GB and 128 GB on the 16-core-CPU part. That is where min_vram_gb: 36 comes from — it is a filter floor naming a machine you can actually buy, not this recipe's measured peak.

Installation

1. Download one ExecuTorch variant

The repo publishes 16 pre-exported variants and is 372 GB in total, so --include is mandatory. The metal builds are self-contained: unlike the CUDA builds they carry their weights inside the .pte, with no separate blob to fetch.

pip install huggingface_hub

REPO=meta-models/Muse-Glimmer-30B-ExecuTorch-PTE
VARIANT=muse-glimmer-k-quant-17G-128K-text-image-dflash-metal
LOCAL_DIR=./muse-glimmer-pte

hf download "$REPO" \
  --include "$VARIANT/*" \
  --include "tokenizer.json" \
  --include "tokenizer_config.json" \
  --include "chat_template.jinja" \
  --local-dir "$LOCAL_DIR"

⚠️ Re-pull chat_template.jinja if you downloaded it before 2026-08-15. The ExecuTorch repo kept its own copy of the template, and until commit fc6fa93c that copy was a stale one — it still named a different model family in its raise_exception string, and, more importantly, it lacked the system-prompt normalisation the base repo's template had gained. The current file is byte-identical to the base repo's (SHA-256 cfc67e5f349f…, 9,992 B against the old 7,167 B). What the new logic does, in its own words: it normalises four casings of "Reasoning effort" to "Reasoning strength", then emits the kwarg-driven directive only {%- if 'reasoning strength' not in (sys_text | lower) -%}. Under the old template a system prompt that already carried a reasoning line got a second, conflicting one appended — the same defect the GGUF card describes for the pre-2026-08-12 GGUFs, which survived three days longer here.

The variant name is a fixed scheme — muse-glimmer-<quant>-128K-<modality>-<decoding>-<backend>. k-quant-17G is the ~4-bit build Meta targets at a 24 GB envelope, text-image adds the perception encoder, dflash bundles the speculative-decoding drafter, and metal is the Apple Silicon backend. The repo publishes no CPU build: the card states "There is no CPU variant."

Two byte-level cross-checks on that artifact, both of which you can repeat against the HF tree API. Subtracting the text-solo build from the text-image-solo build gives a perception encoder of 1,413,208,832 B; doing the same subtraction inside the dflash pair gives 1,413,208,704 B — the same figure to within 128 bytes of serialization padding, and within 1% of the 1,400,328,928 B mmproj-Muse-Glimmer-30B-Q4_K_M.gguf that the GGUF repo publishes separately. The drafter subtracts out at 1,695,925,376 B and 1,695,925,248 B by the same two routes.

2. Build the ExecuTorch runner

⚠️ The Metal compiler is not in the Command Line Tools. ExecuTorch's MLX backend states its own requirement plainly. It needs the Metal compiler, "which ships with Xcode (not the standalone Command Line Tools)", and it "requires Apple Silicon (M1 or later)" — a floor this machine clears comfortably. Check before you build:

xcrun -sdk macosx --find metal

A path means you are set. An error means either Xcode is not installed, or it is installed and the active developer directory still points at the Command Line Tools — in which case sudo xcode-select -s /Applications/Xcode.app/Contents/Developer fixes it.

A download is not enough — the pre-exported artifact skips the export step, not the native runtime. Follow the ExecuTorch build-from-source guide to get a checkout configured, then build the model preset from the repo root. The Apple preset is muse-glimmer-mlx, and it is gated to Darwin in CMakePresets.json:

(cd examples/models/muse-glimmer && cmake --workflow --preset muse-glimmer-mlx)
pip install -r examples/llm_server/python/requirements.txt

Binaries land in cmake-out/examples/models/muse-glimmer/: solo_runner, dflash_runner, and muse_glimmer_worker. The server needs the last one.

This machine has no CUDA and none of the NVIDIA escape hatches apply: skip the muse-glimmer-cuda preset, skip every sm80+ptx variant in the repo, and ignore the third-party FP8 / NVFP4 conversions on the Hub, which target Blackwell and Hopper tensor hardware Apple Silicon does not have. Meta ships no such build itself — the meta-models org has exactly four repos and the card never mentions FP8 or FP4 — and its RTX 5090 figure is the K-Quant-17GB build under llama.cpp, by the speed table's own footnote. Skip pip install flash-attn as well; on Metal the attention path is MLX-native. Under its MLX heading, PyTorch's announcement lists that "RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations are lowered to MLX-native or custom Metal implementations".

Running

Start the OpenAI-compatible server from the ExecuTorch repository root. The Metal build takes no --data-path; text-image needs --pos-embed-path, and dflash needs no flag at all because the exported method contract is auto-detected.

VARIANT=muse-glimmer-k-quant-17G-128K-text-image-dflash-metal
LOCAL_DIR=./muse-glimmer-pte

python -m executorch.examples.models.muse_glimmer.serving.serve \
  --model-path "$LOCAL_DIR/$VARIANT/$VARIANT.pte" \
  --pos-embed-path "$LOCAL_DIR/$VARIANT/pos_embed.bin" \
  --tokenizer-path "$LOCAL_DIR/tokenizer.json" \
  --hf-tokenizer "$LOCAL_DIR" \
  --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
  --model-id muse-glimmer-30B \
  --tool-parser atem \
  --host 127.0.0.1 --port 8000

Smoke-test it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"muse-glimmer-30B","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":32,"temperature":0}'

--tool-parser atem is what converts Muse Glimmer's native tool output into OpenAI tool_calls; the default is none, so an agent harness needs it passed explicitly. Reasoning depth is a system-prompt setting (Reasoning strength: low|medium|high|xhigh), not the OpenAI reasoning_effort parameter, which this server rejects with a structured 400.

One Apple-specific bonus worth knowing. The DFlash block length is exported dynamically and the accepted range differs by backend: the Metal build accepts --dflash-block-length in [2, 16] while the CUDA build accepts only [2, 4]. The Metal path is the one that can run the drafter at its full trained block size of 16.

⚠️ Two different DFlash mechanisms share a name — do not mix their flags. On this ExecuTorch path the drafter is baked into the exported .pte method contract: the text-image-dflash variant is the drafter, --artifact-mode defaults to auto and detects it, and --dflash-block-length is an ExecuTorch server flag. On the llama.cpp path below the drafter is a separate GGUF file bound with -md / -ngld and driven by llama.cpp's own draft-dflash speculative type. Neither set of flags exists in the other runtime.

Alternative: llama.cpp-Metal with the first-party GGUF

Mainline llama.cpp registers this architecture — LLM_ARCH_MUSE_GLIMMER is at line 74 of src/llama-arch.cpp, added by PR #26841 (merged 2026-08-10, commit 62bf73d) and first released in build b10353. Metal is on by default on macOS, so omit any GPU backend flag:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DBUILD_SHARED_LIBS=OFF
cmake --build build --config Release -j --target llama-cli llama-mtmd-cli llama-server

hf download meta-models/Muse-Glimmer-30B-GGUF \
    --local-dir Muse-Glimmer-30B-GGUF \
    --include "Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf" \
    --include "mmproj-Muse-Glimmer-30B-Q4_K_M.gguf" \
    --include "dflash-Muse-Glimmer-30B-Q4_K_M.gguf"

./build/bin/llama-server \
    -m       Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
    --mmproj Muse-Glimmer-30B-GGUF/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
    -md      Muse-Glimmer-30B-GGUF/dflash-Muse-Glimmer-30B-Q4_K_M.gguf -ngld 99 \
    --spec-type draft-dflash \
    -a muse-glimmer-30B \
    -ngl 99 -c 131072 -np 1 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64

Image input on this path goes through llama-mtmd-cli rather than llama-cli, and vision is genuinely implemented rather than merely implied by the presence of an mmproj file: llama.cpp's multimodal layer carries a dedicated PROJECTOR_TYPE_MUSE_GLIMMER, a graph builder at tools/mtmd/models/muse-glimmer.cpp, and its own image preprocessor.

Speculative decoding is a first-class type here too, not an improvisation: COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH has a dedicated implementation in common/speculative.cpp that reads the trained block size from the dflash.block_size GGUF metadata key, and LLM_ARCH_DFLASH is a registered architecture.

⚠️ --spec-type without a draft model silently does nothing — and the reverse used to be true too, until 2026-08-13. Worth the paragraph, because neither case errors.

common_speculative_init adds the DFlash implementation only when both conditions hold: the type bit is set and a draft context exists.

• --spec-type without a draft model — still true today. The DFlash config is gated on params.draft.ctx_dft != nullptr, so with no draft path there is no context to bind and the implementation is skipped before it can assert. dmpr, in GGUF repo discussion #1, posts a command carrying --spec-type draft-dflash and three --spec-draft-* flags but no draft file of any kind — this half, in the wild.
• -md without --spec-type — true only on builds from before 2026-08-13. common_params_speculative::types used to default to { COMMON_SPECULATIVE_TYPE_NONE } with nothing writing to it: NONE is enum 0 and DRAFT_DFLASH is 4, so the default produced the bitmask 1, which matches nothing, and the drafter loaded into memory unused. PR #26814 ("common : auto-detect spec type from draft GGUF metadata", merged 2026-08-13T10:34:28Z, first released in b10413) changed that: common/arg.cpp now infers the type whenever none was requested and a -md path is set, and common_speculative_types_from_gguf reads the draft's general.architecture — answering draft-dspark for a draft carrying markov_w1.weight and draft-dflash for one that does not. Meta's drafter declares dflash and carries no Markov head — checked by walking all 58 tensor-info records of the file, not just its metadata block, because the DSpark/DFlash discriminator is a tensor — so from b10413 onward -md alone binds it as draft-dflash.

Two consequences worth stating plainly. The GGUF card's own "add speculative decoding" snippet — -md … -ngld 99 with no --spec-type — is correct from b10413 onward; an earlier revision of this recipe called it broken, which was true when written and is not now. And passing --spec-type draft-dflash anyway is harmless and still right for anyone on an older binary, which is why the command above keeps it. Either way, check that speculation actually ran: look for adding speculative implementation 'draft-dflash' in the startup log, or for a non-zero acceptance rate. A drafter that never bound reports neither.

⚠️ Use these filenames, not the ones you may have seen earlier. Meta republished the GGUFs on 12 Aug 2026 under canonical Q4_K names with a corrected embedded chat template; the card says the old one "Earlier GGUFs shipped a template that skipped both" normalisation steps and tells you to "re-download if you pulled before this fix". Since --jinja above uses the embedded template, this path is affected. Meta deleted the superseded muse-glimmer-30B-kquant-17gb.gguf / dflash-kquant.gguf on 2026-08-18, so those names now 404; the new files are only ~2,800 bytes larger, so no memory figure in this recipe moves. mmproj-Muse-Glimmer-30B-Q4_K_M.gguf is byte-identical to the old mmproj-kquant.gguf — same object, new name.

Weights on this path total 18.429 GiB (15.606 + 1.304 + 1.519), so with the same 1.701 GiB KV the working set is 20.130 GiB — comfortably inside the pool. If you disable the sliding-window crop with --swa-full the KV rises to 6.500 GiB and the total to 24.929 GiB, which still fits.

Alternative: MLX via mlx-vlm

mlx-lm does not implement this architecture — there is no muse_glimmer.py under mlx_lm/models/, and a community port is still open as a PR. mlx-vlm does, with a full muse_glimmer module including a real vision.py. It landed in v0.6.12: the module is absent at tag v0.6.10 and present at v0.6.12 (verified against a control file that resolves at both tags, so the 404 is absence rather than a missing tag).

pip install -U "mlx-vlm>=0.6.12"
mlx_vlm.generate --model mlx-community/Muse-Glimmer-30B-4bit \
  --image screenshot.png --prompt "What does this dialog ask the user to do?"

That build is clean by the test that matters: its 815 vision tensors resolve through model.safetensors.index.json into model-00003/4-of-00004.safetensors — the same shards as the language weights — rather than into a vendor-specific side artifact that stock loaders never open. Its 19,414,804,113 B of safetensors (18.081 GiB) plus KV leaves 16.218 GiB spare.

Alternative: Ollama, for a one-liner

ollama run muse-glimmer:30b

What that tag actually is, checked rather than assumed — and the check has three separate answers. Its registry manifest carries two content layers. The projector layer, 1,400,328,928 B at sha256:f48b452316f9…, is bit-identical to Meta's mmproj-kquant.gguf, whose HF LFS oid is the same hash. The model layer, 16,756,681,056 B at sha256:71b5c9c9abbc…, matches the byte count of Meta's muse-glimmer-30B-kquant-17gb.gguf exactly but not its digest (7e9b74b7c887…). Reading both headers, the metadata key order differs — Meta emits general.quantization_version immediately before the tokenizer block and Ollama's copy does not — which changes the hash at identical length. (general.name is Muse Glimmer Hf in both files, so it witnesses nothing.) So muse-glimmer:30b is Meta's K-Quant-17GB build repackaged, not a different quantisation and not the 19.65 GB K-Quant-Dynamic build; the 18 GB the library page displays is the sum of the two layers, not a third artifact's size. muse-glimmer:30b-q4_K_M-dflash is this same pair plus a third layer of 1,631,205,312 B at sha256:27d9a805fa29…, bit-identical to Meta's dflash-kquant.gguf — a superset of this tag, again not a different quant. Skip the 57–65 GB bf16 tags on a 48 GB machine. The MLX tags are a different matter, and the naming misleads: muse-glimmer:30b-mlx and muse-glimmer:30b-nvfp4-dflash are the same image, layer for layer — 1,503 layers, 21,232,870,780 B, 19.775 GiB — so "NVFP4" here is MLX's own microscaling quantisation mode, not a CUDA format, and dismissing the tag by its name discards the build actually aimed at this machine. It carries the drafter, and it fits the 36.000 GiB pool with room to spare; prefer it over :30b if you want Ollama to run the MLX path rather than the GGUF one. 30b-mxfp8-dflash is the larger sibling at 32.823 GiB — 1,505 layers — which also fits, with less headroom.

⚠️ Ollama's blobs predate the chat-template fix — but that may not be what you get. Its layer digests still match the superseded GGUFs. The template defect does not reach you on this route, though, whatever the blobs' age. Every muse-glimmer tag's config blob — GGUF and MLX alike — declares "renderer": "glimmer" and "parser": "glimmer", and in server/prompt.go renderPrompt returns from the named renderer before it can reach m.Template.Execute, so the artifact's embedded Jinja is never consulted. The Go renderer carries both of the fixes the corrected template gained: model/renderers/glimmer.go maps all four casings of Reasoning effort to Reasoning strength, then appends its own directive only if defaultSystem || !glimmerHasSystemReasoning(content) — the Go equivalent of the template's {%- if 'reasoning strength' not in (sys_text | lower) -%}. Ollama's own reference test pins that renderer against a fixture its comment records as copied byte-for-byte from the chat template at publisher revision a4e59da5…, SHA-256 cfc67e5f349f… — the corrected one. Scope: this settles prompt formatting only. The weights behind these tags are still the superseded objects, and nothing here says what parser: glimmer does on the output side.

Those Ollama blobs are now the last copies of the superseded objects in public circulation, since Meta removed the old names from the Hub on 2026-08-18.

Results

• Speed (ExecuTorch only): Meta reports 23.7 tok/s without speculation and 37.8 tok/s with the DFlash drafter on an Apple M4 Max, batch size 1, greedy decoding, using the K-Quant-17GB build. Two caveats that are part of the number. First, the card names "MacBook M4-Max" without a memory configuration, and the M4 Max ships in a 410 GB/s bin (36 GB only) and a 546 GB/s bin (48 GB and up) — so this figure may have been produced on the slower bin, in which case it is a floor for a 48 GB machine rather than a match. Second, it is an ExecuTorch measurement and does not transfer to the llama.cpp or MLX paths above.
• Speed (llama.cpp-Metal): this round turned up no M4 Max figure. The space searched, so you can judge the gap: every discussion thread on the canonical repo (49), on unsloth/Muse-Glimmer-30B-GGUF (14), on the first-party GGUF and ExecuTorch repos, and on the mlx-community build — each fetched and read individually rather than sampled — plus two web searches. The nearest independent datapoint that surfaced is a hands-on run on an M4 Pro 24 GB — a different, lower-bandwidth chip at 273 GB/s against this machine's 546 GB/s — which recorded a median 14.40 decode tok/s text-only on this exact quant. Because the generation gap runs toward the slower part, treat 14.40 as a pessimistic floor the M4 Max should beat, never as this card's number. Measured it yourself? Please contribute it so /check/muse-glimmer-30b/m4-max stops being empty.
• Speed (MLX): none found across the same space. Omitted rather than estimated — /contribute.
• Unified memory usage: 21.316 GiB working set for the ExecuTorch multimodal-plus-drafter build at the full 131,072-token context, against a 36.000 GiB addressable pool. Derived from published artifact bytes and the architecture's own config.json, not observed — see /check/muse-glimmer-30b/m4-max for live data as it lands.
• Quality notes: Meta puts K-Quant-17GB at 1.0% average degradation across 15 benchmarks and K-Quant-Dynamic at 0.2%. K-Quant-Dynamic's text-image-dflash-metal build is 23.85 GB and also fits this machine, so on a 48 GB M4 Max the higher-fidelity build is the free upgrade the 24 GB target hardware cannot take.

For the full benchmark data, see /check/muse-glimmer-30b/m4-max.

Troubleshooting

unknown model architecture: 'muse-glimmer'

Your llama.cpp predates the architecture. Releases b10344 and older do not register it and reject the file in about a fifth of a second, before allocating any memory — which is why changing context length or offload settings appears to do nothing. Use b10353 or newer, or confirm a source checkout with grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp. A source build from before that release is fine as long as it is after the merge: a user in GGUF repo discussion #1 answers this exact error with "Works fine if you build latest from source," and shows version: 10352.

In LM Studio the same failure surfaces as Engine protocol runtime llama-server exited before becoming healthy. exitCode=1, with the real message only in ~/.lmstudio/server-logs/. A community reporter who diagnosed it on an M5 Pro documents the fix in discussion #42: update the llama.cpp runtime to llama.cpp-mac-arm64-apple-metal-advsimd 2.28.2 or newer via lms runtime update. They note that "any Apple-silicon machine needs runtime 2.28.2 or newer to load Muse Glimmer at all."

Throughput collapses when the perception encoder is loaded

This is a memory-fit failure, and on an M4 Max it should not happen — but it is worth understanding, because it is the reason this recipe specifies a 48 GB machine. The M4 Pro 24 GB run cited above dropped from 14.40 tok/s text-only to 3.67 tok/s once the 1.40 GB projector was loaded. The author reports the measurement without diagnosing it; the arithmetic is ours, and it lands exactly on the boundary. A 24 GB Mac addresses 2/3 of 24 GiB = 16.000 GiB. The text weights alone are 15.606 GiB — inside the pool by 0.394 GiB. Adding the 1.304 GiB projector takes the total to 16.910 GiB, which overshoots by 0.910 GiB, before any KV cache. A 48 GB M4 Max addresses 36.000 GiB and never approaches that wall.

The same run also found the DFlash drafter reduced median decode speed by 27.6% when it had to be CPU-offloaded. That is a symptom of the same shortage, not a property of the drafter — but it is a reminder that the speculative path only pays when the drafter fits alongside everything else.

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

Scope: the llama.cpp-Metal alternative only. The ExecuTorch path above never reaches this code, because the drafter is part of the exported artifact rather than a separately bound model.

The message is an out-of-memory condition wearing a bounds-check error's clothes. llama.cpp prints it when every visible device reports zero free memory as a model loads — nothing to do with this model, this drafter, or Metal.

Issue #26894 reads as though it were. It was filed against Meta's own GGUF and traced the crash to one metadata key, muse-glimmer.attention.sliding_window_pattern, which Meta's builds encode as a 52-entry boolean array where third-party conversions write the scalar 4 — a difference I measured directly out of the file headers at the time, and which is genuinely there. The reporter withdrew the diagnosis on 2026-08-13 — "My original diagnosis was wrong" — and named the real mechanism instead: "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the file the issue names, verified its SHA-256 against the Hub, and bound the drafter 10 times out of 10, concluding that the published file is fine as it stands. The encoding difference was real and irrelevant: he had tested the rewritten file later, on a machine that happened to have memory free.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum.
• The zero guard there catches only free == 0 and total == 0 — a device with nothing at all to report (#18577). A device that is merely full reports free == 0 against a real total and falls through it.
• Every splits entry is then 0, so split_sum == 0 and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs matches nothing (x < NaN is false), returns end(), and devices.at(n_devices()) throws.

It surfaced on the draft model in the original report because the ~20 GB target loads first and takes the device with it. The reporter reproduced it with an LD_PRELOAD shim forcing free = 0 and naturally, by saturating the GPU from another process — including on an unrelated dense model with no speculative decoding at all.

What to do on this machine. Unified memory makes this less likely than on a discrete card, but not impossible: Metal's default working-set limit is a fraction of installed RAM, not all of it, and this build's derived working set is 21.316 GiB against a 36.000 GiB addressable pool. Quit whatever else is holding the GPU, check with sudo powermetrics --samplers gpu_power or Activity Monitor, and re-run. If you still cannot fit, lower -c or drop -md … --spec-type draft-dflash -ngld 99.

Do not rewrite the GGUF's metadata. The scalar-sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and leaves you running a file whose checksum no longer matches the Hub.

Two things about the surrounding evidence are worth keeping, because they were true before the retraction and remain true after it. Three independent users bind this drafter successfully — on b10358 against this recipe's exact kquant-17gb file on a 24 GB RX 7900 XTX under ROCm (GGUF repo discussion #2), and twice in discussion #34, once with a startup log containing "adding speculative implementation 'draft-dflash'". And every report either way is CUDA or ROCm: nobody has exercised this path on Metal in either direction, so if you run the llama.cpp alternative here you are the first datapoint — please contribute it.

One genuinely separate change did land: PR #26900 ("disallow integer dflash sliding_window_pattern", merged 12 August) swaps get_key_or_arr for get_arr in the drafter's own hparams read in src/models/dflash.cpp. Its author struck through the Nixes #26894 line in the PR description and GitHub's timeline records the reference with willCloseTarget: false — correctly, as it turns out. The issue remains open, now as a request for a clear "insufficient device memory" message in place of the out_of_range that sent the original investigation down the wrong path.

The M5 tensor-API warning does not apply to this machine

If you read reports of the tensor API is not supported in this environment - disabling costing 2–3× on prompt processing, that is M5-only. The same discussion is explicit that "the tensor API targets the GPU Neural Accelerators introduced with the M5 generation" and that "llama.cpp explicitly disables it for pre-M5 devices because the hardware is not there". An M4 Max has no such accelerators to leave idle.

Nothing comes back on a tool call

Never treat <|eom|> as a stop token. The stop tokens are <|end_of_text|> (200001) and <|eot|> (200008); <|eom|> ends a single message and the turn continues past it. On a tool call the model emits a private reasoning message closed by <|eom|> first, so stopping there means the tool call is never generated. The bundled runtimes handle this; it bites only if you drive the raw runner from your own client.

Reasoning cannot be turned off, and long generations can silently return nothing

Scope: the llama.cpp paths. On the ExecuTorch server above, reasoning_effort is rejected with a structured 400 rather than ignored, and --reasoning off / --reasoning-budget do not exist at all.

The chat template opens the thinking channel unconditionally, so --reasoning off and "reasoning_effort": "none" have no effect — control the depth instead with reasoning_strength (low/medium/high/xhigh, default high), or hard-cap it with --reasoning-budget N. On llama-server, also remember that -c is divided across -np slots, so "a single request gets" -c / -np; check n_ctx_slot in the startup log. A generation that exhausts its slot context produces no answer and logs no error, which reads as a wrong result rather than a failure.

One request at a time

The ExecuTorch server is deliberately serial: --num-runners must be 1, every exported method is batch-1, and "One request executes at a time." Named sessions give isolation between conversations, not concurrency. Video is not supported on this path either — "Text and images only, one image per request, JPEG or PNG."

Report anything else via the submission form.