Muse Glimmer 30B on Apple M2 Max: 8-bit MLX with vision and the DFlash drafter

What You'll Build

A vision-capable Muse Glimmer 30B running on a 64 GB MacBook Pro or Mac Studio at 8-bit — not the 4-bit K-quant every other card in this catalogue is limited to — with Meta's DFlash speculative drafter loaded alongside it, and still 10 GiB of Metal working set to spare. This is the one machine in the catalogue where that combination fits, and working out why is most of the recipe.

Hardware data: Apple M2 Max (64 GB unified memory, ~400 GB/s) · working set 37.585 GiB against a 48.000 GiB addressable pool · See benchmark data

⚠️ The vendor's "Full Precision → 64GB VRAM" row does not mean this 64 GB machine. The model card's quantization table lists Full Precision against 64GB VRAM. On a discrete accelerator that reads straight across; on Apple it does not, because Apple's memory is unified and Metal hands the GPU only a fixed fraction of it. The arithmetic is two sections below and it is decidable, not a judgement call: bf16 weights are 55.463 GiB against a 48.000 GiB pool. The honest top of this machine is 8-bit, which is still a rung above every other card here.

Requirements

Which M2 Max this is. The M2 Max does not split by memory bandwidth the way later Max parts do — both its 30-core and 38-core GPU bins run at roughly 400 GB/s, and 64 GB is the full 38-core die. Per Apple's MacBook Pro specifications it ships in 32 GB, 64 GB and 96 GB configurations only; there is no 48 GB M2 Max. Token generation is memory-bandwidth-bound, so 400 GB/s is the figure that governs speed here, and numbers measured on later, faster Max parts do not forward to it.

What 64 GB actually buys

Apple has no dedicated VRAM. Our catalogue stores the machine's full unified memory in the GPU row — 64 GB — but Metal's recommendedMaxWorkingSetSize is what the inference runtimes treat as the hard ceiling, and it is exactly 3/4 on machines of 36 GB and above.

You do not have to take that on trust for this chip, because a primary log exists for this exact machine. In llama.cpp issue #9112 the reporter opens with "System Details - M2 Mac Pro with 64 GB Memory", the Metal init block identifies the device as "ggml_metal_init: found device: Apple M2 Max", and the line that matters reads "ggml_metal_init: recommendedMaxWorkingSetSize = 51539.61 MB". Under ggml's current /1e6 print convention that is 51,539,607,552 bytes — 48.000 GiB, or 0.75 × 64 GiB to six figures. A 64 GB Mac addresses 48 GiB, and the remaining 16 GiB belongs to macOS.

So the quant ladder for this machine is not the vendor's three-row table. It is this:

Why full precision does not land, in full. The base repository's safetensors total 59,553,435,272 B = 55.463 GiB, and that is weights alone; adding the KV cache derived below brings the working set to 57.165 GiB, which overshoots the default pool by 9.165 GiB. The escape hatch on Apple is sudo sysctl iogpu.wired_limit_mb, and it does not rescue this: fitting the weights alone requires iogpu.wired_limit_mb=56795, and the weights plus a full-context KV cache require 58537 — on a machine with 65,536 MiB total, leaving 6.835 GiB for macOS and everything else. The largest raise this project considers safe is 56 GiB (iogpu.wired_limit_mb=57344, keeping 8 GiB of OS headroom), and the full-precision working set is still 1.165 GiB past that. Do not raise the wired limit for this recipe. The install below sits well inside the default pool and a raise would buy nothing.

Two independent facts point the same way. Meta's own Apple runtime never exports full precision: the ExecuTorch artifact repository publishes eight metal directories and every one is a K-quant, topping out at k-quant-dynamic-128K-text-image-dflash-metal at 22.211 GiB — there is no bf16 Metal build to download. And the one M2-generation full-precision report this round turned up is on a far larger machine: in discussion #36, Omnomynous writes "Ran the BF16 variant from unsloth on my M2 Ultra 192GB. Gets about 14tok/s" — a 192 GB part, three times this machine's memory, and enough that 55.463 GiB of weights is not the binding question there. Full precision on M2-generation silicon is real; it is just not a 64 GB proposition.

The memory budget (derived, not measured)

The KV cache is derived from the model's own config.json, not from prose. Its text_config declares num_key_value_heads: 2, head_dim: 128, sliding_window: 2048, and a layer_types array holding 39 sliding_attention and 13 full_attention entries. Per-token KV is 2 (K+V) × 2 KV heads × 128 head dim × 2 bytes = 1024 B per layer — 1 KiB, which is why a 131,072-token context costs under 2 GiB. The first-party GGUF card puts it in prose: "KV cache stays cheap".

The drafter is a separate 5-layer model with its own, wider KV geometry (num_key_value_heads: 8), but all five of its layers are sliding_attention at a 2,048-token window, so it costs 4 KiB per token across a fixed window and never grows with context.

Why min_vram_gb is 64 and not 48. This exact configuration does not fit a 48 GB Mac. That machine addresses 36.000 GiB — the same exact-3/4 rule, corroborated by primary logs on 48 GB parts in lmstudio-bug-tracker #271 and ollama #12012 — and 37.585 GiB overshoots it by 1.585 GiB. Dropping the drafter brings 8-bit alone to 32.785 GiB, which a 48 GB machine does hold; that is a different recipe, and /check/muse-glimmer-30b/m4-max is where the 48 GB tier is documented on its own terms. The floor here is 64 GB because 64 GB is what the documented install needs, and it is a real M2 Max configuration you can buy.

The KV figure above is per sequence. mlx_vlm.server decodes concurrently with no cap by default — its own help text for --max-num-seqs reads "Default: unbounded. Maps to MLX_VLM_MAX_NUM_SEQS." — so N concurrent requests cost N KV caches. The commands below pin it to 1. If you raise it, multiply the 1.701 GiB target cache by the number of slots before deciding it fits.

Installation

1. Install mlx-vlm

mlx-lm does not implement this architecture. mlx-vlm does, with a full muse_glimmer module including a real vision.py, and a native DFlash drafter under mlx_vlm/speculative/drafters/muse_glimmer_assistant/ whose registry maps the drafter's model_type to the dflash round loop automatically.

pip install -U "mlx-vlm>=0.6.13"

That floor is the drafter, not the model. The muse_glimmer_assistant drafter directory is absent at tag v0.6.12 and present from v0.6.13 (released 12 Aug 2026) — verified against a control file in the same directory that resolves at both tags, so the 404 is genuine absence rather than a missing ref. Vision alone works from v0.6.12, which is also the point at which someone else confirms the route in public: in discussion #42, divinetribe — who wrote the still-unmerged mlx-lm text-tower port — notes that "mlx-community's builds load today with mlx-vlm".

2. Download the 8-bit build and the first-party drafter

pip install huggingface_hub

hf download mlx-community/Muse-Glimmer-30B-8bit --local-dir ./muse-glimmer-8bit
hf download meta-models/Muse-Glimmer-30B-assistant --local-dir ./muse-glimmer-drafter

The 8-bit build is clean by the only test that matters for a VLM conversion: its 815 vision tensors resolve through model.safetensors.index.json into the same model-*-of-00153.safetensors shards as the language weights, not into a vendor-specific side artifact that stock loaders never open. Its config.json reports quantization: {group_size: 64, bits: 8, mode: affine} and a text_config.model_type of muse_glimmer_text — which is exactly what the drafter checks for before it will bind. mlx-vlm's own validator refuses anything else with "Muse Glimmer DFlash requires a Muse Glimmer text target, got" and the offending type.

The drafter is Meta's published checkpoint, 5,111,976,608 B of bf16 safetensors, declaring model_type: muse_glimmer_assistant. It shares the target's token embeddings and output head rather than carrying its own, which is why a 5-layer drafter for a 52-layer model costs under 5 GiB.

3. Repair the chat template — this build ships the stale one

This step is not optional and it is easy to miss, because nothing errors. Every mlx-community conversion of this model carries a 7,167-byte chat_template.jinja; the base repository's is 9,992 bytes. The short one is the pre-fix template, and the tell is inside it: its raise_exception string still names a different model family, and it contains the phrase "Reasoning strength" once where the current template contains it seven times.

The cause is a timestamp, not a mistake. The build's card says "This model was converted to MLX format from" the base repository using mlx-vlm 0.6.12, and its last commit is dated 2026-08-10T23:43:12Z. Meta fixed the template in commit a4e59da5 at 2026-08-11T19:23:35Z — about twenty hours later. A quantization is a frozen copy of upstream as of its upload, and this one froze the day before.

What the fix does: it normalises four casings of Reasoning effort to the Reasoning strength the model was trained on, then emits its own reasoning directive only when your system prompt does not already carry one. Under the old template a system prompt that already set a reasoning level got a second, conflicting one appended. Meta describes the same defect for the pre-fix GGUFs and tells you to "re-download if you pulled before this fix".

Neither repository puts a chat_template key in tokenizer_config.json, so the .jinja file is the single source and overwriting it is a complete fix:

curl -L -o ./muse-glimmer-8bit/chat_template.jinja \
  https://huggingface.co/meta-models/Muse-Glimmer-30B/resolve/a4e59da52a7bc87ae7251dd5545c0dd437c44b68/chat_template.jinja

shasum -a 256 ./muse-glimmer-8bit/chat_template.jinja
# expect: cfc67e5f349f37690dfd31ed1f18bc4442a9dd32fe39a648f993cb4eb3cae678
wc -c < ./muse-glimmer-8bit/chat_template.jinja
# expect: 9992

This machine has no CUDA and none of the NVIDIA escape hatches apply: skip pip install flash-attn, and do not reach for bitsandbytes, GPTQ, AWQ, Marlin or ExLlamaV2 — none of them has a Metal kernel. On Apple the attention path is MLX-native, and the quant path is MLX's own quantization or GGUF K-quants.

⚠️ Do not filter the third-party builds by name. The obvious reflex — "NVFP4 and MXFP8 are NVIDIA tensor-core formats, so skip them" — is wrong here, and it is wrong in the direction that costs you options. MLX has adopted the microscaling formats as quantization modes: mlx-community/Muse-Glimmer-30B-mxfp8 and -nvfp4 declare "mode": "mxfp8" and "mode": "nvfp4" in their MLX config.json, carry library_name: mlx, and were produced by mlx-vlm, exactly like the affine 8-bit build this recipe installs. There are also genuine CUDA-only NVFP4 conversions of this model from other publishers, under names that look identical. Read the repository's config.json and library_name, not its title. Meta itself publishes none of these: the meta-models org holds exactly four repositories — the bf16 base, the GGUF pack, the drafter checkpoint and the ExecuTorch exports — and its model card mentions no FP8 or FP4 format at all.

Running

One-shot, with an image:

python -m mlx_vlm.generate \
  --model ./muse-glimmer-8bit \
  --draft-model ./muse-glimmer-drafter \
  --image screenshot.png \
  --system "Reasoning strength: high." \
  --prompt "What does this dialog ask the user to do?" \
  --max-tokens 512 --temperature 1.0

As an OpenAI-compatible server, which is what an agent harness wants:

python -m mlx_vlm.server \
  --model ./muse-glimmer-8bit \
  --draft-model ./muse-glimmer-drafter \
  --max-num-seqs 1 \
  --host 127.0.0.1 --port 8080

Always pass --host 127.0.0.1. The flag's own help text says "Host for the HTTP server (default:0.0.0.0)", and for a model that reads screenshots and calls tools, the default is an unauthenticated endpoint exposed to your whole LAN.

--draft-kind and --draft-block-size exist but you should not need them: the kind is auto-detected from the drafter's model_type, and the block size comes from the drafter's own config at its trained value of 16.

Reasoning depth is a system-prompt setting, not an API parameter. The card is explicit that "Reasoning strength can be defined as part of the system prompt as" a Reasoning strength: <value> line, with levels low / medium / high / xhigh and a default of high. OpenAI's reasoning_effort field does nothing here — which is precisely what step 3's template repair is about, since the fixed template is the thing that reconciles the two spellings.

Alternative: llama.cpp-Metal at Q8_0, with a better projector

If you would rather stay on GGUF, this machine has room for a step nobody else in the catalogue can take: an 8-bit text model and an 8-bit perception encoder. Meta publishes only a Q4_K_M projector; Unsloth publishes Q8_0 and BF16 ones.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DBUILD_SHARED_LIBS=OFF
cmake --build build --config Release -j --target llama-cli llama-mtmd-cli llama-server

hf download unsloth/Muse-Glimmer-30B-GGUF --local-dir Muse-Glimmer-30B-GGUF \
    --include "Muse-Glimmer-30B-Q8_0.gguf" \
    --include "mmproj-Muse-Glimmer-30B-Q8_0.gguf"
hf download meta-models/Muse-Glimmer-30B-GGUF --local-dir Muse-Glimmer-30B-GGUF \
    --include "dflash-Muse-Glimmer-30B-Q4_K_M.gguf"

./build/bin/llama-server \
    -m       Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-Q8_0.gguf \
    --mmproj Muse-Glimmer-30B-GGUF/mmproj-Muse-Glimmer-30B-Q8_0.gguf \
    -md      Muse-Glimmer-30B-GGUF/dflash-Muse-Glimmer-30B-Q4_K_M.gguf -ngld 99 \
    --spec-type draft-dflash \
    -a muse-glimmer-30B \
    -ngl 99 -c 131072 -np 1 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64

Metal is on by default on macOS, so there is no backend flag to pass. Mainline llama.cpp registers LLM_ARCH_MUSE_GLIMMER and a dedicated PROJECTOR_TYPE_MUSE_GLIMMER with its own graph builder, so vision here is genuinely implemented rather than implied by the presence of an mmproj file; image input goes through llama-mtmd-cli. Speculative decoding is a first-class type, COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH, which reads the trained block size from the dflash.block_size metadata key.

That stack totals 32.710 GiB — 27.579 (text) + 1.911 (projector) + 1.519 (drafter) + 1.701 (KV) — leaving 15.290 GiB spare. If you disable the sliding-window crop with --swa-full the KV rises to 6.500 GiB and the total to 37.509 GiB, which still fits.

⚠️ --spec-type without a draft model silently does nothing — and the reverse was true too, until 2026-08-13. common_speculative_init adds the DFlash implementation only when the type bit is set and a draft context exists, so --spec-type with no -md is skipped before it can assert. The other half has an expiry date: common_params_speculative::types used to default to { COMMON_SPECULATIVE_TYPE_NONE } with nothing writing to it, so -md alone loaded the drafter and never used it — until PR #26814 (merged 2026-08-13) made common/arg.cpp infer the type from the draft GGUF's general.architecture whenever none was requested. Meta's drafter declares dflash and carries no markov_w1.weight, so on a current build -md alone binds it. Passing --spec-type draft-dflash anyway is harmless and still right on an older binary. Either way there is no error and no warning if it fails to bind — look for adding speculative implementation 'draft-dflash' in the startup log, or for a non-zero acceptance rate.

⚠️ The old GGUF filenames are gone. Meta republished the GGUFs on 12 Aug 2026 under canonical Q4_K names with a corrected embedded chat template, then deleted the superseded muse-glimmer-30B-kquant-17gb.gguf and dflash-kquant.gguf on 18 Aug — those URLs now return 404. Unsloth still mirrors the old dflash-kquant.gguf, and it is the pre-fix object, not the current one: 1,631,205,312 B at 27d9a805… against Meta's 1,631,208,128 B at b2e808bf… — 2,816 bytes of corrected chat template apart. It is bit-identical to the draft layer Ollama serves, which is the same superseded object. Pull the drafter from Meta.

Alternative: Ollama, for a one-liner

ollama run muse-glimmer:30b-mxfp8-dflash

The default muse-glimmer:30b is Meta's K-Quant-17GB build repackaged — its projector layer is bit-identical to Meta's mmproj-Muse-Glimmer-30B-Q4_K_M.gguf, same 1,400,328,928 bytes and the same sha256:f48b452316f9… digest — and on this machine that leaves most of the pool idle. Layer totals read from the registry manifests on 2026-08-21:

ℹ️ -mxfp8 and -nvfp4 are MLX quantization modes here, not NVIDIA ones — do not skip them on reflex. The names come from the microscaling formats, but the mlx-community builds that back these tags declare "mode": "mxfp8" and "mode": "nvfp4" in their MLX config.json and were converted with mlx-vlm. Two consequences worth knowing. 30b-mxfp8 is an 8-bit build in the same fidelity class as the affine 8-bit this recipe installs (30.271 GiB of weights against 31.084). And muse-glimmer:30b-mlx is the same image as muse-glimmer:30b-nvfp4-dflash — pull both manifests and their 1,503 layers are digest-for-digest identical — so the friendly-looking mlx tag is a 4-bit NVFP4 build with the drafter bundled, not a generic "MLX version". One of its small blobs is the drafter's own config.json, declaring model_type: muse_glimmer_assistant, which is how you can tell without downloading 21 GB.

The step-3 template repair does not transfer to this route, and cannot be checked here either. Every tag's config blob declares "renderer":"glimmer","parser":"glimmer" — GGUF and MLX alike — so Ollama renders prompts with its own built-in code path rather than with whatever Jinja the artifact carries. That may well be correct; the point is that you cannot inspect it, patch it, or diff it against Meta's 9,992-byte template the way you can on the MLX path above. If the reasoning-line behaviour matters to you, run the MLX path and check the bytes yourself.

Alternative: ExecuTorch Metal — the runtime Meta actually measured

Meta's published Apple tok/s figures were produced on ExecuTorch, not on MLX-LM and not on llama.cpp: the card's footnote reads "M4/M5 measurements were done using ExecuTorch, and RTX using llama.cpp." If you want the configuration those numbers describe, the pre-exported Metal artifacts run here — the MLX delegate's own documentation puts the floor at Apple Silicon M1 or later, and the build gate is a macOS deployment target, whose CMake check is commented "Validate deployment target - MLX requires macOS 14.0+ / iOS 17.0+". An M2 Max on Sonoma or newer is inside both.

Three practical notes before you start. First, this path is a build, not a download: get an ExecuTorch checkout configured per the build-from-source guide, because the cmake --workflow line below runs from inside it. Second, the delegate needs the Metal compiler, "which ships with Xcode (not the standalone Command Line Tools)" — the Command Line Tools alone are not enough, which is a build blocker rather than a nuisance. Check with xcrun -sdk macosx --find metal; if it fails while Xcode is installed, point the active developer directory at it with sudo xcode-select -s /Applications/Xcode.app/Contents/Developer. Third, the artifact repository is 372 GB in total, so --include is mandatory. On this machine take the top of the Metal ladder, k-quant-dynamic-128K-text-image-dflash-metal at 22.211 GiB, rather than the 24 GB-targeted k-quant-17G build; there is no CPU fallback, the card states "There is no CPU variant."

REPO=meta-models/Muse-Glimmer-30B-ExecuTorch-PTE
VARIANT=muse-glimmer-k-quant-dynamic-128K-text-image-dflash-metal

hf download "$REPO" --local-dir ./muse-glimmer-pte \
  --include "$VARIANT/"'*.pte' \
  --include "$VARIANT/pos_embed.bin" \
  --include tokenizer.json \
  --include tokenizer_config.json \
  --include chat_template.jinja

ls -lh "./muse-glimmer-pte/$VARIANT"   # expect exactly the .pte and pos_embed.bin

(cd examples/models/muse-glimmer && cmake --workflow --preset muse-glimmer-mlx)
pip install -r examples/llm_server/python/requirements.txt

python -m executorch.examples.models.muse_glimmer.serving.serve \
  --model-path "./muse-glimmer-pte/$VARIANT/$VARIANT.pte" \
  --pos-embed-path "./muse-glimmer-pte/$VARIANT/pos_embed.bin" \
  --tokenizer-path "./muse-glimmer-pte/tokenizer.json" \
  --hf-tokenizer "./muse-glimmer-pte" \
  --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
  --model-id muse-glimmer-30B \
  --tool-parser atem \
  --host 127.0.0.1 --port 8000

The Metal build is self-contained — no --data-path — and dflash needs no flag, because --artifact-mode defaults to auto and detects the exported method contract. --tool-parser atem is what turns Muse Glimmer's native tool output into OpenAI tool_calls; the default is none. The ExecuTorch repository's own copy of chat_template.jinja was stale until 15 Aug 2026 and is now byte-identical to the base repository's, so a fresh pull is fine and an older one is not — the same 9,992-byte, cfc67e5f349f… file as in step 3.

Results

• Speed: omitted. No first-party or community tok/s figure for an M2 Max on this model surfaced this round — searched across all 96 discussion threads on the five canonical repositories (each fetched individually through the HF API rather than sampled), plus English and Chinese web search, YouTube, Reddit and GitHub. The same searches did return chip-named Apple measurements for other parts, so the method works and the gap is real. Please contribute a measurement so /check/muse-glimmer-30b/m2-max stops being empty.
• What the vendor measured, and on what: Meta's model card reports 23.7 → 37.8 tok/s on an Apple M4 Max and 26.6 → 50.2 on an M5 Max, batch size 1, greedy decoding, K-Quant-17GB — all on ExecuTorch. Those are newer chips at higher memory bandwidth, on a different runtime and a lighter quant than this recipe installs, so none of the three numbers forwards to this machine.
• One caveat about the drafter on Metal, worth reading before you count on it: the only measured Apple llama.cpp run this round found the DFlash drafter delivering nothing. CogniTechSystems, documenting a MacBook Pro M4 Max 36 GB — "the hardware Meta benchmarked" — records speculative decoding at "0.9x to 1.0x measured, bursts to ~20 tok/s in code blocks" against Meta's 1.5×, on a correctly-flagged --spec-type draft-dflash server. That is a different chip, a different runtime from the one this recipe leads with, and a machine with far less headroom; treat it as a reason to verify the acceptance rate on your own box rather than as a verdict on the drafter. Nobody has published a Metal measurement of mlx-vlm's DFlash path in either direction.
• Unified memory usage: 37.585 GiB working set for the 8-bit multimodal-plus-drafter stack at the full 131,072-token context, against a 48.000 GiB addressable pool. Derived from published artifact bytes and the architecture's own config.json, not observed.
• Quality notes: Meta puts K-Quant-Dynamic at 0.2% average degradation across 15 benchmarks and K-Quant-17GB at 1.0%, and describes both as compressions that fit "within a 24 GB or 32 GB envelope". The 8-bit MLX build sits above both of them on the fidelity ladder and below full precision; the vendor publishes no degradation figure for it, so treat "better than 0.2%" as the shape of the claim rather than a number.

For the full benchmark data, see /check/muse-glimmer-30b/m2-max.

Troubleshooting

The model answers, but ignores or double-sets the reasoning level

You skipped step 3, or you pulled the weights before repairing the template. Check it:

wc -c < ./muse-glimmer-8bit/chat_template.jinja   # 7167 = stale, 9992 = fixed

Nothing errors in this state — the model runs and produces plausible output, with a second, conflicting Reasoning strength: line silently appended to a system prompt that already set one.

Muse Glimmer DFlash requires a Muse Glimmer text target

The drafter refused to bind. mlx-vlm validates the pairing on architecture fields rather than repository names, so quantized conversions and local checkpoints are accepted — but the target's language config must report model_type: muse_glimmer_text, 52 layers, hidden size 6656 and vocabulary 202,048, and the drafter's weight keys must match Meta's published checkpoint exactly. Two ways to trip it: pointing --draft-model at a non-Muse drafter, or pointing --model at a text-only conversion of some other family.

unknown model architecture: 'muse-glimmer'

Scope: the llama.cpp alternative. Your build predates the architecture. Use b10353 or newer, or confirm a source checkout with grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp. In LM Studio the same failure surfaces as an unhelpful exited before becoming healthy, with the real message only in ~/.lmstudio/server-logs/; a community reporter documents the fix in discussion #42 — update the llama.cpp runtime to llama.cpp-mac-arm64-apple-metal-advsimd 2.28.2 or newer via lms runtime update.

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

Scope: the llama.cpp alternative only. The MLX path binds the drafter in-process and never reaches this code; the ExecuTorch path carries the drafter inside the exported artifact.

The message is an out-of-memory condition wearing a bounds-check error's clothes. Issue #26894 reads as though it were a file problem — it was filed against Meta's own GGUF and blamed one metadata key, muse-glimmer.attention.sliding_window_pattern, which Meta's builds encode as an array where third-party conversions write a scalar. The reporter withdrew that diagnosis on 13 August 2026: "My original diagnosis was wrong", and "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the file the issue names, checked its SHA-256 against the Hub, bound the drafter 10 times out of 10, and concluded the published file "is fine as published". His own summary of the original mistake is "a timing coincidence that I mistook for causation" — he had tested the rewritten file later, on a machine that happened to have memory free.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum.
• The zero guard there catches only free == 0 and total == 0 — a device with nothing at all to report (#18577). A device that is merely full reports free == 0 against a real total and falls straight through it.
• Every splits entry is then 0, so split_sum == 0 and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs matches nothing (x < NaN is false), returns end(), and devices.at(n_devices()) throws.

The reporter reproduced it with an LD_PRELOAD shim forcing free = 0, and naturally by saturating the GPU from another process — including on an unrelated dense model with no speculative decoding at all: "So it is not specific to Muse Glimmer or to DFlash."

What to do on this machine. Unified memory makes this less likely than on a discrete card, but not impossible: Metal's working-set limit is a fraction of installed RAM, not all of it. Quit whatever else is holding the GPU, check with sudo powermetrics --samplers gpu_power or Activity Monitor, and re-run. If you still cannot fit, lower -c or drop the drafter.

Do not rewrite the GGUF's metadata. The scalar sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and leaves you running a file whose checksum no longer matches the Hub. The issue remains open, now as a request for a clear insufficient-memory message in place of the out_of_range that sent the investigation down the wrong path.

Should I use the DFlash 2 drafter?

Not yet, and not on this machine. Two community repositories appeared on 18–19 August 2026 — incoai/Muse-Glimmer-30B-DFlash2 and its GGUF mirror z-lab/Muse-Glimmer-30B-DFlash2-GGUF — reporting meaningfully better acceptance lengths than Meta's drafter. The reasons to wait are structural, not about quality:

• Its llama.cpp support is an unmerged pull request, #27342 ("spec : add DFlash2 support (local convolution + candidate selector)"), opened 18 August and still open. Mainline registers dflash and does not register a DFlash 2 architecture; using it means building a checkout of someone's branch.
• Its own quickstart targets SGLang and a vLLM pull-request branch, and its evaluation was run on an NVIDIA H200 with FlashAttention 3. Neither runtime has a GPU path on Apple Silicon.
• On the MLX path it is not loadable at all. mlx-vlm keys the drafter class on the checkpoint's model_type; Meta's drafter declares muse_glimmer_assistant, and DFlash 2 declares qwen3 with a DFlash2DraftModel architecture and two-tap dynamic convolutions plus a candidate selector that no registered drafter class implements.

The right time to revisit is when #27342 merges and a release carries it. Until then Meta's Muse-Glimmer-30B-assistant is the drafter both runtimes on this page actually support. As the DFlash 2 card itself puts it, "It is not a standalone language model: it runs inside a speculative decoding server and drafts tokens for the target model to verify." — the server has to know how.

Nothing comes back on a tool call

Never treat <|eom|> as a stop token. The stop tokens are <|end_of_text|> (200001) and <|eot|> (200008); <|eom|> ends a single message and the turn continues past it. On a tool call the model emits a private reasoning message closed by <|eom|> first, so stopping there means the tool call is never generated. The bundled runtimes handle this; it bites only if you drive a raw runner from your own client.

Report anything else via the submission form.