Muse Glimmer 30B on RTX 3090: vision + DFlash speculation at full 131K context

What You'll Build

A local agent endpoint on a single RTX 3090: llama-server running Meta's Muse Glimmer 30B with image input and DFlash speculative decoding, at the model's full 131,072-token context — not a reduced one. Every component is resident on the card; nothing offloads to system RAM.

Hardware data: RTX 3090 (24 GB VRAM) · derived resident set 20.198 GiB at 131,072 context, 3.802 GiB spare · See benchmark data

ℹ️ A retracted bug report you will still find in search results. llama.cpp issue #26894 originally blamed Meta's own GGUF for a crash when binding the DFlash drafter. Its reporter withdrew that diagnosis on 2026-08-13 — the crash is llama.cpp's response to a card with no free VRAM, not to anything in the file. If you meet vector::_M_range_check, read Troubleshooting: it is an out-of-memory message in disguise.

ℹ️ Multimodal means text + images, not audio. Muse Glimmer accepts interleaved text and images through a ViT-G/14 perception encoder, and emits text only. The model card is explicit: "Audio input/output is not supported." Video is not a supported input either — the card notes the model is not explicitly optimized for it and processes it as individual frames.

Requirements

Ampere has no FP4 or FP8 tensor path, so the NVFP4 and MXFP8 conversions on the Hub are not options here — and note they are third-party work, not Meta's: the meta-models org publishes exactly four repos (the bf16 base, the GGUF set, the drafter source and the ExecuTorch export), and the model card never mentions FP8 or FP4 in any form. On an RTX 3090 the route is the K-quant GGUF stack below.

Installation

1. Get a llama.cpp build that knows the architecture

Muse Glimmer support landed in PR #26841, merged 2026-08-10 as commit 62bf73d2. Release b10353 is the first tag that contains it — b10344 and earlier sit five commits behind the merge and fail with unknown model architecture: 'muse-glimmer'.

⚠️ b10353 is the floor for loading the model, not for driving it as an agent. PR #26879, "Muse Glimmer: fix detection of tool calls after EOM" merged 2026-08-11T20:15:21Z; b10358 was published 2026-08-11T09:04:42Z, eleven hours earlier, and b10353 earlier still. Both load the model perfectly and both mis-parse a tool call that follows an end-of-message marker — which on an agentic model is the whole point of the thing. That PR also added llama.cpp's own bundled models/templates/muse-glimmer.jinja. Take a build dated 2026-08-12 or later; this recipe pins today's release.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10549
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

CMAKE_CUDA_ARCHITECTURES=86 is the RTX 3090's compute capability (Ampere, sm_86). Prebuilt release binaries from the same tag work equally well if you prefer not to compile.

2. Download the three GGUF files

hf download meta-models/Muse-Glimmer-30B-GGUF \
  Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --local-dir ./muse-glimmer

Three files, three jobs: the 16.76 GB quantised language model, the 1.40 GB vision projector, and the 1.63 GB DFlash drafter. Meta's model card describes the K-Quant-17GB build as compressing weights to roughly 4-bit, "shrinking the language model to under 20 GB", and states that this "leaves enough headroom for the model's KV cache, the perception encoder for image understanding, and the speculative decoding drafter to run simultaneously within a 24 GB or 32 GB envelope." The budget under Results checks that claim against the actual bytes.

⚠️ Use the canonical Q4_K_M names above, not the older *-kquant*.gguf ones. Meta republished these builds on 2026-08-12 with a corrected embedded chat template; the previous template could leave a conflicting reasoning-strength line in the prompt, and the GGUF card says to "re-download if you pulled before this fix". Meta removed the superseded copies on 2026-08-18, so a stale filename now fails outright rather than quietly fetching the old build — but if you pulled between 10 and 12 August you still have it on disk. The projector is the exception — mmproj-Muse-Glimmer-30B-Q4_K_M.gguf is the same object as the old mmproj-kquant.gguf (identical SHA-256 f48b4523…), so there is nothing to re-download there.

Running

./build/bin/llama-server \
  -m ./muse-glimmer/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --mmproj ./muse-glimmer/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -ngl 99 -c 131072 -np 1 \
  --jinja \
  --host 127.0.0.1 --port 8080

This is the shape Meta publishes in its own llama.cpp deployment doc, which also names b10353 as the minimum release. --jinja is not optional: the model ships an ATEM tool-calling chat template, and without --jinja tool calls and reasoning separation break. -np 1 matters for the memory budget — each extra server slot multiplies the sliding-window KV ring.

To add speculative decoding, append the drafter:

  -md ./muse-glimmer/dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --spec-type draft-dflash -ngld 99 --spec-draft-n-max 4

The drafter adds 1.568 GiB to the resident set — its weights plus its own KV — which the budget below accounts for. If the card is already close to full, read Troubleshooting first: llama.cpp reports that condition as a bounds-check error rather than as an out-of-memory one.

Meta's recommended sampling is --temp 1.0 --top-k 64 --top-p 0.95. Reasoning strength is set in the system prompt as Reasoning strength: <low|medium|high|xhigh>, or through --chat-template-kwargs '{"reasoning_strength":"low"}'.

Ollama, if you would rather not build anything

ollama run muse-glimmer:30b pulls the same build — though not, for one of the two layers, the same file. Its registry manifest lists layers of 16,756,681,056 and 1,400,328,928 bytes, and the distinction matters:

• The projector layer is bit-identical to Meta's projector — same SHA-256, f48b4523…, which is both mmproj-kquant.gguf and its renamed twin.
• The model layer matches muse-glimmer-30B-kquant-17gb.gguf to the byte in size, but its content hash differs (71b5c9c9… against Meta's 7e9b74b7…). It is the same K-Quant-17GB build, re-packaged, not the same bytes.

Ollama labels the tag q4_K_M and displays 18 GB, but this is the 24 GB-target build — not the 32 GB-target K-Quant-Dynamic one.

One caveat, re-checked 2026-08-21: Ollama's blobs are unchanged and still correspond to the pre-fix files. The template defect does not reach you on this route, though, whatever the blobs' age. Every muse-glimmer tag's config blob — GGUF and MLX alike — declares "renderer": "glimmer" and "parser": "glimmer", and in server/prompt.go renderPrompt returns from the named renderer before it can reach m.Template.Execute, so the artifact's embedded Jinja is never consulted. The Go renderer carries both of the fixes the corrected template gained: model/renderers/glimmer.go maps all four casings of Reasoning effort to Reasoning strength, then appends its own directive only if defaultSystem || !glimmerHasSystemReasoning(content) — the Go equivalent of the template's {%- if 'reasoning strength' not in (sys_text | lower) -%}. Ollama's own reference test pins that renderer against a fixture its comment records as copied byte-for-byte from the chat template at publisher revision a4e59da5…, SHA-256 cfc67e5f349f… — the corrected one. Scope: this settles prompt formatting only. The weights behind these tags are still the superseded objects, and nothing here says what parser: glimmer does on the output side.

For the drafter as well, pull muse-glimmer:30b-q4_K_M-dflash: the same two layers plus a third of exactly 1,631,205,312 bytes at digest 27d9a805…, which is bit-identical to Meta's dflash-kquant.gguf.

Results

• Speed: no first-party RTX 3090 figure exists. Meta's speed table publishes an RTX 5090, an Apple M4 Max and an M5 Max and nothing else — the string "3090" does not appear anywhere in the model card. Community reports on a single 3090 are collected below; they are single-source each and disagree, so treat them as orientation, not as a benchmark. If you measure this pair, please contribute the numbers.
• VRAM usage: 20.198 GiB derived resident set at full context — see the budget below and /check/muse-glimmer-30b/rtx-3090.
• Quality notes: the model card rates the K-Quant-17GB build at 1.0% average degradation across 15 benchmarks, against 0.2% for the 32 GB-target K-Quant-Dynamic build.

The 24 GB budget

The interesting question on a 24 GB Ampere card is not whether 16.76 GB of weights fit — it is whether they still fit once the vision projector, the drafter and a 131,072-token KV cache sit beside them. They do, with room to spare, and the reason is architectural.

Muse Glimmer uses grouped-query attention with 32 query heads against 2 KV heads and a 128-wide head, so one token costs just 1,024 bytes of f16 KV per layer. On top of that, its attention pattern is [Local, Local, Local, Global] repeating — which the GGUF encodes as a 52-entry sliding_window_pattern of 39 sliding-window layers and 13 full-attention layers. llama.cpp allocates these as two separate caches (build_attn_inp_kv_iswa): only the 13 full-attention layers scale with context, while the 39 sliding-window layers are pinned to a small ring of sliding_window + n_ubatch cells — 2,560 at the default -ub 512. So 75% of the layers cost nothing as context grows.

That leaves 3.802 GiB of the card's 24 GiB. Two things live in that margin and are not in the table, because we did not measure them: llama.cpp's CUDA compute buffer, which scales with -ub, and the CUDA context plus whatever your desktop is holding. Budget accordingly on a machine driving a monitor.

One cross-check says the arithmetic is sound: running the same derivation on the K-Quant-Dynamic build gives 22.897 GiB — only 1.103 GiB clear, which is why Meta targets that build at 32 GB rather than 24 GB. Our numbers reproduce the vendor's own two-tier split without being fitted to it.

Community RTX 3090 reports

None of these are ours, and none are first-party. Each names its own configuration, and the configurations differ more than the numbers do.

The closest match to this recipe is the first: "Just tried the gguf 17gb k-quant with llama.cpp" — the same file this recipe installs — reporting that "when using the dflash drafter, it does more than 60 tok/sec on my 3090" and that "vision (with the additional mmproj model) seems to work really well". That is one community datapoint from one person, not a measurement, and the same commenter was unimpressed with the model's output quality.

For the full benchmark data, see /check/muse-glimmer-30b/rtx-3090.

Troubleshooting

unknown model architecture: 'muse-glimmer'

Your llama.cpp predates the architecture. Release b10344 and earlier are five commits behind the #26841 merge; upgrade to b10353 or newer. Users on the official GGUF repo hit this on b10344 and resolved it by moving to b10352 and b10358.

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

This is an out-of-memory condition wearing a bounds-check error's clothes. It is not specific to Muse Glimmer, to the DFlash drafter, or to any GGUF — it is what llama.cpp prints when every visible device reports zero free VRAM at the moment a model is loaded.

That is not how issue #26894 reads at first. It was filed against Meta's own GGUF and blamed the way that file encodes muse-glimmer.attention.sliding_window_pattern — an array of 52 booleans, where third-party conversions write a scalar. The reporter withdrew that diagnosis on 2026-08-13 — "My original diagnosis was wrong" — and named the real mechanism instead: "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the exact file the issue was filed against, checked its SHA-256 against the Hub, and bound the drafter successfully 10 times out of 10, concluding that the published file is fine as it stands. The array-versus-scalar difference was a coincidence: he had tested the rewritten file later, on a card that happened to be free.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum of those weights.
• There is a zero guard, but it only catches a device reporting free == 0 and total == 0 — a device with nothing at all to report (#18577). A real GPU that is merely full reports free == 0 against a real total, and falls straight through it.
• Every entry in splits is then 0, so split_sum == 0, and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs finds no element (x < NaN is false), returns end(), and devices.at(n_devices()) throws. With one visible device that is __n (which is 1) >= this->size() (which is 1), verbatim.

It surfaced on the draft model in the original report for an ordinary reason: the ~20 GB target loads first and takes the card with it, so the drafter is simply the next allocation to ask a full GPU for room. The reporter reproduced it two ways — an LD_PRELOAD shim forcing free = 0, and naturally, by saturating the GPU from a separate process — including on an unrelated dense model with no speculative decoding at all.

What to do. Free the card, then re-run. Close other CUDA processes and check with nvidia-smi. On 24 GB this recipe's resident set is 20.198 GiB, leaving 3.802 GiB for the compute buffers, the CUDA context and your desktop — a compositor holding a gigabyte is enough to matter, so running headless or on the integrated display output is the durable fix. If you cannot free enough, lower -c, or drop -md … --spec-type draft-dflash -ngld 99 and recover 1.568 GiB at the cost of the speculative speedup.

Do not rewrite the GGUF's metadata. The scalar-sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and it leaves you running a file whose checksum no longer matches the Hub.

The configuration itself is well attested. Every public report that actually loads a drafter file:

A sixth report, dmpr, runs K-Quant-Dynamic at 131,072 context on build 10352 and calls it working — but that thread is about the model failing to load, and the command as posted names no draft model file. llama.cpp only auto-discovers a dflash- sidecar for -hf downloads, so with a local --model path and no -md there is no draft context to bind. Read it as evidence that the architecture loads at 10352, not that the drafter binds there.

One genuinely separate change did land: PR #26900, merged 2026-08-12, swaps get_key_or_arr for get_arr in the drafter's own hparams read in src/models/dflash.cpp. Its author struck through the line claiming it closed #26894 — correctly, as it turns out. The issue remains open at the time of writing, now as a request for a clear "insufficient device memory" message in place of the out_of_range that sent the original investigation down the wrong path.

VRAM climbs after the first image, and prefill gets slower

Community issue #26873 reports that the first use of the vision projector costs a further 1.1 GB beyond the mmproj weights already accounted for, and that prompt-processing throughput then falls from 3,062 to 1,875 tokens per second in the reporter's own logs — and stays down even on a fresh context.

The prefill half of that has since been narrowed, and the narrowing lets a single-card reader off the hook. A second reporter pinned it down on 2026-08-15: "the regression only happens when pipeline parallelism is enabled", and the original reporter then confirmed on his own machine that forcing a no-op tensor override makes the slowdown disappear — self-confirmation rather than an independent third datapoint, and the thread has only those two participants. Both are consistent with llama.cpp's own gate, which is worth reading rather than trusting: in src/llama-context.cpp, pipeline_parallel is set only when model.n_devices() > 1 && … && !model.has_tensor_overrides(). On one GPU it can never be true — which is also why a tensor override "fixes" it on two. Every report in that thread is a multi-GPU configuration.

The extra allocation on first image use has not been attributed to the same cause, so keep budgeting for it. It eats into the 3.802 GiB margin above: if you serve images and run close to the edge, quantise the KV cache or lower -c.

DSpark is not DFlash, and it can be slower

Meta ships a DFlash drafter; llama.cpp also supports a DSpark type, which adds a Markov head and outranks DFlash in llama.cpp's sidecar auto-detection when both are present. They are not interchangeable. On a single RTX 3090 bartowski measured his own Q4_K_M build at "On my 3090 with Q4_K_M I get around 38 tok/s, with dspark it drops to ~28-32 depending on the task". Pass --spec-type draft-dflash explicitly rather than relying on auto-detection.

-hf picks the wrong file

llama.cpp's repo auto-selection looks for Q4_K_M or Q8_0 in the filename, then falls back to the first model-shaped GGUF in the repo. Since the 2026-08-12 republish the canonical name carries Q4_K_M, so a bare -hf meta-models/Muse-Glimmer-30B-GGUF now resolves to Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf by a real tag match rather than by ordering. Meta deleted the superseded *-kquant*.gguf files on 2026-08-18, so a stale pinned name now 404s instead of resolving to the old build. Pin the canonical one: --hf-file Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf.