Muse Glimmer 30B on RTX 4090: vision and DFlash in llama.cpp at full 131K context

What You'll Build

A local agent endpoint on a single RTX 4090: llama-server running Meta's Muse Glimmer 30B with image input and DFlash speculative decoding, at the model's full 131,072-token context. Every component is resident on the card; nothing offloads to system RAM.

Hardware data: RTX 4090 (24 GB VRAM, Ada Lovelace, sm_89) · derived resident set 20.198 GiB at 131,072 context, 3.802 GiB spare · See benchmark data

ℹ️ A retracted bug report you will still find in search results. llama.cpp issue #26894 originally blamed Meta's own GGUF for a crash when binding the DFlash drafter. Its reporter withdrew that diagnosis on 2026-08-13 — the crash is llama.cpp's response to a card with no free VRAM, not to anything in the file. The issue's title still states the retracted version, which is why the story keeps circulating. If you meet vector::_M_range_check, read Troubleshooting: it is an out-of-memory message in disguise.

ℹ️ Multimodal means text + images, not audio. Muse Glimmer accepts interleaved text and images through a perception encoder and emits text only. The model card is explicit: "Audio input/output is not supported." Video is accepted but not optimised for — the card notes the model is not explicitly optimized for it and processes it as individual frames.

Requirements

What Ada buys you here, and what it does not

The RTX 4090 is Ada Lovelace, not Ampere, so it does have FP8 (E4M3/E5M2) tensor cores — and on this model that changes nothing about which files you download. Three separate reasons, each worth checking rather than assuming:

• Meta ships no FP8 or FP4 build. The meta-models organisation publishes exactly four Muse Glimmer repos — the bf16 base, the GGUF repo, the drafter's unquantised source, and the ExecuTorch PTE exports. There is no first-party FP8 or NVFP4 artifact to run.
• The community FP8 build does not fit. Red Hat's RedHatAI/Muse-Glimmer-30B-FP8-block totals 34.39 GB of safetensors — 32.03 GiB, over the card before a single KV byte. FP8 compute being native on Ada is not the same claim as the FP8 weights fitting.
• The one build that would fit by size is Blackwell-only. RedHatAI/Muse-Glimmer-30B-NVFP4 is 23.38 GB (21.78 GiB), which would clear 24 GB — but NVFP4 tensor cores arrived with Blackwell and Ada has none, and the one attempt to route that checkpoint through llama.cpp instead is open bug #27178, where the converted GGUF fails to load and then emits a single repeated token.

So on an RTX 4090 the route is the K-quant GGUF stack below, exactly as it is on the 24 GB Ampere card — reached by different reasoning, not by inheritance.

Installation

1. Get a llama.cpp build that knows the architecture

Muse Glimmer support landed in PR #26841, merged 2026-08-10 as commit 62bf73d2. Release b10353 is the first tag that contains it; b10344 and earlier fail with unknown model architecture: 'muse-glimmer'. Meta's GGUF card names the same floor.

b10353 is the floor, not the recommendation. Two changes landed after it that matter here, and the command below therefore checks out b10549, the current release at the time of writing: PR #26879, which fixes detection of tool calls after an end-of-message marker, merged 2026-08-11 at 20:15 UTC — eleven hours after b10358 was cut, so the tags people were using that week all mis-parse that case; and PR #26814, merged 2026-08-13, which teaches llama.cpp to infer the speculative type from the drafter's metadata (see Running). For an agentic workload the first of those is not optional.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10549
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

CMAKE_CUDA_ARCHITECTURES=89 is the RTX 4090's compute capability (Ada, AD102, sm_89) — the single line that differs from the same recipe on an Ampere card. Prebuilt release binaries from the same tag work equally well if you prefer not to compile.

If you are on a checkout rather than a tag, confirm all three halves of this model are present before you download 20 GB of weights:

grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp          # text model
grep -c PROJECTOR_TYPE_MUSE_GLIMMER tools/mtmd/clip-impl.h # vision
grep -c COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH common/speculative.cpp  # drafter

Non-zero on all three means the architecture, the perception encoder and the DFlash speculator are all in your tree. On llama.cpp master at 17197474 they are.

2. Download the three GGUF files

hf download meta-models/Muse-Glimmer-30B-GGUF \
  Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --local-dir ./muse-glimmer

Three files, three jobs: the 16.76 GB quantised language model, the 1.40 GB vision projector, and the 1.63 GB DFlash drafter. Meta's model card describes the K-Quant-17GB build as compressing weights to roughly 4-bit, "shrinking the language model to under 20 GB", and says this "leaves enough headroom for the model's KV cache, the perception encoder for image understanding, and the speculative decoding drafter to run simultaneously within a 24 GB or 32 GB envelope." The GGUF card marks this build as the one to start with and states that it fits 24 GB of VRAM. The budget under Results checks both claims against the actual bytes.

⚠️ Use the canonical Q4_K_M names above, not the older *-kquant*.gguf ones. Meta republished these builds on 2026-08-12 with a corrected embedded chat template; the previous template could leave a conflicting reasoning-strength line in the prompt, and the GGUF card says to "re-download if you pulled before this fix". Meta removed the superseded copies on 2026-08-18, so a stale filename now fails outright rather than quietly fetching the old build. The projector is the exception — mmproj-Muse-Glimmer-30B-Q4_K_M.gguf is the same object as the old mmproj-kquant.gguf (identical SHA-256 f48b4523…), so there is nothing to re-download there.

Running

./build/bin/llama-server \
  -m ./muse-glimmer/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --mmproj ./muse-glimmer/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -ngl 99 -c 131072 -np 1 \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --host 127.0.0.1 --port 8080

--jinja is not optional: the model ships a tool-calling chat template, and without it tool calls and reasoning separation break. The sampling triple is Meta's own recommendation.

-np 1 is load-bearing, and the vendor's own example uses -np 4. The GGUF card states plainly that llama-server divides -c across the slots, so a single request gets -c divided by -np — meaning the vendor's example gives one request 32,768 tokens, not 131,072. A single slot is also what the budget below assumes, and it sidesteps open issue #27117, in which DFlash draft acceptance degrades as concurrent sequences rise (that report was traced on an AMD APU, not on this card, so treat the concurrency threshold as unverified here — the -c-division reason stands on its own).

To add speculative decoding, append the drafter:

  -md ./muse-glimmer/dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --spec-type draft-dflash -ngld 99 --spec-draft-n-max 15

Whether --spec-type draft-dflash is strictly required depends on how new your build is, and the answer changed on 2026-08-13. Since PR #26814, merged that day, llama.cpp infers the speculative type from the draft GGUF's own general.architecture: a dflash-arch draft with no Markov head resolves to draft-dflash, and the startup log says auto-detected speculative type … from the draft model metadata. On such a build -md alone is enough. On anything older there was no inference at all, and the outcome was worse than a downgrade: the type stayed at { COMMON_SPECULATIVE_TYPE_NONE }, and since common_get_enabled_speculative_configs builds its mask as 1u << type, {NONE} yields the bitmask 1 while every real implementation's bit is 2 or higher — so no speculator was added at all and the 1.5 GiB of drafter sat in memory unused. That is not a hypothetical: a reader on b10354 running an RTX 5090 in discussion #34 hit it and called --spec-type draft-dflash "this was the key missing flag" — his own diagnosis in that thread was that the server had fallen back to draft-simple, which the source does not support; the bit for DRAFT_SIMPLE is not in the mask either. Every community configuration this page cites predates the change — common_speculative_types_from_gguf appears zero times in common/arg.cpp at both b10353 and b10358, and once in the release this recipe installs. Pass the flag either way. It costs nothing, it is correct on both sides of that date, and it is what pins the type when a repo sidecar is in play, where DSpark outranks DFlash — see Troubleshooting.

The drafter adds 1.568 GiB to the resident set (its weights plus its own KV), which the budget below accounts for.

15 is the ceiling, not a tuning choice. The drafter's own config declares block_size: 16, and llama.cpp denoises that block in place and yields at most block_size - 1 of it; ask for more and it clamps with exceeds the trained block size 16 -- clamping to 15. Values of 3 or 4 circulate widely — Ollama's own dflash tag ships draft_num_predict: 3, and Meta's CUDA ExecuTorch export caps at 3 — and are worth trying if acceptance on your prompts is poor, since a rejected long draft costs verification for nothing. There is no measurement on this card either way; start at the ceiling and lower it if the startup log's acceptance rate disappoints.

Reasoning strength is a template variable, not a prompt line: --chat-template-kwargs '{"reasoning_strength":"low"}' server-wide, or per request. It defaults to high and cannot be switched off; --reasoning-budget N caps it.

Ollama, if you would rather not build anything

ollama run muse-glimmer:30b pulls the same build — though not, for one of the two layers, the same file. Re-checked against the registry manifest on 2026-08-21, the digests are unchanged since this model's first recipe round:

• The projector layer, 1,400,328,928 bytes at f48b4523…, is bit-identical to Meta's projector.
• The model layer matches Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf to the byte in size (16,756,681,056 against 16,756,683,904 is a 2,848-byte header difference), but its content hash is 71b5c9c9… against Meta's 4cc57c0f…. It is the same K-Quant-17GB build, repackaged, not the same bytes.
• muse-glimmer:latest is the same manifest as :30b — same config digest, same two layers. There is no separate "latest" artifact to choose between.

For the drafter, pull muse-glimmer:30b-q4_K_M-dflash: the same two layers plus a third of exactly 1,631,205,312 bytes at digest 27d9a805…. That tag also ships a 64-byte parameters layer setting draft_num_predict to 3, so out of the box it drafts three tokens per step rather than the fifteen the llama.cpp path allows.

Ollama labels the tag q4_K_M and displays 18 GB, which is the sum of its two layers rather than either file. This is the 24 GB-target build, not the 32 GB-target K-Quant-Dynamic one. One caveat: Ollama's blobs still correspond to the pre-fix files, so this route carries the older chat template that the 2026-08-12 republish corrects. If you rely on reasoning-strength control, prefer the llama.cpp path with the canonical files.

The vendor's own CUDA export, and the choice it forces on 24 GB

Meta also ships pre-exported ExecuTorch artifacts in meta-models/Muse-Glimmer-30B-ExecuTorch-PTE, and one of the two backends is CUDA: the repo describes sm80+ptx as "NVIDIA CUDA, SM80 and newer", which includes this card at sm_89. It is a genuine alternative runtime, but on 24 GB it is a narrower one than llama.cpp, and the repo's own download-size table shows why. For the k-quant-17G quant the CUDA variants weigh 19.8 GB text-only, 21.2 GB with vision, 27.2 GB with the drafter and 28.6 GB with both — so the two dflash combinations are larger than the card before anything else is allocated, while the llama.cpp stack fits vision and the drafter in 20.198 GiB. The CUDA export is also more restricted in what the drafter may do: the README notes "the exported range differs by backend", allowing a block length of 2–16 on Apple's Metal export but only 2–4 on sm80+ptx, with at most 3 draft tokens. Take this route if you want PyTorch's ahead-of-time export path; take llama.cpp if you want images and speculation at once.

Results

Speed. Meta publishes no RTX 4090 figure. Its speed table covers, in its own words, "We measure the speed of our K-Quant-17GB model alongside the quantized DFlash drafter on MacBook M4-Max, M5-Max and on an Nvidia RTX-5090" — three machines, none of them this one — and its footnote adds that "M4/M5 measurements were done using ExecuTorch, and RTX using llama.cpp". One third-party measurement does name this card:

Those rows come from Hardware Corner's Muse Glimmer hardware article, bylined Allan Witt and updated 2026-08-17. Note where it sits: this is the publisher's per-model article, not one of its per-GPU benchmark tables, and its RTX 4090 benchmark page — the surface that carries Qwen3 8B, 14B, 32B and 30B-A3B rows — has no Muse Glimmer row at all. The article does state its methodology: "All benchmarks were run on Ubuntu 24.04 with CUDA 13, NVIDIA driver 595.58.03, and llama.cpp build 153d324bc", and "Our tests use the Q4_K Medium GGUF quantization". It compares the card against a 3090 measured on the same harness — "Prompt processing on the 4090 is close to double the 3090 at short context" — and its 5090 row reaches 83.22 tok/s at 4k.

Read that table as a baseline for a different configuration, not as this recipe's speed, for three reasons found by checking it rather than by trusting it:

1. No drafter and no projector. The article never mentions DFlash or an mmproj. Its numbers are plain autoregressive decode; Meta's own 5090 pairing shows the drafter is worth a multiple, not a percentage.
2. The quant is probably not the one installed here. The article reports its file as 14.78 GiB, and no Q4_K_M in any of the main repos is that size — Meta's is 15.606 GiB, LM Studio's mirror of it is the same file, and bartowski's Q4_K_M is 16.12 GiB. The nearest match is unsloth's UD-Q4_K_XL at 14.79 GiB, which is a quantiser-specific dynamic family rather than a generic K-quant.
3. Its context axis overshoots the model. The article says the model "fits a full 256k context window into just 20 GB" and that a single 24 GB card is "enough to cover the entire context range the model supports". Every published GGUF of this model declares a context length of 131,072 — Meta's, LM Studio's, unsloth's and bartowski's alike, read from HuggingFace's own GGUF header parse — so its longest rows sit past the trained context and are omitted from the table above.

There is no first-party RTX 4090 measurement, and no measurement at all of this recipe's exact configuration on this card. If you run one, please contribute the numbers — a baseline-and-DFlash pair on the K-Quant-17GB build would turn /check/muse-glimmer-30b/rtx-4090 from an empty page into an anchor.

VRAM usage: 20.198 GiB derived resident set at full context — see the budget below and /check/muse-glimmer-30b/rtx-4090.

Quality notes: the model card rates the K-Quant-17GB build at 1.0% average degradation across 15 benchmarks, against 0.2% for the 32 GB-target K-Quant-Dynamic build.

The 24 GB budget

The interesting question on a 24 GB card is not whether 16.76 GB of weights fit — it is whether they still fit once the vision projector, the drafter and a 131,072-token KV cache sit beside them. They do, with room to spare, and the reason is architectural.

Muse Glimmer uses grouped-query attention with 32 query heads against 2 KV heads and a 128-wide head, so one token costs 1,024 bytes of f16 KV per layer. Its layer_types list is [sliding, sliding, sliding, full] repeating across 52 layers — 13 full-attention layers and 39 sliding-window layers with a 2,048-token window. llama.cpp allocates these as two separate caches: only the 13 full-attention layers scale with context, while the 39 sliding-window layers are pinned to a ring of sliding_window + n_ubatch cells — 2,560 at the default -ub 512. So three quarters of the layers cost nothing as context grows.

That leaves 3.802 GiB of the card's 24 GiB. Two things live in that margin and are not in the table, because we did not measure them: llama.cpp's CUDA compute buffer, which scales with -ub, and the CUDA context plus whatever your desktop is holding. Budget accordingly on a machine driving a monitor.

Two independent cross-checks say the arithmetic is sound. Meta's own GGUF card publishes a rough total for exactly this combination — the 17gb build with vision and the drafter — and puts it at ~20 GB. And running the same derivation on the K-Quant-Dynamic build gives 22.897 GiB, only 1.103 GiB clear of the card, which is why Meta targets that build at 32 GB rather than 24 GB. Our numbers reproduce the vendor's own two-tier split without being fitted to it.

Troubleshooting

unknown model architecture: 'muse-glimmer'

Your llama.cpp predates the architecture. Fetching src/llama-arch.cpp at each tag and counting LLM_ARCH_MUSE_GLIMMER gives a clean boundary: b10344 resolves and has zero hits; b10353 and b10358 resolve and have one each. Upgrade to b10353 or newer, or build from source. The tags in between were never published, so a 404 there means "no such release", not "feature missing".

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

This is an out-of-memory condition wearing a bounds-check error's clothes. It is not specific to Muse Glimmer, to the DFlash drafter, or to any GGUF — it is what llama.cpp prints when every visible device reports zero free VRAM at the moment a model is loaded.

That is not how issue #26894 reads at first, and its title still says otherwise. It was filed against Meta's own GGUF and blamed the way that file encodes muse-glimmer.attention.sliding_window_pattern — an array of 52 booleans, where third-party conversions write a scalar. The reporter withdrew that diagnosis on 2026-08-13 — "My original diagnosis was wrong" — and named the real mechanism instead: "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the exact file the issue was filed against, checked its SHA-256 against the Hub, and bound the drafter successfully 10 times out of 10, concluding that the published file is fine as it stands. The array-versus-scalar difference was a coincidence: he had tested the rewritten file later, on a card that happened to be free.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum of those weights.
• There is a zero guard, but it only catches a device reporting free == 0 and total == 0 — a device with nothing at all to report (#18577). A real GPU that is merely full reports free == 0 against a real total, and falls straight through it.
• Every entry in splits is then 0, so split_sum == 0, and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs finds no element (x < NaN is false), returns end(), and devices.at(layer_gpu) throws. With one visible device that is __n (which is 1) >= this->size() (which is 1), verbatim.

It surfaced on the draft model in the original report for an ordinary reason: the ~20 GB target loads first and takes the card with it, so the drafter is simply the next allocation to ask a full GPU for room. The reporter reproduced it two ways — an LD_PRELOAD shim forcing free = 0, and naturally, by saturating the GPU from a separate process — including on an unrelated dense model with no speculative decoding at all.

What to do. Free the card, then re-run. Close other CUDA processes and check with nvidia-smi. On 24 GB this recipe's resident set is 20.198 GiB, leaving 3.802 GiB for the compute buffers, the CUDA context and your desktop — a compositor holding a gigabyte is enough to matter, so running headless or on the integrated display output is the durable fix. If you cannot free enough, lower -c, or drop -md … --spec-type draft-dflash -ngld 99 and recover 1.568 GiB at the cost of the speculative speedup.

Do not rewrite the GGUF's metadata. The scalar-sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and it leaves you running a file whose checksum no longer matches the Hub. The issue remains open at the time of writing, now as a request for a clear "insufficient device memory" message in place of the out_of_range that sent the original investigation down the wrong path; nothing has been added to the thread since the retraction.

VRAM climbs after the first image (and the prefill drop that comes with it on multi-GPU rigs)

Open issue #26873 reports two effects after the first image: the projector costs a further 1.1 GB beyond the mmproj weights already accounted for, and prompt processing then falls and stays down even on a fresh context. Only the first half can reach you on one card, and that is worth knowing before you copy a workaround. The thread's second and only other participant narrowed the prefill regression to a single condition — it happens only when pipeline parallelism is enabled — and the issue's own reporter then confirmed on his rig that forcing a tensor override (-ot "zzz_never_matches=CUDA0") makes it disappear; self-confirmation rather than an independent third voice. That is consistent with llama.cpp's own gate: src/llama-context.cpp enables pipeline parallelism only when model.n_devices() > 1 and !model.has_tensor_overrides(), among other conditions, so on a single RTX 4090 it is never on and the prefill drop cannot occur. The memory growth is a separate claim, reported on a two-GPU rig and never tested on one card. Treat it as a reason to watch the 3.802 GiB margin the first time you send an image, not as a reason to give up the drafter.

DSpark is not DFlash, and it can be slower

Meta ships a DFlash drafter; llama.cpp also supports a DSpark type, which outranks DFlash in the sidecar auto-detection when both are present — the resolution order in common/arg.cpp tries MTP, then DSpark, then DFlash, with the comment that DSpark wins because its sidecar carries an extra Markov head. They are not interchangeable: the quantiser bartowski, reporting on his own RTX 3090, measured his Q4_K_M build at "around 38 tok/s" falling to "~28-32 depending on the task" once DSpark was used instead. Pass --spec-type draft-dflash explicitly rather than relying on auto-detection.

"DFlash 2" is not in any llama.cpp release yet

Two repos published on 2026-08-18 — z-lab/Muse-Glimmer-30B-DFlash2-GGUF and its mirror incoai/Muse-Glimmer-30B-DFlash2-GGUF — offer a second-generation drafter for this exact target, at a Q4_K_M size of 1.65 GB, close enough to the shipped drafter to look like a drop-in. It is not one yet. Its own card tells you to build llama.cpp with DFlash 2 support from PR #27342, and that pull request is open and unmerged; the string dflash2 appears nowhere in master's common/speculative.cpp, common/arg.cpp or src/llama-arch.cpp, whose speculative-type table lists draft-dflash and draft-dspark and no successor. The trap is that it will not look unsupported: the file declares general.architecture = dflash, and llama.cpp auto-detects any dflash-arch draft without a Markov head as plain draft-dflash, so a release binary will classify it happily while implementing none of the local convolution or candidate selector the second generation is. Stay on Meta's dflash-Muse-Glimmer-30B-Q4_K_M.gguf until that PR merges.

-hf picks the wrong file

llama.cpp's repo auto-selection looks for Q4_K_M or Q8_0 in the filename, then falls back to the first model-shaped GGUF in the repo. Since the 2026-08-12 republish the canonical name carries Q4_K_M, so a bare -hf meta-models/Muse-Glimmer-30B-GGUF resolves to Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf by a real tag match rather than by ordering. Meta deleted the superseded *-kquant*.gguf files on 2026-08-18, so a stale pinned name now 404s instead of resolving to the old build. Pin the canonical one: --hf-file Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf.