Muse Glimmer 30B on RTX 5090: K-Quant-Dynamic, vision and DFlash in llama.cpp

What You'll Build

An OpenAI-compatible llama-server on a single RTX 5090 running Meta's Muse Glimmer 30B — an agentic model that takes interleaved text and images and emits tool calls — loaded from Meta's own K-Quant-Dynamic GGUF, with the vision projector attached and the DFlash block-diffusion drafter wired in for speculative decoding.

This is deliberately the 32 GB configuration. Meta ships two 4-bit builds and targets them at different cards: K-Quant-17GB at 24 GB and K-Quant-Dynamic at 32 GB, with the quantisation section of the model card noting "This leaves enough headroom for the model's KV cache, the perception encoder for image understanding, and the speculative decoding drafter to run simultaneously within a 24 GB or 32 GB envelope." The RTX 5090 is the consumer card that gets the larger, less lossy one — the card puts K-Quant-Dynamic at 0.2% average degradation across its 15-benchmark suite against 1.0% for K-Quant-17GB.

Hardware data: RTX 5090 (32 GB VRAM) · derived working set 22.90 GiB at the full 131,072-token context · See benchmark data

⚠️ Two known issues, both in Troubleshooting. Builds from before the architecture landed reject the file outright with llama_model_load: error loading model: unknown model architecture: 'muse-glimmer'. Separately, llama.cpp issue #26894 reads like a warning against this recipe's exact target and drafter files — it is not. Its reporter withdrew the diagnosis on 2026-08-13; the crash is what llama.cpp prints when the card has no free VRAM. Read that entry if you meet vector::_M_range_check.

ℹ️ Meta's headline speed numbers are for the other quant. The model card's RTX 5090 row (74.9 tok/s baseline → 233.4 tok/s with DFlash, a 3.1× speedup) is prefaced by "We measure the speed of our K-Quant-17GB model alongside the quantized DFlash drafter" — so it describes the 24 GB build, not the K-Quant-Dynamic weights this recipe installs. That table is the card's only speed table, so the card publishes no first-party tok/s figure for K-Quant-Dynamic on any hardware. It is at least the right runtime for this page: the card's own footnote reads "M4/M5 measurements were done using ExecuTorch, and RTX using llama.cpp." Treat the 233.4 as context, not as a target for this configuration, and read the Results section before you calibrate expectations.

Requirements

Where the 32 GB floor comes from

Every byte below is either a file size read from the HuggingFace tree API for meta-models/Muse-Glimmer-30B-GGUF, or arithmetic over the published architecture. Nothing here was measured on a card.

That leaves 9.10 GiB of the card's 32 GiB for the CUDA context, compute buffers and image tensors — comfortable, and the reason this quant belongs on a 5090 rather than a 24 GB card, where the same three files plus KV would land inside 1.1 GiB of the ceiling.

The KV figure is small for a 30B because of two architecture choices, both in the model's config.json: GQA at 32 query heads to 2 KV heads, and a repeating [local, local, local, global] layer pattern with a 2,048-token sliding window. Of 52 layers, 13 are full attention and 39 are sliding. llama.cpp models this with two separate caches (llama_kv_cache_iswa): the full-attention cache gets n_ctx cells, the sliding one gets GGML_PAD(min(n_ctx, n_swa + n_ubatch), 256) = 2,560 cells at the default -ub 512. At f16 one cell of one layer costs 2 × 2 KV heads × 128 head-dim × 2 bytes = 1,024 bytes, so:

• full-attention layers: 13 × 131,072 × 1,024 = 1,744,830,464 bytes
• sliding layers: 39 × 2,560 × 1,024 = 102,236,160 bytes

The drafter is 5 layers, all sliding, at 8 KV heads × 128 head-dim → 4,096 bytes per cell per layer, so 5 × 2,560 × 4,096 = 52,428,800 bytes. Going from 131,072 down to 32,768 context only saves 1.22 GiB, because 39 of the 52 layers do not grow with context at all — there is little reason to run this model short.

Installation

1. Build llama.cpp with CUDA for Blackwell

Support for this architecture merged as PR #26841 on 2026-08-10, in commit 62bf73d. That commit is build number 10349, but llama.cpp does not publish a tag for every build — b10345 through b10352 do not exist as tags — so the earliest release you can download with the architecture in it is b10353.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j
./build/bin/llama-server --version

2. Confirm your checkout actually has the architecture

Cheaper than discovering it at load time, and it answers the question for whatever build you happen to have:

grep -n 'LLM_ARCH_MUSE_GLIMMER' src/llama-arch.cpp
grep -n 'PROJECTOR_TYPE_MUSE_GLIMMER' tools/mtmd/clip-impl.h
grep -n 'COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH' common/speculative.cpp

Three hits means the text model, the vision projector and the DFlash speculator are all present in your tree. On master at 9558fa44 all three are there, with a dedicated clip_graph_muse_glimmer builder and a mtmd_image_preprocessor_muse_glimmer — vision for this model is implemented in mainline mtmd, not merely permitted by the presence of an mmproj file.

3. Download the three first-party GGUF files

hf download meta-models/Muse-Glimmer-30B-GGUF \
  Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf \
  mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --local-dir ~/models/muse-glimmer-30b

If you reach for llama.cpp's -hf shorthand instead of downloading by name, tag it. A bare -hf meta-models/Muse-Glimmer-30B-GGUF looks for Q4_K_M and then Q8_0, and since the rename the only file in the repo matching either is the 24 GB-target Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf — so you would quietly get the smaller build this recipe is not about. -hf meta-models/Muse-Glimmer-30B-GGUF:Q4_K_XL picks the Dynamic build instead.

All three live in the same first-party repo under Apache 2.0. Use these names, not the *-kquant*.gguf ones. Meta republished the text and drafter builds under canonical Q4_K names on 12 August with a corrected embedded chat template; the superseded objects were deleted from the repo on 2026-08-18, and the GGUF card says of the old template that it "skipped both, which could leave a conflicting Reasoning strength: high. in the prompt" and to re-download if you pulled before the fix. The projector is the same object under both names — byte-identical, oid f48b452316f9… — so only the text and drafter files actually changed. The drafter is a genuinely separate model — its unquantised source is meta-models/Muse-Glimmer-30B-assistant, 5 layers with block_size: 16 and target_layer_ids: [1, 13, 25, 37, 49], and llama.cpp gives it its own GGUF architecture (dflash) rather than treating it as an ordinary small draft model.

Running

~/llama.cpp/build/bin/llama-server \
  -m  ~/models/muse-glimmer-30b/Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf \
  -mm ~/models/muse-glimmer-30b/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -md ~/models/muse-glimmer-30b/dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 15 \
  --spec-draft-n-min 2 \
  -ngl all -ngld all \
  -c 131072 -np 1 \
  -fa on --jinja --reasoning-preserve \
  --host 127.0.0.1 --port 8080

The four flags that are not boilerplate:

• -md + --spec-type draft-dflash — pass both, and note that why changed on 2026-08-13. Before that date the speculative type stayed at its default { COMMON_SPECULATIVE_TYPE_NONE } and -md did not write to it, so the drafter loaded and was never used — not downgraded to a simpler speculator, but skipped entirely: common_get_enabled_speculative_configs builds its mask as 1u << type, {NONE} gives 1, and every real implementation's bit is 2 or higher. The reporter in discussion #34 hit exactly this and called the flag the key missing one; his own guess that the server had fallen back to draft-simple is not what the source does, but the fix he found was right, and every community build in this page's corpus (b10352–b10358) predates the change. Since #26814 ("common : auto-detect spec type from draft GGUF metadata", merged 2026-08-13T10:34:28Z, first released in b10413 — b10412, tagged 41 minutes earlier, does not have it) llama.cpp infers it instead: common/arg.cpp calls common_speculative_types_from_gguf whenever no type was requested and a -md path is set, and that function reads general.architecture, answering draft-dspark for a draft carrying markov_w1.weight and draft-dflash for one that does not. So from b10413 the flag is redundant and below it essential — which is why this page passes it either way. You can confirm it engaged from the startup log line common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'. If instead this is the step that crashes, jump to the drafter-binding entry in Troubleshooting — there is an open issue against exactly this pairing.
• --spec-draft-n-max 15 — DFlash denoises a block of block_size tokens in place and yields at most block_size - 1 of them, so with block_size: 16 the ceiling is 15. Ask for 16 and llama.cpp clamps it and logs a warning (common/speculative.cpp).
• -mm — the vision projector. Drop it and you have a text-only server; the model's own tokenizer still carries image and video token ids either way, so nothing will complain.
• -np 1 — a single slot keeps the whole KV budget on one sequence, which is what the derivation above assumes.

Sampling follows the model card's recommendation: --temp 1.0 --top-p 0.95 --top-k 64. Set reasoning strength through the template variable rather than by hand-writing a line into your system prompt — --chat-template-kwargs '{"reasoning_strength":"xhigh"}' server-wide, or "chat_template_kwargs": {"reasoning_strength": "high"} per request. It defaults to high, and xhigh suits coding and agentic work. Reasoning cannot be turned off at all; to cap it, use --reasoning-budget N. --jinja is what activates the embedded template, so it is not optional here.

For images, POST to /v1/chat/completions with an image_url content part in the usual OpenAI shape — llama-server is one of the binaries llama.cpp documents as multimodal-capable. Beyond the source-level support, one first-hand report of the vision path working end to end with this mmproj comes from a reader running the K-Quant-17GB build on an RTX 3090 in discussion #13; that is a different card, so take it as evidence that the path works, not as a number for this one.

Results

Speed — three independent RTX 5090 reports, all community, all on llama.cpp. The backend has no benchmark row for this pair yet (/check/muse-glimmer-30b/rtx-5090 returns verdict: unknown with an empty benchmark list), so everything below is attributed to the person who ran it:

Two things to take from that table. First, the baselines reproduce Meta's 74.9 tok/s closely, and the second reporter says so himself: "consistent with Meta's published 74.9 tok/s baseline". Second, the DFlash speedup is prompt-dependent to a degree the single headline number hides. A Meta engineer replying in discussion #21 confirms it: "So 233.4 tok/s is the mean, it's a wide interval as you can see from the image", and "In general the dflash head has been mostly optimized for agentic and coding flows, so that's where I'd expect the highest speed-ups." The same reporter who measured 137 tok/s on a general prompt then measured 238.83 tok/s on a coding prompt with acceptance rising from 13.7% to 28.1% — above Meta's published mean, on the same machine and the same build. If your workload is agentic, the headline is reachable; if it is open-ended chat, expect the 137–168 tok/s the first two reporters saw.

The one datapoint on the quant this recipe actually installs is the third row, and it is the slowest of the three — a fair result, since K-Quant-Dynamic is 2.9 GB larger than K-Quant-17GB and was run at the full 131,072-token context. It is also a decode figure with no matching baseline, so it cannot be turned into a speedup. Searching for a complete baseline-vs-DFlash pair on K-Quant-Dynamic, I read all 49 discussion threads on the canonical repo and all 4 on the GGUF repo, and ran a web search for the model plus the card name: none of them carries one. If you run one, please contribute it — it is the single most useful missing number for this page, and it would turn /check/muse-glimmer-30b/rtx-5090 from an empty page into an anchor.

VRAM usage: derived at 22.90 GiB for weights + projector + drafter + KV at full context, as broken down above; not measured on a card by us.

Quality notes: Meta puts K-Quant-Dynamic at 0.2% average degradation across 15 benchmarks against full precision, versus 1.0% for K-Quant-17GB — which is the whole argument for spending a 5090's extra 8 GB on this build. Community reaction in the model's discussions is mixed on the model itself — one thread is titled "Works well, comparable to Qwen 3.6 27B and then some", while a reader in discussion #13 reports the reasoning forgetting and re-iterating. That is a judgement about the model, not about this configuration.

Troubleshooting

unknown model architecture: 'muse-glimmer'

The build predates the architecture. This is the most common report on the GGUF repo, filed against b10338 and reproduced on b10344 (GGUF discussion #1, #2).

Fetching src/llama-arch.cpp at each tag and grepping for LLM_ARCH_MUSE_GLIMMER gives a clean boundary: b10344 resolves and has 0 hits; b10353, b10354 and b10358 resolve and have 1 each. The tags in between — b10345 to b10352 — return HTTP 404, meaning they were never published, not that the feature is missing; the arch commit's own build number, 10349, falls in that unpublished gap, which is why the issue reporter's from-source build says b10349 and no such download exists. Use b10353 or newer, or build from source. Meta's GGUF card independently states the same floor — b10353, with b10344 and older not registering the architecture — which is a useful cross-check on the bisect above rather than a substitute for it. Either way, run the greps in Installation step 2 against your own checkout rather than trusting a build number.

Tool calls after an end-of-message marker get missed

Relevant if you are driving this as an agent, which is the point of the model. PR #26879, "Muse Glimmer: fix detection of tool calls after EOM", merged on 2026-08-11 — a day after the architecture itself. A build cut on 2026-08-10 loads the model perfectly and still mis-parses this case. For agentic use, take a build dated 2026-08-12 or later.

DFlash engages but you see a 1.6× speedup, not 3.1×

Expected on a general-purpose prompt; see Results. One thing not to chase: the GGUF card notes that a [spec] failed to measure draft model memory warning at startup is harmless and the drafter loads and serves normally afterwards. Two things are worth checking before you chase the speedup further. Confirm the startup log shows block_size=16 and a non-zero n_extract — if the drafter loaded as a plain draft model instead, the log will not name draft-dflash at all. And note there is an optimisation in flight that is not in your build. In discussion #21 Meta's ruanslv points the reporter at PR #26842, describing it as "an important optimization, we haven't managed to merge it in time"; the reporter then posts a table in that same thread showing 129.35 tok/s on master against 139.79 with the patch applied. Both of those numbers are from the discussion thread — PR #26842 itself is still a draft, and the benchmark on its page is for different hardware and a different quantisation, so do not read it as a prediction for this card.

--spec-draft-n-max 16 looks accepted but is not

DFlash yields at most block_size - 1 = 15 draft tokens. llama.cpp clamps a larger request and emits requested draft size … exceeds the trained block size 16 -- clamping to 15. Some invocations circulating in the discussions pass 16; they run, they just do not do what they look like they do.

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

This is an out-of-memory condition wearing a bounds-check error's clothes, and it is not specific to Muse Glimmer, to DFlash, or to any GGUF. llama.cpp prints it when every visible device reports zero free VRAM at the moment a model loads.

You will find issue #26894 saying otherwise. It was filed against this recipe's exact target and drafter files and blamed the way Meta's GGUFs encode muse-glimmer.attention.sliding_window_pattern — a 52-entry boolean array where third-party conversions write the scalar 4. Its reporter withdrew that diagnosis on 2026-08-13 — "My original diagnosis was wrong" — and named the real mechanism instead: "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the file the issue names, verified its SHA-256 against the Hub, and bound the drafter 10 times out of 10, concluding that the published file is fine as it stands. The array-versus-scalar difference was real and irrelevant — he had tested the rewritten file later, on a card that happened to be free.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum.
• The zero guard there catches only free == 0 and total == 0 — a device with nothing at all to report (#18577). A real GPU that is merely full reports free == 0 against a real total and falls through it.
• Every splits entry is then 0, so split_sum == 0 and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs matches nothing (x < NaN is false), returns end(), and devices.at(n_devices()) throws. On one visible device that is the message above, verbatim.

The crash landed on the draft model in the original report for a mundane reason: the ~20 GB target loads first and takes the card, so the drafter is the next allocation to ask a full GPU for room. The reporter reproduced it with an LD_PRELOAD shim forcing free = 0, and naturally by saturating the GPU from another process — including on an unrelated dense model with no speculative decoding at all.

What to do. Free the card and re-run: close other CUDA processes, check nvidia-smi, and prefer a headless box or the integrated display output. On 32 GB this recipe has real headroom, so if you meet this here, something else is holding the card. If you cannot free enough, lower -c or drop the drafter flags.

Do not rewrite the GGUF's metadata. The scalar-sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and it leaves you running a file whose checksum no longer matches the Hub.

The configuration is well attested. Every public report that actually exercises the drafter — each had to name a draft file to qualify, because --spec-type draft-dflash alone does nothing (llama.cpp builds a DFlash speculator only when a draft model is set, and its constructor asserts on having both contexts):

A posted command carrying the flag but no -md, --model-draft or -hfd is not evidence about the drafter at all — the automatic dflash- sidecar lookup only runs for -hf-style downloads and cannot fill the gap. An acceptance rate is the other reliable tell, since a drafter that never bound cannot produce one.

One genuinely separate change did land. PR #26900 ("disallow integer dflash sliding_window_pattern", merged 12 August) is a one-line change to src/models/dflash.cpp, swapping get_key_or_arr for get_arr in the drafter's own hparams loader. Its author struck through the line claiming it closed #26894 — correctly, as it turns out. It is also newer than every published tag above: src/models/dflash.cpp at b10353, b10354 and b10358 still has the old call, so on a release binary you are running the pre-#26900 loader either way. The issue remains open at the time of writing, now as a request for a clear "insufficient device memory" message in place of the out_of_range.

Ollama's muse-glimmer:30b is Meta's K-Quant-17GB, repackaged

The library entry exists, takes image input, and labels its default q4_K_M — which reads like a generic community quantisation and is not one. Pulling the registry manifest for :30b and comparing its layer digests against the HuggingFace LFS object ids settles what you actually get:

Range-reading that model layer's own header returns 731 tensors, 32 metadata pairs, general.name = "Muse Glimmer Hf" and the 52-entry sliding_window_pattern array — the same census, name and encoding as Meta's K-Quant-17GB build. So q4_K_M here is Ollama's label on Meta's K-Quant-17GB build, not a separate quantisation. The displayed 18 GB is the sum of the layers (16.76 + 1.40 = 18.16 GB), which is why it matches neither file on its own; that arithmetic is what makes the tag easy to misread.

Two consequences. First, :30b is the 24 GB-target build, so on a 32 GB card it leaves the 0.2%-degradation K-Quant-Dynamic on the table — the whole reason this recipe leads with llama.cpp. Second, :30b-q4_K_M-dflash is a superset, not a different quant: identical projector and model layers plus a 1,631,205,312-byte draft layer whose digest 27d9a805fa29… is bit-for-bit Meta's superseded dflash-kquant.gguf. The plain :30b tag simply has no draft layer, so pulling it and expecting DFlash speedups will disappoint. That draft layer is bit-identical to Meta's superseded dflash-kquant.gguf, which is now the only place that object still exists — Meta deleted the old names from the Hub on 2026-08-18.

One more reason to prefer the first-party path today: those layers were cut before Meta's 12 August republish. The model layer's 16,756,681,056 bytes match the superseded muse-glimmer-30B-kquant-17gb.gguf exactly, and the draft layer's digest is the superseded drafter's, so the artifacts behind this route are the pre-fix ones; the projector is the one layer unaffected, because that object did not change. The template defect does not reach you on this route, though, whatever the blobs' age. Every muse-glimmer tag's config blob — GGUF and MLX alike — declares "renderer": "glimmer" and "parser": "glimmer", and in server/prompt.go renderPrompt returns from the named renderer before it can reach m.Template.Execute, so the artifact's embedded Jinja is never consulted. The Go renderer carries both of the fixes the corrected template gained: model/renderers/glimmer.go maps all four casings of Reasoning effort to Reasoning strength, then appends its own directive only if defaultSystem || !glimmerHasSystemReasoning(content) — the Go equivalent of the template's {%- if 'reasoning strength' not in (sys_text | lower) -%}. Ollama's own reference test pins that renderer against a fixture its comment records as copied byte-for-byte from the chat template at publisher revision a4e59da5…, SHA-256 cfc67e5f349f… — the corrected one. Scope: this settles prompt formatting only. The weights behind these tags are still the superseded objects, and nothing here says what parser: glimmer does on the output side.