Muse Glimmer 30B on RTX 3090 Ti: vision + DFlash speculation at full 131K context

What You'll Build

A local agent endpoint on a single RTX 3090 Ti: llama-server running Meta's Muse Glimmer 30B with image input and DFlash speculative decoding, at the model's full 131,072-token context — not a reduced one. Every component is resident on the card; nothing offloads to system RAM.

Hardware data: RTX 3090 Ti (24 GB VRAM) · derived resident set 20.198 GiB at 131,072 context, 3.802 GiB spare · See benchmark data

ℹ️ A retracted bug report you will still find in search results. llama.cpp issue #26894 originally blamed Meta's own GGUF for a crash when binding the DFlash drafter. Its reporter withdrew that diagnosis on 2026-08-13 — the crash is llama.cpp's response to a card with no free VRAM, not to anything in the file. If you meet vector::_M_range_check, read Troubleshooting: it is an out-of-memory message in disguise.

ℹ️ Multimodal means text + images, not audio. Muse Glimmer accepts interleaved text and images through a ViT-G/14 perception encoder, and emits text only. The model card is explicit: "Audio input/output is not supported." Video is not a supported input either — the card notes the model is not explicitly optimized for it and processes it as individual frames.

Requirements

Ampere has no FP4 or FP8 tensor path, so the NVFP4 and MXFP8 conversions on the Hub are not options here — and note they are third-party work, not Meta's: the meta-models org publishes exactly four repos (the bf16 base, the GGUF set, the drafter source and the ExecuTorch export), and the model card never mentions FP8 or FP4 in any form. On an RTX 3090 Ti the route is the K-quant GGUF stack below.

The power row is the one requirement that genuinely separates this board from the rest of the 24 GB tier. NVIDIA's own RTX 3090 / 3090 Ti spec page lists 450 W graphics-card power and an 850 W recommended system supply against 350 W and 750 W for the plain 3090, and gives the 3090 Ti a different connector story — three PCIe 8-pin cables, or a single 450 W-or-greater PCIe Gen 5 cable. A long llama-server session holds the card near its power limit for as long as it is generating, which is exactly the load that finds an undersized supply.

What the Ti changes, and what it does not

It does not change the build flags. The 3090 Ti is the full GA102 die, but it is still Ampere and still compute capability 8.6 — the same sm_86 as the plain 3090. CMAKE_CUDA_ARCHITECTURES=86 below is correct for this card, unchanged.

It does not change the memory budget. Both boards hold 24 GiB, the GGUF files are the same three files, and llama.cpp sizes its caches from the model header rather than from the board — so the 24 GB budget below is arithmetically identical on either card. A recipe that invented a different number here would be inventing it.

It does change how fast the same work finishes, by a small and measurable amount. Hardware Corner's RTX 3090 Ti page gives the board 24 GB of GDDR6X at 21 Gbps on a 384-bit bus — 1,008 GB/s — and puts the plain RTX 3090 at 92% of it on their token-generation index. The same page is blunt about what that is worth: "the performance difference between this and the standard RTX 3090 is often negligible".

Our own catalogue can check that index, because five other models carry benchmarks on both boards from that same source. They are a see-also — none of them is this model — but they measure the thing this section is about:

Seven to nine percent, consistently, on five unrelated models. That is the number to carry into the Results section: it is what tells you a figure measured on a plain 3090 is a floor for this card rather than a target.

Installation

1. Get a llama.cpp build that knows the architecture

Muse Glimmer support landed in PR #26841, merged 2026-08-10 as commit 62bf73d2. Release b10353 is the first tag that contains it — b10344 and earlier sit five commits behind the merge and fail with unknown model architecture: 'muse-glimmer'.

⚠️ b10353 is the floor for loading the model, not for driving it as an agent. PR #26879, "Muse Glimmer: fix detection of tool calls after EOM" merged 2026-08-11T20:15:21Z; b10358 was published 2026-08-11T09:04:42Z, eleven hours earlier, and b10353 earlier still. Both load the model perfectly and both mis-parse a tool call that follows an end-of-message marker — which on an agentic model is the whole point of the thing. That PR also added llama.cpp's own bundled models/templates/muse-glimmer.jinja. Take a build dated 2026-08-12 or later; this recipe pins today's release.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10549
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

CMAKE_CUDA_ARCHITECTURES=86 is the RTX 3090 Ti's compute capability (Ampere, sm_86). b10358 is pinned because it is the build with the most public Muse Glimmer + DFlash successes behind it; any later release works too. Prebuilt release binaries from the same tag are fine if you prefer not to compile.

2. Download the three GGUF files

hf download meta-models/Muse-Glimmer-30B-GGUF \
  Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --local-dir ./muse-glimmer

Three files, three jobs: the 16.76 GB quantised language model, the 1.40 GB vision projector, and the 1.63 GB DFlash drafter. Meta's model card describes the K-Quant-17GB build as compressing weights to roughly 4-bit, "shrinking the language model to under 20 GB", and states that this "leaves enough headroom for the model's KV cache, the perception encoder for image understanding, and the speculative decoding drafter to run simultaneously within a 24 GB or 32 GB envelope." The budget under Results checks that claim against the actual bytes.

⚠️ These four names are the only ones that still exist. Meta republished the builds on 2026-08-12 with a corrected embedded chat template — the previous template could leave a conflicting reasoning-strength line in the prompt, and the GGUF card says to "re-download if you pulled before this fix". On 2026-08-18 Meta then deleted the superseded *-kquant*.gguf copies (commit 70bf1b61, "Remove superseded old-named GGUFs"), so any command still naming muse-glimmer-30B-kquant-17gb.gguf, mmproj-kquant.gguf or dflash-kquant.gguf now returns HTTP 404 rather than quietly fetching the old build. That includes most of the boot scripts posted in the model's discussion threads during the first week — see Troubleshooting. The projector is the one file that did not really change: mmproj-Muse-Glimmer-30B-Q4_K_M.gguf is the same object as the old mmproj-kquant.gguf (identical SHA-256 f48b4523…).

Running

./build/bin/llama-server \
  -m ./muse-glimmer/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
  --mmproj ./muse-glimmer/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
  -ngl 99 -c 131072 -np 1 \
  --jinja \
  --host 127.0.0.1 --port 8080

This is the shape Meta publishes in its own llama.cpp deployment doc, which also names b10353 as the minimum release. --jinja is not optional: the model ships an ATEM tool-calling chat template, and without --jinja tool calls and reasoning separation break. -np 1 matters twice over — each extra server slot multiplies the sliding-window KV ring in the budget below, and the drafter's acceptance rate degrades under concurrent slots (#27117, below).

To add speculative decoding, append the drafter:

  -md ./muse-glimmer/dflash-Muse-Glimmer-30B-Q4_K_M.gguf \
  --spec-type draft-dflash -ngld 99 --spec-draft-n-max 4

The drafter adds 1.568 GiB to the resident set — its weights plus its own KV — which the budget below accounts for. If the card is already close to full, read Troubleshooting first: llama.cpp reports that condition as a bounds-check error rather than as an out-of-memory one.

Meta's recommended sampling is --temp 1.0 --top-k 64 --top-p 0.95. Reasoning strength is set in the system prompt as Reasoning strength: <low|medium|high|xhigh>, or through --chat-template-kwargs '{"reasoning_strength":"low"}' — and not through the OpenAI-style reasoning_effort field, which is a different knob (Troubleshooting).

Ollama, if you would rather not build anything

ollama run muse-glimmer:30b pulls the same build — though not, for one of the two layers, the same file. Its registry manifest lists layers of 16,756,681,056 and 1,400,328,928 bytes, and the distinction matters:

• The projector layer is bit-identical to Meta's projector — same SHA-256, f48b4523…, which is both mmproj-kquant.gguf and its renamed twin.
• The model layer matches the old muse-glimmer-30B-kquant-17gb.gguf to the byte in size, but its content hash differs (71b5c9c9… against Meta's 7e9b74b7…). It is the same K-Quant-17GB build, re-packaged, not the same bytes.

Ollama labels the tag q4_K_M and displays 18 GB, but this is the 24 GB-target build — not the 32 GB-target K-Quant-Dynamic one. The :latest tag resolves to the same manifest as :30b.

One caveat, re-checked on 2026-08-21: Ollama's three blob digests are unchanged since the model's release week, so this route still carries the pre-fix chat template that Meta's 2026-08-12 republish corrects. If you rely on reasoning-strength control, prefer the llama.cpp path with the canonical files.

For the drafter as well, pull muse-glimmer:30b-q4_K_M-dflash: the same two layers plus a third of exactly 1,631,205,312 bytes at digest 27d9a805…, which is bit-identical to Meta's original dflash-kquant.gguf.

Why not Meta's own ExecuTorch build on this card

Meta also ships pre-exported ExecuTorch artifacts, and their CUDA variants are not gated away from Ampere — the repo's own schema says "sm80+ptx = NVIDIA CUDA, SM80 and newer.", and this card is sm_86. The reason this recipe does not use them is size. Summing the files in each variant directory of meta-models/Muse-Glimmer-30B-ExecuTorch-PTE through the tree API, for the same K-Quant-17GB weights at the same 128K context:

Both variants that bundle the drafter are already larger than the card before a single byte of KV cache, and the two that fit are the ones with no speculative decoding — which is the feature this model is built around. The llama.cpp GGUF stack does the same job, with vision and the drafter, in 20.198 GiB. (Note also that the repo is 372 GB in total, so a bare hf download of it is a mistake; the card insists on --include for a single variant.)

DFlash2 is real, but not on this path yet

Two repositories appeared on 2026-08-18 offering a second-generation drafter: incoai/Muse-Glimmer-30B-DFlash2 (safetensors, for SGLang and vLLM) and z-lab/Muse-Glimmer-30B-DFlash2-GGUF (GGUF, for llama.cpp). They are deliberately absent from this recipe: the GGUF card's own quick-start tells you to build from PR #27342, which is open and unmerged, and llama.cpp master has no DFlash2 support — common/common.h enumerates DRAFT_SIMPLE, DRAFT_EAGLE3, DRAFT_MTP, DRAFT_DFLASH and DRAFT_DSPARK, and nothing else. Until that PR merges, using DFlash2 here means shipping a patched fork.

There is also less on offer than the headline suggests. The vendor's own acceptance-length figures are 6.57 on GSM8K under SGLang on an H200 — but the GGUF repo's own table, on the same benchmark, reports 5.44 for its Q4_K_M drafter, next to 5.43 for the official DFlash drafter in the SGLang comparison. Revisit when #27342 lands and someone publishes a llama.cpp-side comparison on one consumer card.

Results

• Speed: we have no RTX 3090 Ti measurement, and neither does Meta — its speed table publishes an RTX 5090, an Apple M4 Max and an M5 Max, and the string "3090" does not appear anywhere in the model card. The closest thing to a bound is the plain RTX 3090, which this card beats by seven to nine percent on every model we have measured on both. Read the two sets of 3090 figures below as a floor, not as a prediction. If you measure this pair, please contribute the numbers.
• VRAM usage: 20.198 GiB derived resident set at full context — see the budget below and /check/muse-glimmer-30b/rtx-3090-ti.
• Quality notes: the model card rates the K-Quant-17GB build at 1.0% average degradation across 15 benchmarks, against 0.2% for the 32 GB-target K-Quant-Dynamic build.

The 24 GB budget

The interesting question on a 24 GB Ampere card is not whether 16.76 GB of weights fit — it is whether they still fit once the vision projector, the drafter and a 131,072-token KV cache sit beside them. They do, with room to spare, and the reason is architectural.

Muse Glimmer uses grouped-query attention with 32 query heads against 2 KV heads and a 128-wide head, so one token costs just 1,024 bytes of f16 KV per layer. On top of that, its attention pattern is [Local, Local, Local, Global] repeating — which the GGUF encodes as a 52-entry sliding_window_pattern of 39 sliding-window layers and 13 full-attention layers. llama.cpp allocates these as two separate caches (build_attn_inp_kv_iswa): only the 13 full-attention layers scale with context, while the 39 sliding-window layers are pinned to a small ring of sliding_window + n_ubatch cells — 2,560 at the default -ub 512. So 75% of the layers cost nothing as context grows.

That leaves 3.802 GiB of the card's 24 GiB. Two things live in that margin and are not in the table, because we did not measure them: llama.cpp's CUDA compute buffer, which scales with -ub, and the CUDA context plus whatever your desktop is holding. Budget accordingly on a machine driving a monitor.

One cross-check says the arithmetic is sound: running the same derivation on the K-Quant-Dynamic build gives 22.897 GiB — only 1.103 GiB clear, which is why Meta targets that build at 32 GB rather than 24 GB. Our numbers reproduce the vendor's own two-tier split without being fitted to it.

The nearest third-party benchmark, and what it actually measured

Hardware Corner benchmarked Muse Glimmer 30B on an RTX 3090 across the context range (article published 2026-08-12, updated 2026-08-17). Their methodology line is specific: "All benchmarks were run on Ubuntu 24.04 with CUDA 13, NVIDIA driver 595.58.03, and llama.cpp build 153d324bc."

Three things keep this from being a number you can put on your own card and expect:

1. It is a different file. They say "Our tests use the Q4_K Medium GGUF quantization" and give its on-disk size as 14.78 GiB. Meta's K-Quant-17GB build — the one this recipe installs — is 15.606 GiB, and the closest match to 14.78 GiB in the ecosystem is unsloth/Muse-Glimmer-30B-UD-Q4_K_XL.gguf at 14.788 GiB.
2. There is no drafter and no projector in it. Their VRAM table tops out at 20 GB for a 256K context, which is the language model alone; 45.30 t/s at 4k is a no-speculation baseline, not the speculative figure this recipe's -md flag is for.
3. It is a slower board. That is the useful part: on the RTX 3090 Ti the same work should land above these numbers, by the margin measured in the section above.

Hardware Corner has no RTX 3090 Ti row for this model, and their RTX 3090 Ti page does not mention Muse Glimmer at all.

Community RTX 3090 reports

None of these are ours, none are first-party, and none is on a 3090 Ti. Each names its own configuration, and the configurations differ more than the numbers do — but they are the only public datapoints that run the drafter on a 24 GB Ampere card.

The closest match to this recipe is the first: "Just tried the gguf 17gb k-quant with llama.cpp" — the same file this recipe installs — reporting that "when using the dflash drafter, it does more than 60 tok/sec on my 3090" and that "vision (with the additional mmproj model) seems to work really well". That is one community datapoint from one person, not a measurement, and the same commenter was unimpressed with the model's output quality.

For the full benchmark data, see /check/muse-glimmer-30b/rtx-3090-ti.

Troubleshooting

unknown model architecture: 'muse-glimmer'

Your llama.cpp predates the architecture. Release b10344 and earlier are five commits behind the #26841 merge; upgrade to b10353 or newer. Users on the official GGUF repo hit this on b10344 and resolved it by moving to b10352 and b10358.

A community script from mid-August now 404s

The model's discussion threads carry several ready-made launch scripts written in the release week — for example a "Running on 3090 with llama.cpp" thread opened on 2026-08-13 with a full llama-server boot loop. Every one of them names the pre-rename files (-hff muse-glimmer-30B-kquant-17gb.gguf, -hfd …:dflash-kquant). Those names were deleted from the repo on 2026-08-18 and now return 404, so the script fails at download rather than at load. Substitute the canonical names from Installation; nothing else in those scripts needs to change.

Model load crashes with vector::_M_range_check

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

This is an out-of-memory condition wearing a bounds-check error's clothes. It is not specific to Muse Glimmer, to the DFlash drafter, or to any GGUF — it is what llama.cpp prints when every visible device reports zero free VRAM at the moment a model is loaded.

That is not how issue #26894 reads at first. It was filed against Meta's own GGUF and blamed the way that file encodes muse-glimmer.attention.sliding_window_pattern — an array of 52 booleans, where third-party conversions write a scalar. The reporter withdrew that diagnosis on 2026-08-13 — "My original diagnosis was wrong" — and named the real mechanism instead: "It has nothing to do with GGUF metadata, on either the target or the drafter side". He re-downloaded the exact file the issue was filed against, checked its SHA-256 against the Hub, and bound the drafter successfully 10 times out of 10, concluding that the published file is fine as it stands. The array-versus-scalar difference was a coincidence: he had tested the rewritten file later, on a card that happened to be free.

The mechanism, read out of src/llama-model.cpp on master at 17197474 rather than taken on trust:

• The default layer split weights each device by its free memory, then normalises by the sum of those weights.
• There is a zero guard, but it only catches a device reporting free == 0 and total == 0 — a device with nothing at all to report (#18577). A real GPU that is merely full reports free == 0 against a real total, and falls straight through it.
• Every entry in splits is then 0, so split_sum == 0, and splits[i] /= split_sum is 0/0 — NaN.
• std::upper_bound over NaNs finds no element (x < NaN is false), returns end(), and devices.at(n_devices()) throws. With one visible device that is __n (which is 1) >= this->size() (which is 1), verbatim.

It surfaced on the draft model in the original report for an ordinary reason: the ~20 GB target loads first and takes the card with it, so the drafter is simply the next allocation to ask a full GPU for room. The reporter reproduced it two ways — an LD_PRELOAD shim forcing free = 0, and naturally, by saturating the GPU from a separate process — including on an unrelated dense model with no speculative decoding at all.

What to do. Free the card, then re-run. Close other CUDA processes and check with nvidia-smi. On 24 GB this recipe's resident set is 20.198 GiB, leaving 3.802 GiB for the compute buffers, the CUDA context and your desktop — a compositor holding a gigabyte is enough to matter, so running headless or on the integrated display output is the durable fix. If you cannot free enough, three levers, cheapest first: quantise the KV cache with -fa on -ctk q8_0 -ctv q8_0, which roughly halves the KV allocation and is what the community 3090 script above uses; lower -c; or drop -md … --spec-type draft-dflash -ngld 99 and recover 1.568 GiB at the cost of the speculative speedup.

Do not rewrite the GGUF's metadata. The scalar-sliding_window_pattern workaround that circulated with the original report treats a symptom that was never the cause, and it leaves you running a file whose checksum no longer matches the Hub.

The issue is still open as of 2026-08-21, with no activity since the retraction — it now stands as a request for a clear "insufficient device memory" message in place of the out_of_range that sent the original investigation down the wrong path. Its title still describes the withdrawn diagnosis, which is why the wrong story is still the first thing search engines return.

The configuration itself is well attested. Every public report that actually loads a drafter file:

One genuinely separate change did land: PR #26900, merged 2026-08-12, swaps get_key_or_arr for get_arr in the drafter's own hparams read in src/models/dflash.cpp. Its author struck through the line claiming it closed #26894 — correctly, as it turns out.

Draft acceptance collapses when you raise -np

A community report, #27117 (open, no maintainer response), traces --spec-type draft-dflash under concurrent server slots: "draft acceptance collapses, and throughput goes into reverse (slower than no speculation)". The reporter's table shows per-slot acceptance falling from a healthy 0.51–0.66 at four concurrent requests to 0.06–0.49 at sixteen, with aggregate throughput below the no-speculation baseline, and shows --spec-draft-n-max 1 recovering it. It was measured on an AMD gfx1151 APU across two ROCm versions, not on CUDA — the reporter explicitly asks for a CUDA datapoint — so treat it as unconfirmed on this card. It is a second reason to keep -np 1 as written, beyond the KV-ring arithmetic in the budget.

reasoning_effort does nothing

Muse Glimmer's reasoning control is reasoning_strength, and the OpenAI-style reasoning_effort field is a different construct that does not map onto it — which means the standardised dropdown in llama.cpp's built-in web UI will appear to do nothing. A llama.cpp contributor states it plainly on #27023: "Muse Glimmer uses reasoning_strength". The reporter confirmed in the same thread that --chat-template-kwargs '{"reasoning_strength": "low"}' works from the CLI. Use that, or the Reasoning strength: <level> line in the system prompt.

VRAM climbs after the first image, and prefill gets slower

Community issue #26873 reports that the first use of the vision projector costs a further 1.1 GB beyond the mmproj weights already accounted for, and that prompt-processing throughput then falls by roughly a third and stays down even on a fresh context.

The prefill half of that has since been narrowed, and the narrowing lets a single-card reader off the hook. A second reporter pinned it down on 2026-08-15: "the regression only happens when pipeline parallelism is enabled", and the original reporter then confirmed on his own machine that forcing a no-op tensor override makes the slowdown disappear — self-confirmation rather than an independent third datapoint, and the thread has only those two participants. Both are consistent with llama.cpp's own gate, which is worth reading rather than trusting: in src/llama-context.cpp, pipeline_parallel is set only when model.n_devices() > 1 && … && !model.has_tensor_overrides(). On one GPU it can never be true — which is also why a tensor override "fixes" it on two. Every report in that thread is a multi-GPU configuration.

The extra allocation on first image use has not been attributed to the same cause, so keep budgeting for it: it eats into the 3.802 GiB margin above. If you serve images and run close to the edge, quantise the KV cache or lower -c.

DSpark is not DFlash, and it can be slower

Meta ships a DFlash drafter; llama.cpp also supports a DSpark type, which adds a Markov head and outranks DFlash in llama.cpp's sidecar auto-detection when both are present. They are not interchangeable. On a single RTX 3090 bartowski measured his own Q4_K_M build at "On my 3090 with Q4_K_M I get around 38 tok/s, with dspark it drops to ~28-32 depending on the task". Pass --spec-type draft-dflash explicitly rather than relying on auto-detection.

-hf picks the wrong file

llama.cpp's repo auto-selection looks for Q4_K_M or Q8_0 in the filename, then falls back to the first model-shaped GGUF in the repo. Since the 2026-08-12 republish the canonical name carries Q4_K_M, so a bare -hf meta-models/Muse-Glimmer-30B-GGUF now resolves to Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf by a real tag match rather than by ordering. Pin it anyway: --hf-file Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf.