Muse Glimmer 30B on Apple M4 Max: ExecuTorch Metal agent server with vision and DFlash

What You'll Build

An OpenAI-compatible agent server running Meta's Muse Glimmer 30B entirely on a MacBook Pro M4 Max, with image input and speculative decoding both switched on — and enough unified memory left over that you never touch a wired-limit override. The interesting part is the runtime: Meta measured this model on Apple Silicon with ExecuTorch, not with llama.cpp and not with MLX-LM, and ships pre-exported Metal artifacts for it.

Hardware data: Apple M4 Max (48 GB unified memory, 546 GB/s) · working set 21.316 GiB against a 36.000 GiB addressable pool · See benchmark data

ℹ️ Three Apple paths exist, and they are not interchangeable evidence. ExecuTorch, llama.cpp-Metal and MLX all run this model today, and this recipe verified each one separately. Meta's published Apple tok/s figures were produced only on ExecuTorch — the model card states "M4/M5 measurements were done using ExecuTorch, and RTX using llama.cpp." A number measured on ExecuTorch says nothing about MLX or llama.cpp throughput, so this recipe attaches every figure to the runtime that produced it and omits figures for the runtimes where this round found no measurement.

Requirements

Which M4 Max this is. Apple sells the M4 Max in two bins and the memory size identifies the bin: per Apple's MacBook Pro specifications, 36 GB unified memory goes with the 14-core-CPU M4 Max at 410 GB/s, and 48/64/128 GB only with the 16-core-CPU / 40-core-GPU part at 546 GB/s. A 48 GB M4 Max is therefore unambiguously the 546 GB/s bin. This matters because token generation is memory-bandwidth-bound.

The memory budget (derived, not measured)

Apple has no dedicated VRAM. Our catalogue stores full unified memory in the GPU row, but Metal's recommendedMaxWorkingSetSize caps what the GPU can actually address: exactly 3/4 on machines of 36 GB and above, so a 48 GB Mac addresses 36.000 GiB. Everything below is framed against that number, not against 48.

The KV cache is derived from the model's own config.json, not from prose. That file declares num_key_value_heads: 2, head_dim: 128, sliding_window: 2048, and a layer_types array containing 39 sliding_attention and 13 full_attention entries — the [Local, Local, Local, Global] repeating pattern from the model card, counted out of the artifact.

Per-token KV is 2 (K+V) × 2 KV heads × 128 head dim × 2 bytes = 1024 B per layer — 1 KiB, which is why a 131K context costs under 2 GiB here. The first-party GGUF card makes the same point in prose: "KV cache stays cheap".

Consequence: no sudo sysctl iogpu.wired_limit_mb raise is needed on this machine, and you should not add one. The full multimodal-plus-drafter stack sits 14.684 GiB inside the default pool.

Why 36 GB is the floor. A 32 GB Mac addresses exactly 2/3 of 32 GiB = 21.333 GiB, against this build's 21.316 GiB working set — a 0.017 GiB margin with macOS still to house. A 36 GB Mac addresses 27.000 GiB and clears it comfortably. And 36 GB is a real shipping configuration, not a rounded-off number: Apple's specification page lists 36 GB unified memory as a sold SKU of the 14-core-CPU M4 Max, alongside 48 GB, 64 GB and 128 GB on the 16-core-CPU part. That is where min_vram_gb: 36 comes from — it is a filter floor naming a machine you can actually buy, not this recipe's measured peak.

Installation

1. Download one ExecuTorch variant

The repo publishes 16 pre-exported variants and is 372 GB in total, so --include is mandatory. The metal builds are self-contained: unlike the CUDA builds they carry their weights inside the .pte, with no separate blob to fetch.

pip install huggingface_hub

REPO=meta-models/Muse-Glimmer-30B-ExecuTorch-PTE
VARIANT=muse-glimmer-k-quant-17G-128K-text-image-dflash-metal
LOCAL_DIR=./muse-glimmer-pte

hf download "$REPO" \
  --include "$VARIANT/*" \
  --include "tokenizer.json" \
  --include "tokenizer_config.json" \
  --include "chat_template.jinja" \
  --local-dir "$LOCAL_DIR"

The variant name is a fixed scheme — muse-glimmer-<quant>-128K-<modality>-<decoding>-<backend>. k-quant-17G is the ~4-bit build Meta targets at a 24 GB envelope, text-image adds the perception encoder, dflash bundles the speculative-decoding drafter, and metal is the Apple Silicon backend. The repo publishes no CPU build: the card states "There is no CPU variant."

Two byte-level cross-checks on that artifact, both of which you can repeat against the HF tree API. Subtracting the text-solo build from the text-image-solo build gives a perception encoder of 1,413,208,832 B; doing the same subtraction inside the dflash pair gives 1,413,208,704 B — the same figure to within 128 bytes of serialization padding, and within 1% of the 1,400,328,928 B mmproj-Muse-Glimmer-30B-Q4_K_M.gguf that the GGUF repo publishes separately. The drafter subtracts out at 1,695,925,376 B and 1,695,925,248 B by the same two routes.

2. Build the ExecuTorch runner

A download is not enough — the pre-exported artifact skips the export step, not the native runtime. Follow the ExecuTorch build-from-source guide to get a checkout configured, then build the model preset from the repo root. The Apple preset is muse-glimmer-mlx, and it is gated to Darwin in CMakePresets.json:

(cd examples/models/muse-glimmer && cmake --workflow --preset muse-glimmer-mlx)
pip install -r examples/llm_server/python/requirements.txt

Binaries land in cmake-out/examples/models/muse-glimmer/: solo_runner, dflash_runner, and muse_glimmer_worker. The server needs the last one.

This machine has no CUDA and none of the NVIDIA escape hatches apply: skip the muse-glimmer-cuda preset, skip every sm80+ptx variant in the repo, and ignore the FP8 / NVFP4 tensor-core paths the RTX 5090 numbers rely on — Apple Silicon has no such hardware. Skip pip install flash-attn as well; on Metal the attention path is MLX-native. Per PyTorch's announcement, "MLX RMSNorm, RoPE, SDPA, KV-cache updates, and quantized linear operations are lowered to MLX-native or custom Metal implementations."

Running

Start the OpenAI-compatible server from the ExecuTorch repository root. The Metal build takes no --data-path; text-image needs --pos-embed-path, and dflash needs no flag at all because the exported method contract is auto-detected.

VARIANT=muse-glimmer-k-quant-17G-128K-text-image-dflash-metal
LOCAL_DIR=./muse-glimmer-pte

python -m executorch.examples.models.muse_glimmer.serving.serve \
  --model-path "$LOCAL_DIR/$VARIANT/$VARIANT.pte" \
  --pos-embed-path "$LOCAL_DIR/$VARIANT/pos_embed.bin" \
  --tokenizer-path "$LOCAL_DIR/tokenizer.json" \
  --hf-tokenizer "$LOCAL_DIR" \
  --worker-bin cmake-out/examples/models/muse-glimmer/muse_glimmer_worker \
  --model-id muse-glimmer-30B \
  --tool-parser atem \
  --host 127.0.0.1 --port 8000

Smoke-test it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"muse-glimmer-30B","messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":32,"temperature":0}'

--tool-parser atem is what converts Muse Glimmer's native tool output into OpenAI tool_calls; the default is none, so an agent harness needs it passed explicitly. Reasoning depth is a system-prompt setting (Reasoning strength: low|medium|high|xhigh), not the OpenAI reasoning_effort parameter, which this server rejects with a structured 400.

One Apple-specific bonus worth knowing. The DFlash block length is exported dynamically and the accepted range differs by backend: the Metal build accepts --dflash-block-length in [2, 16] while the CUDA build accepts only [2, 4]. The Metal path is the one that can run the drafter at its full trained block size of 16.

⚠️ Two different DFlash mechanisms share a name — do not mix their flags. On this ExecuTorch path the drafter is baked into the exported .pte method contract: the text-image-dflash variant is the drafter, --artifact-mode defaults to auto and detects it, and --dflash-block-length is an ExecuTorch server flag. On the llama.cpp path below the drafter is a separate GGUF file bound with -md / -ngld and driven by llama.cpp's own draft-dflash speculative type. Neither set of flags exists in the other runtime.

Alternative: llama.cpp-Metal with the first-party GGUF

Mainline llama.cpp registers this architecture — LLM_ARCH_MUSE_GLIMMER is at line 74 of src/llama-arch.cpp, added by PR #26841 (merged 2026-08-10, commit 62bf73d) and first released in build b10353. Metal is on by default on macOS, so omit any GPU backend flag:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DBUILD_SHARED_LIBS=OFF
cmake --build build --config Release -j --target llama-cli llama-mtmd-cli llama-server

hf download meta-models/Muse-Glimmer-30B-GGUF \
    --local-dir Muse-Glimmer-30B-GGUF \
    --include "Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf" \
    --include "mmproj-Muse-Glimmer-30B-Q4_K_M.gguf" \
    --include "dflash-Muse-Glimmer-30B-Q4_K_M.gguf"

./build/bin/llama-server \
    -m       Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-KQuant-17GB-Q4_K_M.gguf \
    --mmproj Muse-Glimmer-30B-GGUF/mmproj-Muse-Glimmer-30B-Q4_K_M.gguf \
    -md      Muse-Glimmer-30B-GGUF/dflash-Muse-Glimmer-30B-Q4_K_M.gguf -ngld 99 \
    --spec-type draft-dflash \
    -a muse-glimmer-30B \
    -ngl 99 -c 131072 -np 1 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64

Image input on this path goes through llama-mtmd-cli rather than llama-cli, and vision is genuinely implemented rather than merely implied by the presence of an mmproj file: llama.cpp's multimodal layer carries a dedicated PROJECTOR_TYPE_MUSE_GLIMMER, a graph builder at tools/mtmd/models/muse-glimmer.cpp, and its own image preprocessor.

Speculative decoding is a first-class type here too, not an improvisation: COMMON_SPECULATIVE_TYPE_DRAFT_DFLASH has a dedicated implementation in common/speculative.cpp that reads the trained block size from the dflash.block_size GGUF metadata key, and LLM_ARCH_DFLASH is a registered architecture.

⚠️ -md and --spec-type are a pair, and either one alone silently does nothing. This is the easiest thing to get wrong here, so it is worth the paragraph.

common_speculative_init adds the DFlash implementation only when both conditions hold: the type bit is set and a draft context exists.

• -md without --spec-type — common_params_speculative::types defaults to { COMMON_SPECULATIVE_TYPE_NONE }, and -md never writes to it. NONE is enum value 0 and DRAFT_DFLASH is 4, so the default produces the bitmask 1, which matches nothing. The drafter file loads into memory and is never used.
• --spec-type without a draft model — the DFlash config is gated on params.draft.ctx_dft != nullptr, so with no draft path there is no context to bind and the implementation is skipped before it can assert.

Either way you get no error, no warning, and no speculation — the server starts and serves normally, just without the drafter. The only case where --spec-type can be omitted is -hfd <repo>, where llama.cpp inspects the draft repo's sidecars and infers the type; with a local file path there is nothing to infer from.

Both halves of this trap are attested in the wild. dmpr, in GGUF repo discussion #1, posts a command carrying --spec-type draft-dflash and three --spec-draft-* flags but no draft file of any kind — the first half. The mirror half is on the vendor: the GGUF card's own "add speculative decoding" snippet is -md … -ngld 99 with no --spec-type, and an earlier revision of this recipe copied it. If you are checking whether speculation is actually running, look for adding speculative implementation 'draft-dflash' in the startup log, or for a non-zero acceptance rate; a drafter that never bound reports neither.

⚠️ Use these filenames, not the ones you may have seen earlier. Meta republished the GGUFs on 12 Aug 2026 under canonical Q4_K names with a corrected embedded chat template; the card says the old one "Earlier GGUFs shipped a template that skipped both" normalisation steps and tells you to "re-download if you pulled before this fix". Since --jinja above uses the embedded template, this path is affected. The superseded muse-glimmer-30B-kquant-17gb.gguf / dflash-kquant.gguf are still in the repo, and the new files are only ~2,800 bytes larger, so no memory figure in this recipe moves. mmproj-Muse-Glimmer-30B-Q4_K_M.gguf is byte-identical to the old mmproj-kquant.gguf — same object, new name.

Weights on this path total 18.429 GiB (15.606 + 1.304 + 1.519), so with the same 1.701 GiB KV the working set is 20.130 GiB — comfortably inside the pool. If you disable the sliding-window crop with --swa-full the KV rises to 6.500 GiB and the total to 24.929 GiB, which still fits.

Alternative: MLX via mlx-vlm

mlx-lm does not implement this architecture — there is no muse_glimmer.py under mlx_lm/models/, and a community port is still open as a PR. mlx-vlm does, with a full muse_glimmer module including a real vision.py. It landed in v0.6.12: the module is absent at tag v0.6.10 and present at v0.6.12 (verified against a control file that resolves at both tags, so the 404 is absence rather than a missing tag).

pip install -U "mlx-vlm>=0.6.12"
mlx_vlm.generate --model mlx-community/Muse-Glimmer-30B-4bit \
  --image screenshot.png --prompt "What does this dialog ask the user to do?"

That build is clean by the test that matters: its 815 vision tensors resolve through model.safetensors.index.json into model-00003/4-of-00004.safetensors — the same shards as the language weights — rather than into a vendor-specific side artifact that stock loaders never open. Its 19,414,804,113 B of safetensors (18.081 GiB) plus KV leaves 16.218 GiB spare.

Alternative: Ollama, for a one-liner

ollama run muse-glimmer:30b

What that tag actually is, checked rather than assumed — and the check has three separate answers. Its registry manifest carries two content layers. The projector layer, 1,400,328,928 B at sha256:f48b452316f9…, is bit-identical to Meta's mmproj-kquant.gguf, whose HF LFS oid is the same hash. The model layer, 16,756,681,056 B at sha256:71b5c9c9abbc…, matches the byte count of Meta's muse-glimmer-30B-kquant-17gb.gguf exactly but not its digest (7e9b74b7c887…). Reading both headers, the metadata key order differs — Meta emits general.quantization_version immediately before the tokenizer block and Ollama's copy does not — which changes the hash at identical length. (general.name is Muse Glimmer Hf in both files, so it witnesses nothing.) So muse-glimmer:30b is Meta's K-Quant-17GB build repackaged, not a different quantisation and not the 19.65 GB K-Quant-Dynamic build; the 18 GB the library page displays is the sum of the two layers, not a third artifact's size. muse-glimmer:30b-q4_K_M-dflash is this same pair plus a third layer of 1,631,205,312 B at sha256:27d9a805fa29…, bit-identical to Meta's dflash-kquant.gguf — a superset of this tag, again not a different quant. Ignore the -nvfp4 and -mxfp8 tags and the 57–65 GB bf16 tags on a 48 GB machine. The tag actually aimed at this hardware is muse-glimmer:30b-mlx — 21 GB, 128K context, text and image — which fits inside the 36.000 GiB pool with room to spare; prefer it over :30b if you want Ollama to run the MLX path rather than the GGUF one.

⚠️ Ollama's blobs predate the chat-template fix. Its layer digests still match the superseded GGUFs, so this route inherits the template defect described in the llama.cpp section until Ollama repackages.

Ollama is not a way around the drafter bug below. Range-reading that model blob's GGUF header shows muse-glimmer.attention.sliding_window_pattern is still a 52-entry boolean array, the same trigger condition as Meta's own file.

Results

• Speed (ExecuTorch only): Meta reports 23.7 tok/s without speculation and 37.8 tok/s with the DFlash drafter on an Apple M4 Max, batch size 1, greedy decoding, using the K-Quant-17GB build. Two caveats that are part of the number. First, the card names "MacBook M4-Max" without a memory configuration, and the M4 Max ships in a 410 GB/s bin (36 GB only) and a 546 GB/s bin (48 GB and up) — so this figure may have been produced on the slower bin, in which case it is a floor for a 48 GB machine rather than a match. Second, it is an ExecuTorch measurement and does not transfer to the llama.cpp or MLX paths above.
• Speed (llama.cpp-Metal): this round turned up no M4 Max figure. The space searched, so you can judge the gap: every discussion thread on the canonical repo (49), on unsloth/Muse-Glimmer-30B-GGUF (14), on the first-party GGUF and ExecuTorch repos, and on the mlx-community build — each fetched and read individually rather than sampled — plus two web searches. The nearest independent datapoint that surfaced is a hands-on run on an M4 Pro 24 GB — a different, lower-bandwidth chip at 273 GB/s against this machine's 546 GB/s — which recorded a median 14.40 decode tok/s text-only on this exact quant. Because the generation gap runs toward the slower part, treat 14.40 as a pessimistic floor the M4 Max should beat, never as this card's number. Measured it yourself? Please contribute it so /check/muse-glimmer-30b/m4-max stops being empty.
• Speed (MLX): none found across the same space. Omitted rather than estimated — /contribute.
• Unified memory usage: 21.316 GiB working set for the ExecuTorch multimodal-plus-drafter build at the full 131,072-token context, against a 36.000 GiB addressable pool. Derived from published artifact bytes and the architecture's own config.json, not observed — see /check/muse-glimmer-30b/m4-max for live data as it lands.
• Quality notes: Meta puts K-Quant-17GB at 1.0% average degradation across 15 benchmarks and K-Quant-Dynamic at 0.2%. K-Quant-Dynamic's text-image-dflash-metal build is 23.85 GB and also fits this machine, so on a 48 GB M4 Max the higher-fidelity build is the free upgrade the 24 GB target hardware cannot take.

For the full benchmark data, see /check/muse-glimmer-30b/m4-max.

Troubleshooting

unknown model architecture: 'muse-glimmer'

Your llama.cpp predates the architecture. Releases b10344 and older do not register it and reject the file in about a fifth of a second, before allocating any memory — which is why changing context length or offload settings appears to do nothing. Use b10353 or newer, or confirm a source checkout with grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp. A source build from before that release is fine as long as it is after the merge: a user in GGUF repo discussion #1 answers this exact error with "Works fine if you build latest from source," and shows version: 10352.

In LM Studio the same failure surfaces as Engine protocol runtime llama-server exited before becoming healthy. exitCode=1, with the real message only in ~/.lmstudio/server-logs/. A community reporter who diagnosed it on an M5 Pro documents the fix in discussion #42: update the llama.cpp runtime to llama.cpp-mac-arm64-apple-metal-advsimd 2.28.2 or newer via lms runtime update. They note that "any Apple-silicon machine needs runtime 2.28.2 or newer to load Muse Glimmer at all."

Throughput collapses when the perception encoder is loaded

This is a memory-fit failure, and on an M4 Max it should not happen — but it is worth understanding, because it is the reason this recipe specifies a 48 GB machine. The M4 Pro 24 GB run cited above dropped from 14.40 tok/s text-only to 3.67 tok/s once the 1.40 GB projector was loaded. The author reports the measurement without diagnosing it; the arithmetic is ours, and it lands exactly on the boundary. A 24 GB Mac addresses 2/3 of 24 GiB = 16.000 GiB. The text weights alone are 15.606 GiB — inside the pool by 0.394 GiB. Adding the 1.304 GiB projector takes the total to 16.910 GiB, which overshoots by 0.910 GiB, before any KV cache. A 48 GB M4 Max addresses 36.000 GiB and never approaches that wall.

The same run also found the DFlash drafter reduced median decode speed by 27.6% when it had to be CPU-offloaded. That is a symptom of the same shortage, not a property of the drafter — but it is a reminder that the speculative path only pays when the drafter fits alongside everything else.

vector::_M_range_check when binding the DFlash drafter on llama.cpp — unresolved

Scope: the llama.cpp-Metal alternative only. The ExecuTorch path above is unaffected — this is a bug in llama.cpp's common/speculative.cpp drafter-bind path, not in the model, the weights, or anything Metal-specific. If you hit it, the recipe is not broken; one of its alternatives is.

llama.cpp issue #26894 (open at the time of writing) reports that binding the DFlash drafter to Meta's own GGUF crashes at load with:

vector::_M_range_check: __n (which is 1) >= this->size() (which is 1)

The reporter traces it to how the target GGUF encodes one metadata key, and notes that both targets load and generate fine without the drafter. I measured that key directly out of the file headers rather than taking the report on trust:

I re-read the headers again after Meta's 12 Aug republish: the renamed Q4_K builds carry 731 tensors, 32 metadata keys and the same 52-entry boolean array, so the rename changes nothing about this issue.

That matters for this recipe specifically: the issue was filed against kquant-dynamic, but the kquant-17gb file documented above carries the same array encoding, so the trigger condition is present in the command in this section — which reaches the drafter-bind path precisely because it passes --spec-type draft-dflash. (Drop that flag and you dodge this bug by never speculating at all, which is not a fix.) The Ollama tag carries the same encoding too.

Why this is written as unsettled rather than as a known bug with a known fix. Four things point in different directions, and none of them has closed it:

• Three independent users bind the drafter without hitting it — two of them with the projector loaded as well. On b10358, a community user binds the drafter to this recipe's exact kquant-17gb file and reports it "now working well for me after I built the latest release of llama.cpp", signing off 24G 7900 XTX ROCm (GGUF repo discussion #2). In discussion #34, one user on b10354 (RTX 5090, Windows) posts a startup log containing "adding speculative implementation 'draft-dflash'" — the bind succeeding — and a second, on b10358, runs kquant-dynamic, the very file the issue was filed against, with the same drafter.
• The one failing configuration is also the only multi-GPU one, and that fits better than a version story. The reporter runs two cards, an RTX 3090 and an RTX 5060 Ti, inside Docker; every report that states its hardware is a single GPU. A build-number explanation is tempting — the failure is on b10349 and the successes are later — but it cannot carry the weight, because pcuenca tried 62bf73d, which is the reporter's own build, and still could not reproduce. Same commit, opposite outcome, so something other than the version differs. Note this is an observation about the reports, not a diagnosis; nobody has isolated it.
• The -fa on hypothesis does not survive either. pcuenca — a llama.cpp contributor and the author of the Muse Glimmer support PR, though not a maintainer — posts a repro command that omits the -fa on the reporter used, which looked like the difference. But the b10354 report above passes --flash-attn on and binds cleanly, so that flag alone does not account for the split.
• A maintainer (CISC, MEMBER) replied only "Please test linked PR." — a request for testing, not a verdict.
• That linked PR, #26900 "model : disallow integer dflash sliding_window_pattern", is merged — but its author struck through the Nixes #26894 line in its own description, and GitHub's timeline records the reference with willCloseTarget: false. A merged PR that retracts its own fix claim is not a fix. The issue is still open.
• Every report either way — the one failure and all three successes — is CUDA or ROCm. Nobody has reported this on Metal in either direction, so its status on an Apple machine is genuinely unknown rather than known-good.

If you hit it — and on a current build you may well not. The reporter's validated workaround is to rewrite that one key to a scalar with gguf-py's copy_with_new_metadata, setting muse-glimmer.attention.sliding_window_pattern to UINT32 = 4. Note this cuts against the direction of the merged PR, so treat it as a local unblock rather than the settled answer. The zero-effort alternatives are to drop -md … -ngld 99 --spec-type draft-dflash and run llama.cpp without speculative decoding — everything else in that command works — or to use the ExecuTorch path, where the drafter is part of the exported artifact and this bind step does not exist.

The M5 tensor-API warning does not apply to this machine

If you read reports of the tensor API is not supported in this environment - disabling costing 2–3× on prompt processing, that is M5-only. The same discussion is explicit that "the tensor API targets the GPU Neural Accelerators introduced with the M5 generation" and that "llama.cpp explicitly disables it for pre-M5 devices because the hardware is not there". An M4 Max has no such accelerators to leave idle.

Nothing comes back on a tool call

Never treat <|eom|> as a stop token. The stop tokens are <|end_of_text|> (200001) and <|eot|> (200008); <|eom|> ends a single message and the turn continues past it. On a tool call the model emits a private reasoning message closed by <|eom|> first, so stopping there means the tool call is never generated. The bundled runtimes handle this; it bites only if you drive the raw runner from your own client.

Reasoning cannot be turned off, and long generations can silently return nothing

Scope: the llama.cpp paths. On the ExecuTorch server above, reasoning_effort is rejected with a structured 400 rather than ignored, and --reasoning off / --reasoning-budget do not exist at all.

The chat template opens the thinking channel unconditionally, so --reasoning off and "reasoning_effort": "none" have no effect — control the depth instead with reasoning_strength (low/medium/high/xhigh, default high), or hard-cap it with --reasoning-budget N. On llama-server, also remember that -c is divided across -np slots, so "a single request gets" -c / -np; check n_ctx_slot in the startup log. A generation that exhausts its slot context produces no answer and logs no error, which reads as a wrong result rather than a failure.

One request at a time

The ExecuTorch server is deliberately serial: --num-runners must be 1, every exported method is batch-1, and "One request executes at a time." Named sessions give isolation between conversations, not concurrency. Video is not supported on this path either — "Text and images only, one image per request, JPEG or PNG."

Report anything else via the submission form.