Qwen3.8-27B on Apple M4 Max: 4-bit MLX Vision-Language at the Full 262K Context

What You'll Build

A local vision-language assistant on an M4 Max: Qwen3.8-27B in MLX 4-bit, driven by mlx-vlm with the model's Multi-Token Prediction head loaded as a speculative drafter, answering questions about images and long documents from the command line or from an OpenAI-compatible server on localhost — with the model's entire native 262,144-token window available and no kernel tunable touched.

Hardware data: Apple M4 Max (48 GB unified memory) · 14.952 GiB of 4-bit weights + a 16.000 GiB KV cache at the full 262,144-token context, inside the 36.000 GiB Metal working set · See benchmark data

⚠️ This is not the 64 GB configuration, and the difference is the whole recipe. The 64 GB M2 Max page leads with the 8-bit build at the full context — 44.039 GiB of working set against a 48.000 GiB pool. A 48 GB Mac addresses 36.000 GiB, so that configuration is over by 8.039 GiB here and no wired-limit raise rescues it. Everything below is re-derived for this machine rather than carried across.

⚠️ Thinking is on by default and it is expensive. The model card states that "Qwen3.8 models operate in thinking mode by default", and its default reasoning effort is xhigh. Multiple users report reasoning traces consuming most of the context window before an answer appears. The ## Running section sets a reasoning budget from the first command rather than leaving this to be discovered.

Requirements

Unified memory is not VRAM, and on this machine the gap decides the recipe. An M4 Max ships 48 GB of unified memory, but Metal reports recommendedMaxWorkingSetSize = 38654.71 MB — 38,654,705,664 bytes, 36.000 GiB, exactly three quarters — as the pool the GPU may fill, and MLX, llama.cpp-Metal and Ollama all treat that as the ceiling. The figure is measured, not a percentage anyone asserted: it is logged verbatim in LM Studio bug-tracker #271, whose reporter describes the machine as "my MacBook Pro with 48 GiB memory" and whose same log independently corroborates the pool with using device Metal (Apple M4 Pro) - 36863 MiB free, and again in ollama #12012. Both logs are from an M4 Pro, not an M4 Max; the cap is an OS policy keyed on installed memory rather than on GPU core count, but that last step is our inference, not a first-party statement. Every budget below is framed against 36.000 GiB, never against 48.

The memory budget, derived

There is no benchmark for this pair — /check/qwen3-8-27b/m4-max returns verdict: unknown with zero benchmarks — so every figure here is arithmetic over byte counts you can re-read yourself.

That leaves 4.682 GiB of the 36.000 GiB pool for the vision tower's activations and everything else macOS is doing on the GPU.

The budget above is for one sequence in flight, and that is a choice you have to make explicitly. Weights and drafter are loaded once and shared, but the KV cache and the Gated DeltaNet state are per-sequence: every concurrent request adds its own 0.143 GiB of recurrent state plus 64 KiB per token of its own context. The one-shot CLI decodes a single sequence, so it is safe by construction; mlx_vlm.server is not, and the ## Running server command pins it with --max-num-seqs 1.

The KV cache is small for a 27B because only a quarter of the stack keeps one. The model card describes the layer stack as "Hidden Layout: 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))", and the released config.json agrees to the index: layer_types is 48 linear_attention and 16 full_attention, the full-attention layers sitting at positions 3, 7, 11 … 63, with num_key_value_heads: 4, head_dim: 256 and max_position_embeddings: 262144. So 16 of the 64 layers hold a KV cache, at 2 × 4 × 256 × 16 × 2 = 65,536 bytes per token — 64 KiB/token, giving 2.000 GiB at 32K, 8.000 GiB at 128K and 16.000 GiB at the full 262,144. A 27B with all 64 layers attending would spend four times that.

The other 48 layers hold a fixed-size recurrent state instead, which does not grow with context: mlx-vlm's implementation allocates a float32 recurrent state plus a bf16 convolution state per layer per sequence, 3,207,168 bytes each, 153,944,064 bytes across all 48. MLX also grows its KV cache in 256-token steps rather than pre-allocating, so 31.318 GiB is the ceiling at a saturated context, not what gets reserved at load.

What another weight bit costs you, in context

On a 64 GB Mac you can have the 8-bit build and the full window. Here you choose. Every row below is the same four addends with a different quant tier and a different KV setting, against the same 36.000 GiB pool:

The two total columns differ only where --kv-bits is set, and the reason is one skipped layer — see the pricing note below. The f16 rows are identical because nothing is being converted.

Two rows are worth reading twice. The 8-bit build at a 131,072-token window misses the pool by 41.7 MB — the kind of margin people hedge about, and here it is decided by arithmetic: the exact ceiling is 130,436 tokens of f16 KV, so 128K needs either a shorter window (98,304 tokens lands at 34.039 GiB) or the wired-limit raise below. And mxfp8 — the tier a commenter recommends for exactly this machine class, "If you have a 48gb mac then use an mxfp8 quant" in discussion #33 — does fit at 8-bit KV, with 0.664 GiB left for the vision tower and the window manager on the CLI path and 0.195 GiB on the server path. That is thinner than this recipe is willing to recommend.

How KV quantization is priced, and why it differs between the CLI and the server. mlx_vlm.generate and mlx_vlm.server both expose --kv-bits, --kv-group-size, --max-kv-size and --quantized-kv-start. The cache stores each group of --kv-group-size values as packed integers plus a scale and a bias in the cache dtype (2 bytes each at f16 or bf16), so at the default group size of 64 a quantized attention layer costs 2,176 bytes/token at 8 bits and 1,152 at 4 bits, against 4,096 unquantized.

The Gated DeltaNet layers never quantize on either path — make_cache() hands back an ArraysCache for each of them and a KVCache for each attention layer, and ArraysCache has no quantized form. What differs is how many attention layers get converted:

• One-shot CLI, default settings: all 16. maybe_quantize_kv_cache's uniform branch — the default --kv-quant-scheme uniform with an integer --kv-bits — converts every cache that offers a to_quantized method, with no exclusions. That is 34 KiB/token at 8 bits and 18 KiB/token at 4 bits, savings of 1.88× and 3.56×.
• Server and any batched path: 15 of 16. Caches are built through should_quantize_kv_layer(i, n), which for stacks deeper than two layers returns layer_idx < num_layers - 1 — its docstring says the last full-attention layer "is sensitive to quantization". Here n is 64, so slot 63 is skipped, and slot 63 is an attention layer (they sit at 3, 7, 11 … 63). One layer therefore stays f16: 36,736 bytes/token at 8 bits and 21,376 at 4 bits, savings of 1.78× and 3.07×.
• The CLI's two non-default branches — --kv-quant-scheme turboquant, which a fractional --kv-bits selects automatically, and any heterogeneous key/value setting — skip the last layer as well. Their packed layout is not the uniform one and is not sized here.

Quantization also does not start immediately: DEFAULT_QUANTIZED_KV_START is 5,000 tokens, so short conversations run at f16 whatever you pass.

What iogpu.wired_limit_mb actually buys

MLX does not guess at the pool: mlx_vlm/generate/common.py reads mx.device_info()["max_recommended_working_set_size"], calls mx.set_wired_limit() with it for the duration of generation, and prints a warning when the model's own bytes exceed 0.9 × that figure — 32.4 GiB on this machine. Two things follow. First, none of the tiers above trips that warning, because it counts weights only; the KV cache, which is what actually decides fit here, is not in the comparison. Second, the number MLX pins itself to is the OS-reported one, so raising it is a kernel-tunable question rather than an MLX flag:

# macOS Sonoma / Sequoia — value in MB, applies until reboot
sudo sysctl iogpu.wired_limit_mb=40960

At 40960 the GPU may hold 40 GiB and macOS keeps 8 GiB, at the outer end of the headroom the tunable's usual reference suggests leaving. That buys exactly two rows of the table above: 8-bit weights with a 131,072-token f16 cache (36.039 GiB) and 8-bit weights with an 8-bit cache at the full window (36.539 GiB). It does not buy the 64 GB recipe's configuration: 44.039 GiB would leave under 4 GiB for everything that is not the model, which is not a working Mac. If you want 8-bit at the full window, that is a 64 GB machine, not a tunable.

Installation

1. Install mlx-vlm

Metal is the native backend — there is no CUDA here, so there is no cu128 wheel index, no pip install flash-attn, no bitsandbytes, no --load-in-4bit, and no ExLlamaV2/GPTQ/AWQ step to carry over from an NVIDIA guide. Qwen's own Qwen3.8-27B-FP8 checkpoint is likewise not the Apple path; FP8 tensor cores are Hopper/Blackwell hardware.

python3 -m venv ~/venvs/qwen38
source ~/venvs/qwen38/bin/activate
pip install -U mlx-vlm

mlx-vlm implements this architecture natively: mlx_vlm/models/qwen3_5/ carries the Gated DeltaNet kernels and a vision tower, and mlx_vlm/speculative/drafters/qwen3_5_mtp/ carries the MTP drafter. Both are present in the released v0.6.13 tag, so no source build is needed. There is an open pull request titled "Add Qwen3.8-27B support" — read it for what it says it does, which is blockwise FP8 loading, ModelOpt NVFP4 adaptation and MTP splitting on top of support that already ships. It is not the gate on running this model.

2. Download the weights and the MTP drafter

pip install -U huggingface_hub
hf download mlx-community/Qwen3.8-27B-4bit     --local-dir ~/models/qwen3.8-27b-4bit
hf download mlx-community/Qwen3.8-27B-MTP-4bit --local-dir ~/models/qwen3.8-27b-mtp-4bit

Both builds were produced with mlx_vlm.convert — the 4-bit card records that it was converted "using mlx-vlm version 0.6.8" — which matters for a vision-language model, because a build converted with the text-only mlx_lm.convert silently drops the vision tower. This one does not: its model.safetensors.index.json lists 2,180 tensors of which 333 are vision_tower.*, and all 333 resolve to model-00001-of-00003.safetensors, the same shard as the language weights. Nothing lives in a side artifact that stock loaders would never open. The quantization is declared in config.json as affine, 4 bits, group size 64.

The drafter is not a model. Its card is explicit: "This is not a standalone model. Use it as the draft model alongside a compatible Qwen3.8 27B target checkpoint."

What you can and cannot check about that pairing. The drafter card declares its provenance under Model details — Source revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 — and that is the only side of the pair that declares one. The 4-bit target card is 632 bytes end to end and names only base_model: Qwen/Qwen3.8-27B; its config.json and its HuggingFace API cardData carry no revision field either. So the two artifacts are not checkable against each other from the outside: the pairing rests on the drafter's own declaration plus the fact that both were converted by mlx-community with mlx_vlm.convert. Treat it as an assurance, not a verification, and if you have reason to doubt it, the failure mode is visible at runtime — a mismatched drafter's proposals get rejected by the target and speculation costs throughput instead of adding it.

Running

One-shot, with an image

mlx_vlm.generate \
  --model ~/models/qwen3.8-27b-4bit \
  --draft-model ~/models/qwen3.8-27b-mtp-4bit \
  --image ~/Pictures/diagram.png \
  --prompt "Read the axis labels and summarise what this chart claims." \
  --max-tokens 1024 \
  --temperature 1.0 \
  --gen-kwargs '{"top_p": 0.95, "top_k": 20, "min_p": 0.0}' \
  --enable-thinking --thinking-budget 2048

The entry point is mlx_vlm.generate, with the dot. pyproject.toml's [project.scripts] declares five console scripts and every one is dotted — mlx_vlm.chat, mlx_vlm.chat_ui, mlx_vlm.convert, mlx_vlm.generate, mlx_vlm.server. There is no bare mlx_vlm executable, so a dotless mlx_vlm generate … fails with "command not found". Worth knowing because the drafter's own model card prints the dotless form; python -m mlx_vlm.generate … also works and is what the 4-bit card uses.

The sampler values are the model card's, but only --temperature is a flag. The card recommends temperature=1.0, top_p=0.95, top_k=20, min_p=0.0 for thinking mode. mlx_vlm.generate exposes --temperature directly; it has no --top-p or --top-k — there are 71 arguments in mlx_vlm/generate/dispatch.py and neither spelling is among them. The supported route for the rest is --gen-kwargs, a JSON blob that is merged into the generation kwargs. Setting them any other way silently leaves the model on its defaults.

--draft-kind is not needed — the drafter's card notes that "--draft-kind mtp is detected automatically from model_type." --thinking-budget 2048 caps the reasoning block; when the budget is exceeded, mlx-vlm forces </think> and the model moves to the answer.

First run streams ~17 GB off disk into the unified pool; subsequent runs start in seconds while the file cache is warm.

As a local OpenAI-compatible server

mlx_vlm.server \
  --model ~/models/qwen3.8-27b-4bit \
  --draft-model ~/models/qwen3.8-27b-mtp-4bit \
  --host 127.0.0.1 --port 8080 \
  --max-num-seqs 1 \
  --enable-thinking --thinking-budget 2048

Pass --host 127.0.0.1 deliberately. mlx_vlm.server defaults to DEFAULT_SERVER_HOST = "0.0.0.0" in mlx_vlm/server/cli.py, which exposes an unauthenticated endpoint to every device on your network. Note also that mlx-vlm's README states "On the server, thinking mode is disabled by default." — the opposite of the CLI and of the model's own default, so a server started without --enable-thinking is answering in non-thinking mode whatever the model card says.

And pass --max-num-seqs 1 deliberately too. The server runs a continuous batch whose size is unbounded by default: --max-num-seqs is type=int, default=None, and its own help text says "Default: unbounded." On a machine with headroom that is a throughput feature; here it is the difference between the 31.318 GiB budget above and running out of pool, because each admitted request brings its own KV cache and its own recurrent state. Two saturated 262,144-token sequences would want 15.175 GiB shared plus 2 × 16.143 GiB — past the pool before the third request arrives. Raise it only after measuring what your actual context lengths cost.

The server takes the same --kv-bits, --kv-group-size, --max-kv-size and --quantized-kv-start flags as the CLI, which is where the ladder table becomes operational: --max-kv-size is the blunt instrument if you would rather cap the window than quantize the cache. Requests take image_url on /v1/chat/completions and can override the server defaults per request with enable_thinking and thinking_budget.

The turnkey alternative: Ollama

ollama run qwen3.8:27b-mlx

Ollama has run on MLX on Apple silicon since 0.19. That release's getting-started block — written around the preview model it shipped with rather than as a general rule for the MLX runtime — asks that you "Please make sure you have a Mac with more than 32GB of unified memory.", which a 48 GB M4 Max satisfies. The 27b-mlx tag is a 16.927 GiB MLX pack (18,174,721,596 bytes across 1,209 layers on the registry) whose config declares "file_type":"nvfp4" and "capabilities":["completion","vision","tools","thinking"]. Do not read that string as a CUDA instruction: nothing NVIDIA-specific is being installed or required here. It names a 4-bit weight format that NVIDIA's ModelOpt produced at packaging time, and Ollama executes the pack on Metal through MLX — there is no NVIDIA hardware, no NVIDIA runtime and no tensor-core path involved on a Mac. It is the fastest turnkey option and the lowest-fidelity one of the tiers on this page; read the Ollama warning in ## Troubleshooting before using the default 27b tag.

Results

• Speed: omitted for this machine — no measurement of this model on an M4 Max exists on any surface searched, and none is invented here. The surfaces: all 103 discussions on the model's HuggingFace repo, fetched individually and grepped across all 317 comment bodies, which contain zero mentions of any "M-something Max" machine; the discussion tabs of the mlx-community 4-bit, 8-bit, MTP-4bit and mxfp8 repositories and of all three lmstudio-community MLX builds; GitHub issue search over ggml-org/llama.cpp and Blaizzy/mlx-vlm for this model together with each chip name (0 results, against a control query for the model alone that returns 30); and two Apple-silicon benchmark aggregators, neither of which lists this model. The one Apple datapoint that exists anywhere is community-reported, single-source, and on a different tier of this same generation: in HuggingFace discussion #80 a user reports, on an M4 Pro whose bandwidth he states as ≈ 270 GB/s, 11.78 tok/s for Ollama's GGUF q4_K_M with no speculation against 20.26–38.36 tok/s for the MLX build. Do not read those rates onto an M4 Max — the quant tiers differ and this chip has roughly twice his memory bandwidth, so his figures are at best a floor, not an estimate. What does carry is the ratio: MLX beat GGUF by a wide margin on the same machine, which is the reason this recipe leads MLX. If you measure this pair, please contribute the numbers so /check/qwen3-8-27b/m4-max stops being empty.
• What the bandwidth allows: token generation is bandwidth-bound, so an upper bound follows from the bytes streamed per token. Apple's tech specs for this generation of MacBook Pro list the M4 Max in two bins — a 14-core-CPU/32-core-GPU part at 410GB/s, and "M4 Max with 16-core CPU and 40-core GPU (546GB/s memory bandwidth)" — and the same page sells 48 GB only on the 16-core-CPU configurations (36 GB is the sole option on the 14-core M4 Max). So a 48 GB M4 Max is unambiguously the 546 GB/s part, and this recipe assumes that bin; a 36 GB M4 Max is the slower one and none of the ceilings below transfer to it. Dividing 546 GB/s by the weights streamed per token caps plain decoding near 34.0 tok/s for this recipe's 16.05 GB of 4-bit weights, near 24.0 tok/s for the 22.78 GB 6-bit build and near 18.5 tok/s for the 29.50 GB 8-bit build. Speculative decoding is what makes the tier you pick comfortable rather than merely possible: accepted drafts emit several tokens per weight read, which is how the M4 Pro report above exceeds its own machine's ceiling.
• Memory usage: 31.318 GiB derived for the 4-bit build at a saturated 262,144-token context, against a 36.000 GiB addressable pool — see the budget table above and /check/qwen3-8-27b/m4-max for live data as it lands.
• What leading 4-bit costs, stated plainly: one commenter, Albert-Liu-Hongkang, ties a multimodal regression to this quant tier: in discussion #69 he writes that the "multi modal ability is inferior to 3.6 27b when comes to 4 bit quant", adding that he intends to retest at 8-bit and 16-bit. That thread has three participants and he is the only one who names a quantisation at all, and the same person opened a similar thread on the unsloth GGUF repo — so it is one person's impression twice, not two reports. On a 64 GB Mac the answer to that is "then use 8-bit"; on this machine the answer is the ladder table, and the honest ordering is: if you want more weight bits, spend context bits, not the other way around. The 6-bit row is the natural next step and its cost is the drafter — lmstudio-community/Qwen3.8-27B-MLX-6bit carries 2,180 tensors with the same 333 vision_tower.* entries and zero mtp.* tensors, and no 6-bit MTP drafter is published under mlx-community (its MTP set is exactly 4bit, 8bit, bf16, mxfp4, mxfp8, nvfp4), so speculation is off unless you leave the org — the same enumeration does surface a third-party one, lukaskremla/Qwen3.8-27B-MTP-6bit-MLX at 0.321 GiB, which we have not tested against this target.
• Vision quality note: the vision projector is the same one the Qwen3-VL family uses. The released config.json sets vision_config.deepstack_visual_indexes to an empty list, and the shipped mmproj GGUF's clip.vision.is_deepstack_layers is a 27-entry boolean array that is all false — the same fact from two artifacts. There is no deepstack feature injection in this model, so any advice inherited from a deepstack-enabled Qwen3-VL recipe does not apply.

For the full benchmark data, see /check/qwen3-8-27b/m4-max.

Troubleshooting

The model thinks forever and burns the whole context

This is the single most reported problem with this release, and it is a configuration issue rather than a fault. The model card states that "Qwen3.8 models operate in thinking mode by default" and that the default reasoning effort is xhigh. Users report reasoning dominating the window: one describes 80% of context going to thinking on coding work, and another that forcing medium cut consumption dramatically (unsloth GGUF discussion #45).

Fix it at the runtime with --thinking-budget, as in the commands above, which forces the closing tag when the budget is spent. Note the model card's own caveat before you set it too low: "In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time."

The 8-bit build loads and then the machine crawls

You have exceeded the pool. The 8-bit weights are 27.475 GiB, which fits 36.000 GiB on its own and will load happily; what does not fit is the cache that grows behind it. Because MLX grows the KV cache in 256-token steps, the failure arrives mid-conversation rather than at load — the exact shape of "it worked at first". The ceiling is 130,436 tokens of f16 KV at that tier, so either quantize the cache (--kv-bits 4 at the full window, 32.539 GiB) or cap the window (--max-kv-size 98304, 34.039 GiB), or raise the wired limit as described above and accept 8 GiB of headroom for macOS.

A related report worth reading correctly: in discussion #33 a user states "i have 48G ram (mac)" and then that he tested a third-party build "by using omlx and the speed is 4.9 tok/s". It is tempting to file that as an over-the-pool symptom, and it may be — but the build he names is 19.831 GiB on the tree API, which is comfortably inside 36.000 GiB, he does not state his chip, his window, or his tool's version, and no one replied. Treat it as a reason to use enumerated builds and a stated budget, not as a measurement of anything.

Speculative decoding on this chip

Two upstream reports touch the Metal kernels this model's speculative path uses, and both name M4 Max hardware explicitly — in opposite directions.

The good one: mlx-vlm's own source records that its ragged-decode attention kernel wants 1,024 threads per threadgroup at D_SIZE == 256 — which is this model's head_dim — and that an M2 Ultra compiles that pipeline with a ceiling of 896 threads, raising ValueError: Thread group size (1024) is greater than the maximum allowed threads per threadgroup (896). The comment then states that the same size "launches fine on applegpu_g16s (M4 Max)". PR #1799, merged 2026-08-05 and so present in the v0.6.13 tag, made this a non-issue anywhere: the launch is probed once per pipeline, cached, and a GPU that rejects it degrades two-pass → one-pass → portable fallback.

The one to know about: issue #1800 reports that _target_verify_quantized_linear — the hand-written kernel that makes speculative verification bit-exact with ordinary decoding — loses that parity on applegpu_g16s, the GPU the reporter's M4 Max identifies as, while passing on an M2 Ultra and on the project's CI. He isolated which side moved: the custom kernel is identical across both GPUs and mx.quantized_matmul is not, and against a float64 oracle MLX's own path is the more accurate one. So the practical reading is not "speculation is broken on M4" but "on this GPU family the drafted path and the plain path may not produce byte-identical greedy output". It was filed as a portability report and closed as completed the same day, with no explanatory comment beyond an automated tracker linkback. If you need reproducible greedy output, drop --draft-model and compare.

ollama run qwen3.8:27b is much slower than expected on some prompts

The default Ollama tag ships speculative decoding on, and on the GGUF path that can be a large net loss. Verifiable from the registry: the manifests for qwen3.8:27b, qwen3.8:latest and qwen3.8:27b-mtp-q4_K_M are byte-identical — same 16,810,714,464-byte model blob, same 931,146,016-byte projector, same params digest — and that params blob reads {"draft_num_predict":4,"min_p":0,"presence_penalty":0,"repeat_penalty":1,"temperature":1,"top_k":20,"top_p":0.95}. The 27b-q4_K_M tag points at the same weights with a params blob that omits draft_num_predict. So the default tag is the MTP tag.

The M4 Pro report above measured that difference directly on his hardware — 5.14 tok/s on unpredictable text against 11.78 tok/s with speculation off — and noted that "on this machine the fastest path isn't GGUF at all". If you want Ollama's GGUF path, pull qwen3.8:27b-q4_K_M explicitly rather than the default. If you want speed on Apple, use 27b-mlx or the mlx-vlm commands above.

llama.cpp-Metal instead of MLX, and what it costs in memory

The GGUF path works — general.architecture = qwen35 is implemented in mainline llama.cpp (src/llama-arch.cpp), and the mmproj gives it the image path. Every source claim below is pinned to commit 0d9ceae so it stays checkable.

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf \
  -c 32768 --parallel 1

Three things differ from MLX and all three cost memory on a 36.000 GiB pool.

--parallel 1 is not redundant on the server. llama-server allocates the Gated DeltaNet state once per sequence slot, and its slot count defaults to 4, not 1. The common_params struct default is indeed n_parallel = 1 (common/common.h:447), but the parser overrides it per example before any argument is read — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default } (common/arg.cpp:1400) — and the server then resolves auto to 4 slots with a unified KV cache, logging n_parallel is set to auto, using n_parallel = 4 and kv_unified = true (tools/server/server.cpp:151). So a bare llama-server pays 0.585 GiB of recurrent state where --parallel 1 pays 0.146 GiB. The attention KV cache is unified and is not multiplied. Note the asymmetry: llama-mtmd-cli and llama-cli are not in that override branch and do default to one sequence.

--cache-type-k/-v cannot shrink the recurrent half. They set the attention cache's types only. For this architecture llama.cpp builds a hybrid memory whose recurrent types are passed as literal GGML_TYPE_F32 (src/llama-model.cpp:2331), so the 48 Gated DeltaNet layers stay float32 whatever you quantise the KV cache to — 156,893,184 bytes per sequence here against MLX's 153,944,064, the difference being the convolution state's dtype.

-c is pre-allocated, unlike MLX's 256-token growth, so -c 262144 reserves the full 16.000 GiB up front rather than as the conversation grows. That is the constraint that bites on this machine: ggml-org's Q4_K_M is 17.671 GiB and the BF16 projector 0.867 GiB, so the full window with one slot lands at 34.684 GiB — inside 36.000 GiB, with 1.316 GiB to spare and nothing left over for a second slot's worth of anything. The Q8_0 GGUF (26.632 GiB) plus the projector only reaches a 131,072-token window at 35.645 GiB.

Pick the build deliberately, because neither the weights nor the projector are one number across publishers: ggml-org's Q4_K_M is 18,973,870,432 bytes, unsloth's, bartowski's and lmstudio-community's differ, and the projectors differ too — 931,145,888 bytes for ggml-org's BF16, 927,607,488 for unsloth's F16, 927,607,008 for bartowski's f16. Use --spec-type draft-mtp with a drafter only when you want speculation; a baked-in MTP head you never invoke is weight you paid for.

Video input: it works, but not everywhere you would expect

The model is genuinely a video model — the card promises "Native support for image and video understanding, from STEM diagrams and documents to hour-scale videos." — but runtime coverage is uneven, and it is worth knowing which door is open before you try one.

• mlx-vlm handles it: the qwen3_5 implementation accepts pixel_values_videos and video_grid_thw, and the MLX repos ship a video_preprocessor_config.json declaring Qwen3VLVideoProcessor.
• llama.cpp implements Qwen-VL-style temporal frame merging for this projector type, and both entry points work — llama-mtmd-cli takes an interactive /video <path>, and llama-server accepts a content[] part of "type": "input_video". But the capability test is not what it looks like: mtmd_helper_support_video is #ifdef MTMD_VIDEO around a plain vision-projector check (return mtmd_support_vision(ctx);, else return false;), so a build compiled without that flag, or a run without --mmproj, both fail at the gate with the same message — while a missing ffmpeg/ffprobe leaves the gate open and fails later at decode with ffprobe failed for '<path>' (is ffprobe in PATH?) (mtmd-helper.cpp:844). There are zero references to either binary in mtmd.cpp and twenty in mtmd-helper.cpp; they are resolved lazily when a clip is actually opened.
• Ollama's 27b-mlx tag lists its capabilities as completion, vision, tools, thinking. Video is not among them.
• Frame-rate control over the API is first-party-documented for vLLM only.

No first-party quantised weights exist for this model

There is no Qwen-published GGUF or MLX build. Enumerating the Qwen org through the HuggingFace API returns exactly four repositories matching Qwen3.8: the base model, Qwen3.8-27B-FP8, and the two Qwen3.8-2.4T-A95B variants — none of them a GGUF or MLX conversion. Qwen does publish GGUF and MLX for other families, so this is a per-model gap rather than an org policy, and a first-party build may appear later. Everything this recipe installs is a third-party conversion of Apache-2.0 weights; mlx-community is the publisher of both files, and lmstudio-community and ggml-org are named above as alternatives.

Nothing else reproduced

Beyond the above, no M4 Max-specific failure surfaced in the model's 103 HuggingFace discussions, the discussion tabs of the MLX and GGUF distributor repositories, or the llama.cpp and mlx-vlm issue trackers. Report problems via the submission form.