Qwen3.8-27B on Apple M2 Max: 8-bit MLX Vision-Language with MTP Speculative Decoding

What You'll Build

A local vision-language assistant on an M2 Max: Qwen3.8-27B in MLX 8-bit, driven by mlx-vlm with the model's Multi-Token Prediction head loaded as a speculative drafter, answering questions about images and long documents from the command line or from an OpenAI-compatible server on localhost.

Hardware data: Apple M2 Max (64 GB unified memory) · 27.475 GiB of 8-bit weights + a 16.000 GiB KV cache at the full 262,144-token context, inside the 48.000 GiB Metal working set · See benchmark data

⚠️ Thinking is on by default and it is expensive. The model card states that "Qwen3.8 models operate in thinking mode by default", and its default reasoning effort is xhigh. Multiple users report reasoning traces consuming most of the context window before an answer appears. The ## Running section sets a reasoning budget from the first command rather than leaving this to be discovered.

Requirements

Unified memory is not VRAM. An M2 Max ships 64 GB of unified memory, but Metal reports only recommendedMaxWorkingSetSize = 51539.61 MB — 48.000 GiB, exactly three quarters — as the pool the GPU may fill, and MLX, llama.cpp-Metal and Ollama all treat that as the ceiling. The figure is a measured one, logged on an Apple M2 Max with 64 GB in llama.cpp issue #9112 and reproduced on M1 Max hardware in llama.cpp #6216 and whisper.cpp #1862. Every budget below is framed against 48.000 GiB, never against 64.

The memory budget, derived

There is no benchmark for this pair — /check/qwen3-8-27b/m2-max returns verdict: unknown with zero benchmarks — so every figure here is arithmetic over byte counts you can re-read yourself.

The budget above is for one sequence in flight, and that is a choice you have to make explicitly. Weights and drafter are loaded once and shared, but the KV cache and the Gated DeltaNet state are per-sequence: every concurrent request adds its own 0.143 GiB of recurrent state plus 64 KiB per token of its own context. Two requests each holding a saturated 262,144-token context would want 27.895 GiB shared + 2 × 16.143 GiB = 60.181 GiB, well past the pool. The one-shot CLI decodes a single sequence, so it is safe by construction; mlx_vlm.server is not, and the ## Running server command pins it with --max-num-seqs 1.

That leaves 3.961 GiB of the 48.000 GiB pool. The headline is that this is the one Apple configuration in the catalogue where the 8-bit build and the model's entire native context fit without touching iogpu.wired_limit_mb.

The KV cache is small for a 27B because only a quarter of the stack keeps one. The model card describes the layer stack as "Hidden Layout: 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))", and the shipped GGUF header agrees with the config: qwen35.block_count = 64, qwen35.full_attention_interval = 4, qwen35.attention.head_count_kv = 4, qwen35.attention.key_length = 256. So 16 of the 64 layers hold a KV cache, at 2 × 4 × 256 × 16 × 2 = 65,536 bytes per token — 64 KiB/token, giving 2.000 GiB at 32K, 8.000 GiB at 128K and 16.000 GiB at the full 262,144. A 27B with all 64 layers attending would spend four times that.

The other 48 layers hold a fixed-size recurrent state instead, which does not grow with context: mlx-vlm's implementation allocates (B, 48, 128, 128) in float32 per layer plus a (B, 3, 10240) convolution state, i.e. 3,207,168 bytes per layer per sequence, 153,944,064 bytes across all 48. MLX also grows its KV cache in 256-token steps rather than pre-allocating, so 44.039 GiB is the ceiling at a saturated context, not what gets reserved at load.

The two configurations that do not work, and why: the bf16 build is 50.96 GiB of weights alone, over the 48.000 GiB pool before a single KV byte — it needs a wired-limit raise and still leaves almost nothing for context, so it is not documented here. And a 36 GB Mac addresses 27.000 GiB, which is 0.475 GiB short of the 8-bit weights; that tier needs the 4-bit build, covered under ## Results.

Installation

1. Install mlx-vlm

Metal is the native backend — there is no CUDA here, so there is no cu128 wheel index, no pip install flash-attn, no bitsandbytes, no --load-in-4bit, and no ExLlamaV2/GPTQ/AWQ step to carry over from an NVIDIA guide. Qwen's own Qwen3.8-27B-FP8 checkpoint is likewise not the Apple path; FP8 tensor cores are Hopper/Blackwell hardware.

python3 -m venv ~/venvs/qwen38
source ~/venvs/qwen38/bin/activate
pip install -U mlx-vlm

mlx-vlm implements this architecture natively: mlx_vlm/models/qwen3_5/ carries the Gated DeltaNet kernels and a vision tower that subclasses the Qwen3-VL one, and mlx_vlm/speculative/drafters/qwen3_5_mtp/ carries the MTP drafter. Both are present in the released v0.6.13 tag, so no source build is needed.

2. Download the weights and the MTP drafter

pip install -U huggingface_hub
hf download mlx-community/Qwen3.8-27B-8bit     --local-dir ~/models/qwen3.8-27b-8bit
hf download mlx-community/Qwen3.8-27B-MTP-8bit --local-dir ~/models/qwen3.8-27b-mtp-8bit

Both builds were produced with mlx_vlm.convert — the 8-bit card records that it was converted "using mlx-vlm version 0.6.8" — which matters for a vision-language model, because a build converted with the text-only mlx_lm.convert silently drops the vision tower. This one does not: its model.safetensors.index.json lists 2,180 tensors of which 333 are vision_tower.*, and all 333 resolve to model-00001-of-00006.safetensors, the same shard as the language weights. Nothing lives in a side artifact that stock loaders would never open.

The drafter is not a model. Its card is explicit: "This is not a standalone model. Use it as the draft model alongside a compatible Qwen3.8 27B target checkpoint."

Running

One-shot, with an image

mlx_vlm.generate \
  --model ~/models/qwen3.8-27b-8bit \
  --draft-model ~/models/qwen3.8-27b-mtp-8bit \
  --image ~/Pictures/diagram.png \
  --prompt "Read the axis labels and summarise what this chart claims." \
  --max-tokens 1024 \
  --temperature 1.0 \
  --gen-kwargs '{"top_p": 0.95, "top_k": 20, "min_p": 0.0}' \
  --enable-thinking --thinking-budget 2048

The entry point is mlx_vlm.generate, with the dot. pyproject.toml's [project.scripts] declares five console scripts and every one is dotted — mlx_vlm.chat, mlx_vlm.chat_ui, mlx_vlm.convert, mlx_vlm.generate, mlx_vlm.server. There is no bare mlx_vlm executable, so a dotless mlx_vlm generate … fails with "command not found". Worth knowing because the drafter's own model card prints the dotless form; python -m mlx_vlm.generate … also works and is what the 8-bit card uses.

The sampler values are the model card's, but only --temperature is a flag. The card recommends temperature=1.0, top_p=0.95, top_k=20, min_p=0.0 for thinking mode. mlx_vlm.generate exposes --temperature directly; it has no --top-p or --top-k — there are 71 arguments in mlx_vlm/generate/dispatch.py and neither spelling is among them. The supported route for the rest is --gen-kwargs, a JSON blob that dispatch.py merges into the generation kwargs and hands to generate_step, whose signature does take top_p, min_p and top_k and passes them to make_sampler. Setting them any other way silently leaves the model on its defaults.

--draft-kind is not needed — the drafter's card notes that "--draft-kind mtp is detected automatically from model_type." --thinking-budget 2048 caps the reasoning block; when the budget is exceeded, mlx-vlm forces </think> and the model moves to the answer.

First run streams ~30 GB off disk into the unified pool; subsequent runs start in seconds while the file cache is warm.

As a local OpenAI-compatible server

mlx_vlm.server \
  --model ~/models/qwen3.8-27b-8bit \
  --draft-model ~/models/qwen3.8-27b-mtp-8bit \
  --host 127.0.0.1 --port 8080 \
  --max-num-seqs 1 \
  --enable-thinking --thinking-budget 2048

Pass --host 127.0.0.1 deliberately. mlx_vlm.server defaults to DEFAULT_SERVER_HOST = "0.0.0.0" in mlx_vlm/server/cli.py, which exposes an unauthenticated endpoint to every device on your network. Note also that mlx-vlm's README states "On the server, thinking mode is disabled by default." — the opposite of the CLI and of the model's own default, so a server started without --enable-thinking is answering in non-thinking mode whatever the model card says.

And pass --max-num-seqs 1 deliberately too. The server runs a continuous batch whose size is unbounded by default: --max-num-seqs is type=int, default=None in mlx_vlm/server/cli.py, its own help text says "Default: unbounded.", and get_max_num_seqs() in mlx_vlm/server/generation.py returns None when the variable is unset, which the scheduler reads as unlimited admission capacity. On a card with headroom that is a throughput feature; here it is the difference between the 44.039 GiB budget above and running out of pool, because each admitted request brings its own KV cache and its own recurrent state. Raise it only after measuring what your actual context lengths cost.

Requests take image_url on /v1/chat/completions and can override the server defaults per request with enable_thinking and thinking_budget.

The turnkey alternative: Ollama

ollama run qwen3.8:27b-mlx

Ollama has run on MLX on Apple silicon since 0.19; its announcement asks that you "Please make sure you have a Mac with more than 32GB of unified memory.", which a 64 GB M2 Max satisfies. The 27b-mlx tag is an 18.17 GB MLX pack whose config declares "file_type":"nvfp4" and "capabilities":["completion","vision","tools","thinking"]. Do not read that string as a CUDA instruction: nothing NVIDIA-specific is being installed or required here. It names a 4-bit weight format that NVIDIA's ModelOpt produced at packaging time, and Ollama executes the pack on Metal through MLX — there is no NVIDIA hardware, no NVIDIA runtime and no tensor-core path involved on a Mac. It is the fastest turnkey option and the lowest-fidelity one of the three quant tiers on this page; read the Ollama warning in ## Troubleshooting before using the default 27b tag.

Results

• Speed: omitted for this machine — no measurement on an M2 Max exists, on any surface searched (see below), and none is invented here. The one Apple datapoint that does exist is community-reported and single-source, on different silicon: in HuggingFace discussion #80 a user reports, on an M4 Pro whose bandwidth he states as ≈ 270 GB/s, 11.78 tok/s for Ollama's GGUF q4_K_M with no speculation against 20.26–38.36 tok/s for the MLX build. Do not read those rates onto an M2 Max: the chip, the bandwidth (400 GB/s here, per Apple's figure below) and the quant tier all differ. What does carry is the ratio — MLX beat GGUF by a wide margin on the same machine, which is the reason this recipe leads MLX. If you measure this pair, please contribute the numbers so /check/qwen3-8-27b/m2-max stops being empty.
• What the bandwidth allows: token generation is bandwidth-bound, so an upper bound follows from the bytes streamed per token. Apple's own announcement specifies the constant — "M2 Max features 67 billion transistors, 400GB/s of unified memory bandwidth, and up to 96GB of fast, low-latency unified memory." — and unlike the M3 Max and M4 Max the M2 Max does not split into bandwidth bins, so 400 GB/s applies to this 64 GB part without further disambiguation. Dividing it by the weights streamed per token caps plain decoding near 13.6 tok/s for the 8-bit build's 29.50 GB and near 24.9 tok/s for the 4-bit build's 16.05 GB. Speculative decoding is what makes the 8-bit tier comfortable rather than merely possible: accepted drafts emit several tokens per weight read, which is how the M4 Pro report above exceeds its own machine's ceiling.
• Memory usage: 44.039 GiB derived for the 8-bit build at a saturated 262,144-token context, against a 48.000 GiB addressable pool — see the budget table above and /check/qwen3-8-27b/m2-max for live data as it lands.
• The 4-bit alternative, and its cost: mlx-community/Qwen3.8-27B-4bit is 16,054,541,349 bytes (14.95 GiB) and pairs with mlx-community/Qwen3.8-27B-MTP-4bit (238,934,137 bytes). It is roughly twice as fast on this machine's bandwidth and it is what a 36 GB or 32 GB Mac needs. It is not the lead here, but the reason is thinner than it may look and is worth stating precisely. One commenter, Albert-Liu-Hongkang, ties a multimodal regression to the quant tier: in discussion #69 he writes that the "multi modal ability is inferior to 3.6 27b when comes to 4 bit quant", adding that he intends to retest at 8-bit and 16-bit. That thread has three participants, and he is the only one who names a quantisation at all — the thread's author compares against a differently-sized model and states no quant, and the third only asks which engine is in use. The same person opened a similar thread on the unsloth GGUF repo, so it is one person's impression twice, not two reports. Treat it as a reason to prefer the tier you can afford anyway rather than as evidence 4-bit is broken. On a 64 GB Mac the 8-bit tier costs you nothing but throughput, so it is the safer default.
• Vision quality note: the vision projector is the same one the Qwen3-VL family uses. The shipped mmproj GGUF declares clip.projector_type = qwen3vl_merger, clip.vision.block_count = 27, clip.vision.patch_size = 16 and a clip.vision.is_deepstack_layers array of 27 entries that are all false — matching deepstack_visual_indexes: [] in the released config.json. There is no deepstack feature injection in this model, so any advice inherited from a deepstack-enabled Qwen3-VL recipe does not apply.

For the full benchmark data, see /check/qwen3-8-27b/m2-max.

Troubleshooting

The model thinks forever and burns the whole context

This is the single most reported problem with this release, and it is a configuration issue rather than a fault. The model card states that "Qwen3.8 models operate in thinking mode by default" and that the default reasoning effort is xhigh. Users report reasoning dominating the window: one describes 80% of context going to thinking on coding work, and another that forcing medium cut consumption dramatically (unsloth GGUF discussion #45).

Fix it at the runtime with --thinking-budget, as in the commands above, which forces the closing tag when the budget is spent. Note the model card's own caveat before you set it too low: "In multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time."

ollama run qwen3.8:27b is much slower than expected on some prompts

The default Ollama tag ships speculative decoding on, and on the GGUF path that can be a large net loss. Verifiable from the registry: the manifests for qwen3.8:27b, qwen3.8:latest and qwen3.8:27b-mtp-q4_K_M are byte-identical — same 16,810,714,464-byte model blob, same 931,146,016-byte projector, same params digest — and that params blob reads {"draft_num_predict":4,"min_p":0,"presence_penalty":0,"repeat_penalty":1,"temperature":1,"top_k":20,"top_p":0.95}. The 27b-q4_K_M tag points at the same weights with the 92-byte params blob that omits draft_num_predict. So the default tag is the MTP tag.

The M4 Pro report above measured that difference directly and concluded, on his hardware, that "Ollama does activate the MTP heads." and that "speculation repays its own overhead in the best case and never more." on the GGUF path — 5.14 tok/s on unpredictable text against 11.78 tok/s with speculation off. If you want Ollama's GGUF path, pull qwen3.8:27b-q4_K_M explicitly rather than the default. If you want speed on Apple, use 27b-mlx or the mlx-vlm commands above; the same reporter notes that "on this machine the fastest path isn't GGUF at all".

llama.cpp-Metal instead of MLX, and what it costs in memory

The GGUF path works — general.architecture = qwen35 is implemented in mainline llama.cpp, and the mmproj above gives it the image path.

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj mmproj-Qwen3.8-27B-BF16.gguf \
  -c 32768 --parallel 1

Three things differ from MLX and all three cost memory.

--parallel 1 is not redundant on the server. llama-server allocates the Gated DeltaNet state once per sequence slot, and its slot count defaults to 4, not 1. The common_params struct default is indeed n_parallel = 1, but common_params_parser_init overrides it per example before any argument is read — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default } — and the server then resolves auto to 4 slots with a unified KV cache. So a bare llama-server pays 0.585 GiB of recurrent state where --parallel 1 pays 0.146 GiB. The attention KV cache is unified and is not multiplied. Note the asymmetry: llama-mtmd-cli and llama-cli are not in that override branch and do default to one sequence, so the CLI figure really is ×1.

--cache-type-k/-v cannot shrink the recurrent half. They set the attention cache's types only. For this architecture llama.cpp builds a llama_memory_hybrid whose recurrent types are passed as literal GGML_TYPE_F32 in llama-model.cpp, so the 48 Gated DeltaNet layers stay float32 whatever you quantise the KV cache to. This is allocation-layer behaviour, identical on Metal and CUDA.

-c is pre-allocated, unlike MLX's 256-token growth, so -c 262144 reserves the full 16.000 GiB up front rather than as the conversation grows.

Pick the build deliberately, because Q4_K_M is not one number across publishers: ggml-org/Qwen3.8-27B-GGUF ships 18.97 GB, unsloth 17.11 GB, bartowski 17.77 GB and lmstudio-community 16.81 GB. They also differ structurally — ggml-org's Q4_K_M carries 851 tensors and publishes the MTP head as a separate mtp-Qwen3.8-27B-Q4_0.gguf (1.68 GB) sidecar, while lmstudio-community's carries 866, the extra 15 being the MTP tensors baked in. Use --spec-type draft-mtp with a drafter only when you want speculation; a baked-in MTP head you never invoke is weight you paid for.

Video input: it works, but not everywhere you would expect

The model is genuinely a video model — the card promises "Native support for image and video understanding, from STEM diagrams and documents to hour-scale videos." — but runtime coverage is uneven, and it is worth knowing which door is open before you try one.

• mlx-vlm handles it: the qwen3_5 implementation accepts pixel_values_videos and video_grid_thw, and the MLX repos ship a video_preprocessor_config.json declaring Qwen3VLVideoProcessor.

• llama.cpp states in docs/multimodal.md that "Currently, we support image, audio and video input.", and its mtmd layer implements Qwen-VL-style temporal frame merging for exactly this projector type. Both entry points work: llama-mtmd-cli takes an interactive /video <path>, and llama-server accepts a content[] part of "type": "input_video" carrying a data or url field, advertising the capability to clients as has_inp_video.

  Two separate things can stop it, and they fail in visibly different places — worth knowing which one you have, because the error text points at the wrong one.

  What is wrong	Where it fails	What you see
  Built without MTMD_VIDEO	At the gate — mtmd_helper_support_video is #ifdef'd to a bare return false, so allow_video is false and has_inp_video is never advertised	video input is not supported - hint: if this is unexpected, you may need to provide the mmproj
  No vision projector loaded (--mmproj omitted)	Same gate — with MTMD_VIDEO defined the function reduces to mtmd_support_vision, which is just ctx->ctx_v != nullptr	The same message, which here is the accurate one; the hint is written for this case
  ffmpeg/ffprobe not on PATH	Not at the gate — at decode. The gate opens, the capability is advertised, the request is accepted, then video-context init fails	ffprobe failed for '<path>' (is ffprobe in PATH?) or failed to start ffmpeg for '<path>' (is ffmpeg in PATH?)

  The ffmpeg requirement is real but it is not part of the capability test: mtmd.cpp and clip.cpp contain zero references to either binary, and all of them live in mtmd-helper.cpp, which resolves the paths lazily when a video is actually opened. So a build that reports video support can still fail on the first clip you hand it.

• Ollama's 27b-mlx tag lists its capabilities as completion, vision, tools, thinking. Video is not among them.

• Frame-rate control over the API is first-party-documented for vLLM only; the model card's note about configuring fps says the feature "is currently supported only in vLLM."

No first-party quantised weights exist for this model

There is no Qwen-published GGUF or MLX build. Enumerating the Qwen org through the HuggingFace API returns exactly four repositories matching Qwen3.8: the base model, Qwen3.8-27B-FP8, and the two Qwen3.8-2.4T-A95B variants. Qwen does publish GGUF and MLX for other families — Qwen/Qwen3-8B-GGUF and Qwen/Qwen3-8B-MLX-4bit both exist — so this is a per-model gap, not an org policy, and a first-party build may appear later. Everything this recipe installs is a third-party conversion of Apache-2.0 weights; mlx-community and ggml-org are the two used here.

Nothing else reproduced

Beyond the above, no M2 Max-specific failure surfaced in the model's 97 HuggingFace discussions, the four distributor repositories' discussion tabs, or the llama.cpp and mlx-vlm issue trackers. Report problems via the submission form.