Apodex 1.1 mini on Apple M3 Max: 4-bit MLX at 131K, and why 48 GB stops short of 262K

What You'll Build

An OpenAI-compatible agent endpoint running Apodex 1.1 mini — Apodex AI's reasoning-first agent model, a 35.95B-parameter Mixture-of-Experts post-trained from Qwen/Qwen3.5-35B-A3B — on a 48 GB M3 Max, served by mlx_lm.server at 4-bit, documented to a 131,072-token conversation.

Two things make this page more than a re-run of the 48 GB M4 Max recipe, which documents the same capacity on a later chip.

The first is that this machine's Metal memory pool is measured here rather than assumed: ggml's own initialisation log, on hardware that identifies itself as Apple M3 Max, prints the exact byte figure this page budgets against.

The second is a correction, and it is the reason this page's headline says 131K where the M4 Max page's says 262K. A resident budget — weights plus KV cache plus recurrent state — clears the model's full 262,144-token window on a 48 GB Mac with room to spare, and that is the budget both published Apple pages were built on. But the peak during a long prefill is a larger number, it grows faster with context than the KV cache does, and on a 48 GB pool it runs out around 200–210K tokens. That is a derivation, not a measurement, and the whole of it is shown below so you can disagree with it in specifics rather than in general.

Hardware data: Apple M3 Max (48 GB unified memory, 400 GB/s) · resident working set 20.729 GiB at 131,072 tokens, derived peak ~31.7 GB during prefill, against a 36.000 GiB (38.65 GB) addressable pool · See benchmark data

⚠️ The resident budget clears the full 262,144-token window on this machine. The peak during a long prefill does not, and the peak is what decides your ceiling. Every weights-plus-KV figure on this page is a resident budget. There are two public measurements of the transient prefill term on this architecture on Apple, both on mlx-lm #1480, and taken at face value they differ by more than a factor of three. The one this page's ceiling is built on is galashko, 2026-08-13: on an M5 Max with 128 GB, running mlx-community/Qwen3.6-35B-A3B-4bit — architecturally identical to this model, field for field — "176k and 192k both complete here", peak memory climbing "1.4 GB per 16k tokens", "the curve has no knee". Rebuilt against this model's own bytes that is about 0.68 GB per 10,000 tokens, which puts 4-bit at 131,072 near 31.7 GB against a 38.65 GB pool and the full 262,144 at 43.3 GB — over the line under every slope in the band. That is why this page documents 131K. The second, lueghausen, 2026-08-30, reports a +35 GiB transient on the same model family — which on this model's weights would need a 96 GB Mac, not this one. This page does not know which is right, and does not pretend to. The two were taken on different quantisations, different runtimes and different workloads; those are differences in the conditions, not a demonstration that any of them causes the gap. What this page can say is which regime each was measured in, and that its own numbers are the first one's: one request at a time, mlx_lm.server, affine 4-bit. If you serve sustained multi-turn agentic sessions, the second is the measurement that applies to you and the floor for that usage is 96 GB. Both, and the bracket, are in the peak-memory section. Read it before you plan a window.

ℹ️ Unified memory is not VRAM, and on this chip the pool is measured rather than assumed. Apple Silicon has no dedicated VRAM. Our catalogue stores full unified memory in the GPU row, but Metal's recommendedMaxWorkingSetSize caps what the GPU may actually address, at exactly 3/4 on machines of 36 GB and above. Two reports that state their machine's capacity fix the ratio: llama.cpp #9112 — "System Details - M2 Mac Pro with 64 GB Memory", whose log line reads ggml_metal_init: GPU name: Apple M2 Max — logs recommendedMaxWorkingSetSize = 51539.61 MB, and llama-cpp-python #948 — "Mac M1 Max Processor with 64GB of ram, running MacOS Sonoma (14.1.1)" — logs 49152.00 MiB. Both are 51,539,607,552 B = 48.000 GiB = 0.75 × 64 GiB, printed once with /1e6 and once with /1024/1024. On the M3 Max specifically, ggml logs both of the values that ratio predicts for this chip's two smallest memory options — 38654.71 MB in llama.cpp #9943 and LLamaSharp #835, and 28991.03 MB in llamafile #577, each of them directly under a ggml_metal_init: GPU name: Apple M3 Max line. Those are exactly 36.000 GiB and 27.000 GiB, i.e. 3/4 of 48 GiB and 3/4 of 36 GiB. Two of those three reports state their machine's capacity outright, so nothing here is inferred. In LLamaSharp #835 the same commenter who posts the 38654.71 MB log writes "CPU & GPU device: Apple M3 Max 48GB RAM" one comment earlier in the same thread. In llamafile #577 the same log carries msg="system memory" total="36.0 GiB" and, from the loader's own planner, memory.available="[27.0 GiB]". Only llama.cpp #9943 is silent on capacity, and it independently confirms the figure in a different unit: using device Metal (Apple M3 Max) - 36863 MiB free, one MiB shy of 36.000 GiB. So 38,654,705,664 B is the pool every figure below is framed against — measured on this chip, on a machine whose owner tells you how much memory it has — never the marketing 48.

ℹ️ This is a text-only recipe, and that is a property of the artifact, not a hedge. mlx-lm's loader for this architecture discards the checkpoint's vision weights before the model is built — mlx_lm/models/qwen3_5_moe.py line 26 skips any key matching if key.startswith("vision_tower") or key.startswith("model.visual"): — and drops the multi-token-prediction head one layer down at qwen3_5.py:313 with weights = {k: v for k, v in weights.items() if "mtp." not in k}. Read out of the safetensors headers directly, the 4-bit build carries 1,757 tensors, zero vision keys and zero mtp. keys. The publisher says so too: abenzerps's card reads "This is a text-only MLX release. Image input is not supported." Apodex's own card is pipeline_tag: text-generation, and the words vision, visual, multimodal and mmproj appear zero times in it against 24 occurrences of agent as a control — even though the first-party config.json does carry a vision_config block. The checkpoint has a tower; the artifact you download does not.

ℹ️ An MoE keeps every expert resident; only the reads are sparse. This model activates 8 of 256 experts per token, which is a routing property, not a memory one. All 256 stay in memory, so the footprint is the whole quantised file. What routing buys is bytes per decode step — which matters on this chip, because the M3 Max bin this page documents runs at 400 GB/s.

Requirements

Which M3 Max this is, and here the memory size does tell you. Apple sells the M3 Max in two bins, and per Apple's MacBook Pro (16-inch, Nov 2023) specifications the memory option identifies the bin outright: the configurable memory line reads "96GB (M3 Max with 14-core CPU) or 48GB, 64GB, or 128GB (M3 Max with 16-core CPU)", and the chip line gives "M3 Max with 16-core CPU and 40-core GPU (400GB/s memory bandwidth)" against "300GB/s memory bandwidth" for the 14-core / 30-core part. So a 48 GB M3 Max is unambiguously the 16-core-CPU, 40-core-GPU, 400 GB/s part. That matters because token generation on Apple Silicon is memory-bandwidth-bound, and it also matters for the floor table below: the 36 GB M3 Max is the other bin, at 300 GB/s.

This is the same situation as the 48 GB M4 Max, where the size also picks the bin, and the opposite of the M2 Max, where 64 GB is sold on both GPU bins and the size tells you nothing. Nothing on this page depends on the bandwidth figure, because this page quotes no throughput number — see Results.

The memory budget (derived, not measured)

Nothing here was observed on a machine. Every input is a byte count read out of a safetensors header or a field read out of the model's own config.json, and the arithmetic is shown so you can redo it.

The architecture is hybrid: of its 40 decoder layers only some carry a context-scaling KV cache. config.json declares full_attention_interval: 4 and a layer_types array that counts out to 30 linear_attention and 10 full_attention entries; mlx-lm derives the identical split independently at qwen3_5.py:212 with self.is_linear = (layer_idx + 1) % args.full_attention_interval != 0 and allocates from it at :305 — [ArraysCache(size=2) if l.is_linear else KVCache() for l in self.layers]. Only 10 of 40 layers hold a cache that grows with context, which is the whole reason a six-figure context is a laptop conversation at all.

The lead build, at the 131,072-token window this page documents, one sequence:

At the model's full 262,144 tokens the same four addends give 24,941,885,696 B = 23.229 GiB resident, 12.771 GiB inside the pool. Read this table alone and the full window looks free. It is not; the next section is why.

Per-token KV is 2 (K + V) × 2 KV heads × 256 head dim × 2 bytes = 2,048 B per full-attention layer, which is 20,480 B/token across the 10 of them: 32,768 tokens costs 0.625 GiB, 131,072 costs 2.500 GiB, and the full 262,144 costs 5.000 GiB. The 2-byte element is the model's own bfloat16 (config.json → text_config.dtype), because mlx_lm.server exposes no KV-quantisation flag — a search for kv-bits in mlx_lm/server.py at v0.31.3 returns zero hits, against a hit at mlx_lm/generate.py:192 as a control.

The recurrent half does not grow with context. Each linear layer holds a Gated-DeltaNet state of 32 value heads × 128 × 128 at float32 = 2.000 MiB, plus a (kernel−1) × conv_dim bfloat16 conv state; linear_conv_kernel_dim: 4 and a conv width of 2 × (16 key heads × 128) + (32 value heads × 128) = 8192 give 3 × 8192 × 2 B = 48 KiB. Thirty layers of that is 0.060 GiB, constant, whether you run 1K or 262K.

A note on which bytes these are. The weights row is the summed tensor payload read from the safetensors headers, not the on-disk file total. The two differ by the JSON headers — 19,508,787,456 B resident against 19,509,024,201 B on disk, a gap of 236,745 B — which is why the Storage row above says 19.51 GB and this one says 18.169 GiB. Disk size and residency are different numbers and this page does not swap them.

The peak during prefill: two measurements, and what separates them

The resident table above does not include the transient workspace a prefill allocates. On this architecture that term is not a rounding error: it is bigger than the KV cache, it grows faster with context, and it is the largest single uncertainty on this page. It has two public measurements, they are three months and one runtime apart, and the first thing to say about them is that they do not agree.

Measurement one — galashko, 2026-08-13, a comment on mlx-lm #1480. Setup, from the harness repo it links: Apple M5 Max (128 GB), macOS 26.6.1, mlx 0.32.0, mlx-lm 0.31.3, mlx-community/Qwen3.6-35B-A3B-4bit, each rung in its own process, greedy, 8 generated tokens, median of two full runs that report identical peaks at every rung. This is the one the ladder below is built on, and the reason is not that it is more convenient: it is the only public measurement taken on MLX's own mode: affine 4-bit — the quantisation every build this page tabulates uses — at the 2048 prefill step this recipe runs, on one request at a time.

Read the rung labels off the raw data, not off the table. The comment tabulates "8k … 192k"; the harness's results/mlx-q4.jsonl records prompt_tokens of 8,022 / 32,022 / 64,022 / 100,022 / 130,022 / 160,022 / 176,022 / 192,022 — decimal thousands plus 22 tokens of chat template, not multiples of 1,024. Reading them as 8,192 … 196,608 flattens the slope by about 3%, which is enough to move a ceiling by ten thousand tokens.

Only the second column is measured. The third is a derivation from that repo's byte count — mlx-community/Qwen3.6-35B-A3B-4bit is 20,402,204,271 B of safetensors against this model's 19,508,787,456 B, so subtracting the wrong one inflates the transient by 0.89 GB — plus the KV and recurrent formulas above, which carry across unchanged because its config.json matches ours field for field. The fourth column is therefore a measurement minus a derivation, not a difference of two measurements.

The ladder was measured at the step size this recipe runs. bench/step_mlx.py calls stream_generate with no prefill_step_size and no draft model, so it takes generate_step's default of 2048 (mlx_lm/generate.py:316) — the same value mlx_lm.server --prefill-step-size defaults to. The 512 default in the same file at :483 belongs to speculative_generate_step, which needs a draft model and is unreachable on this architecture anyway. The unit is settled by source, not by inference: the harness writes peak_gb from mlx-lm's peak_memory, which is mx.get_peak_memory() / 1e9 — decimal GB of MLX's own Metal allocator peak, which is the right quantity to compare against recommendedMaxWorkingSetSize. That is why this section is in GB and the resident table is in GiB.

Fitting those eight points: the transient grows at about 0.68 GB per 10,000 tokens — least squares 6.8005e-5 GB/token with an intercept of 0.522 GB, and per-rung incremental slopes spanning 0.61–0.73 per 10,000. The comment's headline "1.4 GB per 16k tokens" is the slope of the total curve and it reconciles exactly: 1.451 GB per 16,384 tokens measured, minus the 0.336 GB per 16,384 the KV cache contributes at 20,480 B/token, leaves 1.116 against a fitted transient slope of 1.114 in the same unit. Both figures are right about different quantities, and the difference between them is the KV term rather than an error.

Carrying that transient onto this model's own bytes gives a peak per tier. The band is the per-rung slope spread applied to the fitted intercept, not a confidence interval:

The middle column is galashko's own top rung, which makes it a round-trip check on the whole method: this model at 4-bit comes out at 37.1 GB there, and their checkpoint carries 0.893 GB more weight, so the same arithmetic predicts 37.98 GB for their run — against the 38.0 GB they measured.

Measurement two, which this page cannot explain away

lueghausen, 2026-08-30, the same issue's second comment, posted while this page was being written. Same machine class — M5 Max, 128 GB — and a far larger transient:

Carry that +35 GiB onto this model's 4-bit weights and you get 53.169 GiB, against a 36.000 GiB pool here. If that figure describes what this recipe does, this recipe does not run on this machine — and does not run on a 64 GB Mac either. It first clears on a 96 GB Mac, whose pool is 72.000 GiB.

There is an obvious way to discount that number, and it does not work. MLX's own mx.get_peak_memory() is a running maximum of active memory that is never reset between requests — peak_memory_ = std::max(peak_memory_, active_memory_) in mlx/backend/metal/allocator.cpp at v0.32.0, read out in mlx #3896 — so a "peak" logged beside a short prompt need not have been caused by that prompt. That mechanism is real. It is not what produced this number, and the reporter says so himself in the same #3896 comment. He is explicit that "I cannot confirm the 2x gap for mx.get_peak_memory()", because the figure is not from it: "Ollama logs its own memory.peak per prefill chunk", and in his "traces that value tracked the OS closely" — the value in the trace below against a phys_footprint_peak of 56.69 GiB on the same request. Per prefill chunk rather than a session maximum, and corroborated by an OS-level reading that cannot be a counter artifact at all.

mlx prompt forward processed=6144 total=14603 tokens=2048
    memory.active="32.21 GiB" memory.cache="1.50 GiB" memory.peak="54.20 GiB"

The rest of his instrumentation says the same thing. footprint --pid read 93.2 / 93.6 / 94.7 / 99.2 GB at four separate interventions, an external watchdog killing the process each time at 90 GB. And the memory comes back afterwards — footprint returns to ~30 GiB — so this is "large transient allocation during prefill rather than a leak".

Taken at its own face value, against its own trace, the gap is about sevenfold. That traced request is total=14603 tokens; subtracting his ~22 GiB of weights leaves 10.21 GiB of live non-weight memory where this page's ladder, evaluated at 14,603 tokens, gives 1.41 GiB. That is the comparison at the length the trace actually names, and it is not close.

What separates the two measurements — three differences, all verified, none of them an explanation.

• Quantisation. His rows are nvfp4. Every build this page tabulates, and the checkpoint galashko measured, are MLX mode: affine. Nobody has measured one path's prefill workspace against the other's.
• Runtime. Ollama 0.33.1/0.33.2 using MLX for safetensors models, not mlx_lm.server. Different scheduler, different chunking, different cache management.
• Workload. Multi-turn agentic coding sessions with tool calls, prompts growing to 90–115k tokens, against a fresh process serving one request and generating 8 tokens.

Those are differences in the conditions, not a demonstration that any of them produces a sevenfold gap. This page does not claim they do. Naming a mechanism that exists is not the same as showing it made this number — that is the move that has to be resisted here, and the one this page tried in an earlier revision and got wrong.

One more thing the pair settles, and it changes what "the pool" means. His 128 GB machine ran to 93–99 GB of footprint against a 96.000 GiB Metal working set, with 50.1 GiB in the compressor. So recommendedMaxWorkingSetSize is not a wall on unified memory — you can exceed it, and what you get is compression, paging, and eventually a watchdog panic rather than a clean allocation failure. Every ceiling on this page should be read as the point past which behaviour stops being predictable, not the point at which something politely refuses.

The honest bracket

So there are two public measurements of this term and a factor of about seven between them, and no basis for discarding either:

Which one applies to you is a question about your workload, not about your machine, and that is the most useful thing this page can say. Measurement one was taken in the regime this recipe documents: one request at a time, a fresh server, affine 4-bit, the 2048 prefill step. Measurement two was taken in the regime this recipe does not document: sustained agentic multi-turn serving through a different runtime on a different quantisation. This page's numbers are the first regime's. If your usage is the second, the only public measurement of it puts the floor at 96 GB, and you should plan against that instead.

Against this machine's 38.65 GB pool, and under measurement one only, three readings in decreasing confidence:

1. 4-bit at 131,072 clears under every slope in the band, at 31.7 GB with about 7 GB to spare (6.3 GB even on the steepest slope). That is the lead.
2. The full 262,144 does not clear on 48 GB at any published quant. 4-bit is the closest and it lands at 43.3 GB [41.4–44.7] against 38.65 — over by 2.8 GB even at the most optimistic slope in the band. The full window on this model is a 64 GB conversation.
3. The 4-bit ceiling on this pool is roughly 198K–210K tokens (steepest slope to central fit; the shallowest gives 228K). So there is real room above the documented window, and it is not the model's full 262,144.

What this is not. It is an extrapolation from one reporter, on a different checkpoint, a different chip generation and a newer mlx, and past 192,022 tokens it is beyond the measured range entirely — which is exactly where reading 3 lives. The harness README adds a caveat that cuts directly at transferring it: "Content affects the numbers, not just length." Treat the tiers' ordering as solid and the absolute figures as indicative. And if you read the comment yourself, note that its closing 65 GB extrapolation for a full 262k context is about 50% above what its own table supports — carrying its stated 1.4 GB/16k slope four rungs past its top rung gives about 43.6 GB. This page uses that comment's measurements and not its extrapolation.

And the obvious lever has been measured and it does not work. Both Apple siblings tell you to lower --prefill-step-size first. lueghausen tested exactly that question — the issue's original reporter had asked whether prefill was chunked enough — and answered it: "chunked, at a fixed 2048 tokens per forward pass", and the spike happened anyway — "So chunking at 2048 does not bound the transient allocation". He swept the equivalent knob on the other runtime too, llama.cpp's -ub from 512 to 4096 on the same machine, and peak memory moved only from 9.47 to 10.79 GiB — an 8× batch for 1.3 GiB. So the step size is not the lever, on either runtime, and this page does not tell you it is. What is left is unglamorous and is what the run command below already does: one sequence, a small prompt cache, a shorter prompt, and watching the footprint with an OS-level tool rather than trusting a counter that MLX's own maintainer says is not measuring what you want.

Why 4-bit, and what the other tiers cost on this pool

Every published MLX tier below 8-bit is resident-affordable here, which is why the resident column settles nothing:

Two of those rows are worth stating in words, because they are the ones a resident-only reading gets wrong. 6-bit does not reach this page's own window on this machine — it peaks at 40.4 GB against 38.65 — even though its resident set at 131,072 (28.794 GiB) sits 7 GiB inside the pool. And 8-bit is not a tier on a 48 GB Mac at all: its weights alone are 34.299 GiB of a 36.000 GiB pool, so before any prefill there is 1.7 GiB left, which is not enough for a single 131,072-token cache. That is the row where "it fits on disk" and "you can serve with it" part company hardest.

Three costs bear on the choice, and on this machine they point the same way.

Cost one — bytes read per decode step. Token generation here is bound by memory bandwidth, and this bin runs at 400 GB/s. The tiers' payloads are 19.509 / 23.839 / 28.168 / 36.828 GB, so relative to 4-bit a step costs about 1.22× / 1.44× / 1.89×. That ratio is what matters, and it does not depend on assuming anything about the MoE gather. Splitting the safetensors census by tensor name — routed experts are 92.9% of the weight bytes at every tier — a decode step that reads every non-expert tensor plus 8 routed experts of 256 costs 1.821 / 2.221 / 2.621 / 3.420 GiB, i.e. 1.00× / 1.22× / 1.44× / 1.88×. The whole-file ratio and the gather ratio agree to within a percent, because the quantisation is applied almost uniformly across the checkpoint, so you do not have to decide which one the runtime actually does.

Cost two — how many conversations the pool still holds. One distinct prompt cache at 131,072 tokens is 2.560 GiB (KV + recurrent + conv). Against a 36.000 GiB pool, after the weights and resident-only: 6 caches at 4-bit, 5 at 5-bit, 3 at 6-bit, 0 at 8-bit. Read those as steady-state ceilings, not working numbers — a prefill in flight adds its own 9.44 GB [8.49–10.14] at this window, from the fit above. mlx_lm.server defaults --prompt-cache-size to 10 with no byte cap, which is above what this pool holds at every tier, which is why the run command below sets it explicitly.

Cost three — the window. This is the one that decides. 4-bit is the only tier that reaches even 200K on this pool, and nobody has measured what the extra bits buy in quality on this model at any tier.

Two further MLX-format builds of this model exist at 4-bit and 6-bit sizes and are deliberately absent from that table — the two whose names end in -mtp. They are not a fifth and sixth tier; on the mlx-lm release this page installs they are a documented failure mode, and the troubleshooting section says why.

The call. 4-bit at 131,072 leads, with headroom to push toward 200K if your work needs it and you are willing to watch the footprint. If you want more fidelity than 4-bit and your conversations stay under ~150K, 5-bit is the real alternative here — 36.0 GB [35.1–36.7] at 131,072 still clears 38.65 under every slope in the band, at 1.22× the per-step read. 6-bit and 8-bit are documented so you do not reach for them by reflex. Neither is a 48 GB tier: their ceilings here are ~106K–112K and ~13K–14K. Both become reasonable one memory step up — on a 64 GB pool the same arithmetic gives 6-bit a ceiling around 243K–257K — which is a machine, not a flag.

Why 48 GB is this page's floor, and where it disagrees with the M4 Max page

min_vram_gb: 48 is a filter floor: the smallest machine capacity on which this install runs at this window. Running the same arithmetic against every M3 Max memory option Apple sells — "48GB unified memory (M3 Max with 16-core CPU)" and the rest of the configurable line quoted above — gives:

Which measurement sets that floor, since two are on the table, and what the other one would give. Under measurement two the floor is 96 GB: 18.169 GiB of weights plus a 35 GiB transient is 53.169 GiB, which clears neither this machine's 36.000 GiB pool nor a 64 GB Mac's 48.000 GiB, and first fits inside a 96 GB Mac's 72.000 GiB. That is a real number and this page will not hide it — it is in the bracket table above and in the lead.

min_vram_gb: 48 is set under measurement one, because that is the measurement taken in the regime this page documents. Two things support the choice beyond regime-matching, and one limits it.

• A third datapoint sits in exactly this page's regime and is inconsistent with the larger transient. In mlx #3186, ronm92130 served mlx-community/Qwen3.6-35B-A3B-4bit — affine 4-bit, ~19 GB of weights, the same architecture — through mlx_lm.server on a 32 GB Mac mini M4, sequentially, and a ~23,200-token prefill completed ("OK, ~75.4 s wall"), with the replay of that session running "111 minutes without a panic". A 32 GB Mac addresses 25.77 GB. After weights, KV and recurrent state that leaves 6.23 GB for the transient — so the completed request caps the transient at 23,200 tokens at 6.23 GB, which is 3× what measurement one predicts there (2.10 GB) and about a sixth of measurement two's headline. The limit of that argument, stated rather than glossed: it bounds the term at 23k tokens, not at this page's 131,072, and measurement two's session was running prompts of 90–115k. A transient that only misbehaves at long context would not have shown up in it.
• 96 would be a strange floor to file against this card. Our catalogue stores the M3 Max at 48 GB, and Apple sells it at 36, 48, 64, 96 and 128. A recipe filed at 96 tells the two configurations above it that this works and tells everyone else nothing at all — including the 48 GB machine on which the in-regime measurement says it works with about 7 GB to spare.

So 48 is a floor for the workload this page documents, and it is not a guarantee. A reader on a 48 GB machine should treat 131,072 as a target to verify rather than a promise, watch the footprint with footprint <pid> rather than with ps, and read the kernel-panic entry in Troubleshooting before serving anything concurrently. If your usage is sustained agentic multi-turn, plan against 96 GB and not against this page.

This is where this page contradicts its 48 GB sibling, deliberately. The M4 Max page documents the same capacity, leads the same 4-bit build, and sets min_vram_gb: 36 on a floor table that is resident-only — on which a 36 GB Mac has 8.771 GiB left after weights and the 5.000 GiB full-context KV cache fits inside it. That arithmetic is correct and the conclusion does not follow, because the transient term is missing from it: on a 36 GB pool the 4-bit peak passes 28.99 GB at about 100K tokens, so a 36 GB machine cannot hold the 262K window that page documents, nor the 131K window this one does. The same page says of a 48 GB Mac that "the 12.771 GiB of headroom absorbs that comfortably" — 12.771 GiB is 13.71 GB, and the transient at the full window is 18.35 GB.

Two further notes, since a reader may check them against that page. Its statement that the transient scales with --prefill-step-size "rather than with total context" is wrong in both halves, and this page keeps neither: the measured peak climbs monotonically with prompt length across eight rungs, and the step size has since been tested directly and found not to bound the spike at all. And it frames the prefill wall as something seen "on a smaller machine" — #1480 reports it on an "Apple Silicon Mac Studio, 128 GB unified memory", with the reporter noting "System memory is not close to exhausted" and attributing it to transient Metal workspace. The one detailed report of this failure is on a machine with nearly three times this one's pool.

None of this is a disagreement about the artifact, the runtime, the flags or the tier: this page leads the same 4-bit build that page leads, for the same reasons. The whole difference is one term in the budget, and it moves the window and the floor rather than the recipe.

Installation

1. Install mlx-lm

The architecture is implemented in stock mlx-lm: mlx_lm/models/qwen3_5_moe.py is present at tag v0.31.3, which is the current PyPI release (uploaded 2026-04-22, confirmed today). No --trust-remote-code, no git checkout, no third-party model file:

python3 -m venv .venv && source .venv/bin/activate
pip install -U "mlx-lm>=0.31.3"

There is nothing CUDA-shaped to install and nothing to configure for the GPU — MLX targets Metal directly. Skip pip install flash-attn, skip any cu12x wheel index, and ignore the first-party NVFP4 / GPTQ-Int4 / FP8 builds on the Hub: those are CUDA-native formats (NVFP4 is a Blackwell microscaling format, GPTQ kernels are CUDA-only, FP8 tensor cores are Hopper/Blackwell) and Apple Silicon has none of that hardware.

2. Download the 4-bit MLX weights

pip install -U huggingface_hub

hf download abenzerps/Apodex-1.1-mini-MLX --local-dir ./apodex-1.1-mini-mlx

mlx_lm.server --model abenzerps/Apodex-1.1-mini-MLX also resolves the repo id directly and downloads into the Hugging Face cache; the explicit hf download above is only for controlling where ~20 GB lands.

Provenance, since every MLX build of this model is a single-author community conversion. There is no mlx-community build to prefer, so the usual "take the org build" shortcut has nothing to point at. Two checks make a single-author build safe to lead with here, and both are cheap to repeat:

• The pipeline is reproducible. abenzerps/Apodex-1.1-mini-MLX and nicolasembleton/Apodex-1.1-mini-MLX-4bit are not merely the same size — read out of the safetensors headers they are byte-identical at 19,509,024,201 B across 4 shards, 1,757 tensors each. Two people ran mlx_lm.convert and got the same output, which is what a deterministic affine quantiser should do. Either repo works; this page names abenzerps because its card documents the build.
• The chat template did not drift. chat_template.jinja in both 4-bit repos is SHA-256 identical to the one in apodex/Apodex-1.1-mini (d139c773f9995a01434c0266ab2bd8d9859de30e56a27128c63d3439c57ddec0, 8,831 B), as it is in the 6-bit repo. Redistributor template drift is a real failure mode for tool-calling models; these builds do not have it.

Both 4-bit repos declare group_size: 64, bits: 4, mode: affine with 80 per-tensor 8-bit overrides — the 40 layers' mlp.gate and mlp.shared_expert_gate routers, which is what abenzerps's card describes as "The MoE router and shared-expert gate weights remain at 8-bit; other eligible weights use affine 4-bit quantization."

Running

1. Start the server

mlx_lm.server \
  --model ./apodex-1.1-mini-mlx \
  --host 127.0.0.1 --port 8080 \
  --temp 1.0 --top-p 0.95 \
  --max-tokens 32768 \
  --prompt-cache-size 2

Every one of those flags is correcting a default that will otherwise bite you. All four defaults were read out of mlx_lm/server.py at v0.31.3:

• --temp 1.0 --top-p 0.95. Apodex's card recommends temperature: 1.0, top_p: 0.95, repetition_penalty: 1.05, max_tokens: 32768 for agentic work. The server's own default is "Default sampling temperature (default: 0.0)" — greedy — with --top-p at 1.0. You have to pass them.
• --max-tokens 32768. The server default is 512 (:1844). This is a reasoning model that opens a turn inside a <think> block; a 512-token ceiling truncates mid-thought and returns something that reads as a wrong answer rather than as a truncation.
• --prompt-cache-size 2. The default is 10 (:1874) — "Maximum number of distinct KV caches to hold in the prompt cache" — and --prompt-cache-bytes, the only byte cap ("Maximum size in bytes of the KV caches"), is unset by default, so nothing trims by size unless you ask. Ten caches at this page's 131,072-token window is 25.6 GiB on top of 18.169 GiB of weights, over the 36.000 GiB pool before a single prefill runs. Two is a working default that leaves room for the prefill term; the pool holds six at this tier if you never prefill anything long, and you should not plan on that. There is a second, non-memory reason to keep this small, and it is the one that can cost you the machine rather than the session: prompt-cache eviction churn is half of the identified trigger for the kernel panic documented in Troubleshooting.
• --host 127.0.0.1 is already the default on this server (:1766), unlike some MLX servers; it is written out so a copy-paste into a different tool does not quietly expose the endpoint.

There is no repetition-penalty flag on either entry point in this release. On the server it is a request-body parameter (repetition_penalty, default 0.0 = off), so the card's 1.05 goes in the JSON, not on the command line. Note also that the two entry points spell their template options differently: the server takes --chat-template-args (server.py:1848), mlx_lm.generate takes --chat-template-config (generate.py:162).

Nothing here raises the iogpu.wired_limit_mb sysctl, and you should not add one: at this build and this window the peak sits inside the default pool with about 7 GB to spare. Raising that sysctl is a lever for pushing past ~210K, not for making this recipe run.

Keep that separate from mx.set_wired_limit, MLX's own residency-set call, which is a different thing with the same name in it — and which has been attributed as the trigger for a kernel panic on this architecture. That is a driver bug rather than a tuning knob, and it is in Troubleshooting below.

2. Drive it as an agent

The card is explicit that tool schemas go through the API's tools= parameter rather than being pasted into the prompt, "so the system prompt itself only needs to set the role and the high-level objective" — the Qwen3.5 chat template renders them into the <tool_call><function=...> form the model was trained on.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "apodex-1.1-mini",
    "messages": [
      {"role": "system", "content": "You are Apodex, an AI assistant developed by Apodex AI. Current time: 2026-08-30."},
      {"role": "user", "content": "What is (25 + 15) * 3 - 10?"}
    ],
    "temperature": 1.0, "top_p": 0.95, "repetition_penalty": 1.05, "max_tokens": 2048
  }'

A one-shot run without a server, useful for a first smoke test and for the KV-quantisation flags the server does not expose:

mlx_lm.generate \
  --model ./apodex-1.1-mini-mlx \
  --max-tokens 2048 --temp 1.0 --top-p 0.95 \
  --prompt "Plan, step by step, how you would verify a claim in a PDF against its cited source."

mlx_lm.generate additionally accepts --kv-bits 8 --kv-group-size 64 --quantized-kv-start 5000, which roughly halves the KV cache — 2.500 GiB down to about 1.328 GiB at this page's window, since MLX's affine 8-bit at group_size 64 costs 8.5 bits per element rather than 8. Note where that lever is and is not. It shrinks the resident half, which is not what binds this machine, and it does not exist on the server path at all, so it cannot buy back either concurrency or a longer window on the endpoint you actually serve from.

Alternative: llama.cpp-Metal, and why it is not the lead

Mainline llama.cpp registers this architecture — { LLM_ARCH_QWEN35MOE, "qwen35moe" }, sits at line 42 of src/llama-arch.cpp at release b10701 (published 2026-08-30, the tree every llama.cpp reference on this page was read at), llm_arch_is_hybrid includes it at :1077 so the recurrent/attention split is modelled rather than ignored, and the build docs say "On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU." bartowski's GGUF repo also publishes a wider quant ladder than MLX does — IQ4_XS 17.955 GiB, Q4_K_M 20.363 GiB, Q5_K_M 23.741 GiB, Q6_K 28.430 GiB, all re-checked today. It is a real path, and what follows is a comparison of two imperfect ones rather than a dismissal.

It is not the lead here because of one specific report. Issue #27442 — open, labelled bug-unconfirmed, last updated 2026-08-25 — reports that qwen35moe hybrid models return an empty completion through llama-server on Apple Metal once the prompt passes roughly 16.5K tokens. The reporter, a community user, writes that "The native context length of these models is 262K, so being effectively hard-capped at ~16-19K via the server is a severe practical limitation." After a contributor suggested sampler noise they re-tested at temperature 0 with a fixed seed on a build containing the subsequent Metal flash-attention rewrite and reported that "the EOS-first behaviour is deterministic, position-sensitive, checkpoint-specific". The contributor who replied was explicit about the limits of their own input: "We have not reproduced the bug — we run this architecture on CPU and ROCm, not Metal".

Two boundaries on that report, because they cut in opposite directions:

• It is not about this checkpoint. The two models that reproduce it are Ornith-1.5-35B-A3B and a Qwen3.6-35B-A3B IQ2_XXS. Nobody has published a test on an Apodex GGUF, either way.
• It does look Metal-shaped rather than architecture-shaped. A separate report of the same symptom on a 64-layer hybrid, #27756, reproduces on CUDA and CPU at a ~130K threshold and records that the same needle test at 243k tokens "PASSES on Qwen3.5-35B-A3B (30 GDN layers) on the same build, same server, same machine" — and 30 GDN layers with 10 full-attention layers is exactly this model's shape. The family's long-context failures are not uniform, and the one that lands at 16K lands on Metal.

MLX is not the clean option either, and this page will not pretend otherwise. The open mlx-lm issues on this architecture include the Metal prefill OOM this page's whole budget is built on (#1480), unbounded server memory growth past the prompt-cache cap under multi-day serving (#1807, filed 2026-08-30), one-token completions on MTP variants when a system prompt is reused (#1292), and the speculative-decoding dead end below (#1446). Both runtimes have open hybrid-MoE defects on Apple.

The reason to lead MLX is that the defects are different in kind, and one kind is much worse to build an agent on. The llama.cpp Metal report is a silent wrong answer: an empty completion behind a clean HTTP 200, at ~16.5K tokens, on a model whose whole point is long-horizon work — a harness cannot tell that from a model choosing to stop. The MLX defects announce themselves. But "louder" understates them and this page will not leave that impression: the MLX failures on this architecture include a kernel panic and a hard power-off, documented in the Troubleshooting entry below, which costs you the machine and not only the session. So the trade is a runtime that can lie to you quietly against a runtime that can take the box down, and it is a genuinely uncomfortable one.

MLX still leads, on two grounds that survive that. A wrong answer inside an agent loop propagates into everything downstream of it and is discovered late, where a panic is discovered immediately. And the MLX hazard has an identified trigger with a tested mitigation — concurrency and prompt-cache churn, addressed by the flags this page already sets — where the llama.cpp Metal report has no diagnosis at all and no maintainer has reproduced it. Add that the MLX context threshold is an order of magnitude further out, and the choice is defensible enough to write down.

One quantitative point cuts the other way and belongs here rather than in a footnote. The same lueghausen comment measured both runtimes on one box and found llama.cpp needs far less transient memory for the same class of work: 10.0 GiB peak for an 80k-token prefill of a dense 27B, against 38.9 GiB for the same model through MLX — "MLX needs 2–3.5x more transient memory for the same class of work." On memory alone the GGUF path is the safer one on this machine; it is the correctness report above, not the arithmetic, that keeps it second. If you want the GGUF path anyway, for the finer quant ladder or for work that stays short, verify your own long-prompt behaviour before trusting it:

brew install llama.cpp

hf download bartowski/apodex_Apodex-1.1-mini-GGUF \
  --include "apodex_Apodex-1.1-mini-Q4_K_M.gguf" \
  --local-dir ./apodex-gguf

llama-server -m ./apodex-gguf/apodex_Apodex-1.1-mini-Q4_K_M.gguf \
  -ngl 99 -c 65536 -np 1 --no-mmproj \
  --cache-ram 2048 \
  --host 127.0.0.1 --port 8080 \
  --jinja --temp 1.0 --top-p 0.95

Four notes on that command, and none of the flags is inherited without a reason.

• Download the single .gguf explicitly rather than passing a bare -hf …-GGUF: the repo also ships mmproj-apodex_Apodex-1.1-mini-f16.gguf (899,283,488 B), and a bare -hf picks a projector up and puts it in memory for a recipe that has no use for it. --no-mmproj refuses it a second time.
• Pass -c explicitly. fit_params still defaults to true (common/common.h:476 at b10701), so an unset context is silently sized to the device rather than to your workload. 65,536 is chosen against the Metal report above — there is little point provisioning a window this build may not deliver — and it costs 1.250 GiB of KV on top of 19.908 GiB of resident weights (the Q4_K_M payload minus blk.40, the MTP block, which llama.cpp skips by default), for about 21.2 GiB against the 36.000 GiB pool.
• -np 1 because llama-server defaults to four parallel slots, and on this architecture every slot holds its own whole recurrent state — 65,863,680 B here, so the default quadruples a term the budget above counts once.
• --cache-ram 2048 is the flag the unified-memory case makes matter. cache_ram_mib defaults to 8192 (common/common.h:632 at b10701), i.e. the server may commit up to 8 GiB of host RAM to the prompt cache, and on a hybrid recurrent architecture each cache entry carries a whole recurrent state. On a discrete-GPU box that is host RAM you were not otherwise using. On a Mac it comes out of the same physical pool as everything above, which is why this page caps it and the NVIDIA siblings do not have to.

Not a path today: Ollama

There is no entry for this model in Ollama's official library. Checked 2026-08-30: ollama.com/library/apodex and ollama.com/library/apodex-1.1-mini both return 404, and registry.ollama.ai/v2/library/apodex/manifests/{latest,35b} return 404, while the same calls against qwen3 return 200. That covers the library/ namespace only — ollama run hf.co/bartowski/apodex_Apodex-1.1-mini-GGUF would still work, but on a Mac it runs the same GGUF through the same Metal engine and inherits the caveat above.

Results

• Speed: omitted, because no throughput measurement of this model on any Apple chip exists on the surfaces searched, and this page will not invent one. The space, so you can judge the gap: all 22 repos the Hub returns for Apodex-1.1-mini on 2026-08-30 — 11 distinct namespaces, one more than the ten this model had a few hours earlier — each card fetched as raw README and scanned twice. Twenty-one cards returned content; one (aykutx21) has no card at all, so it is counted as unreadable rather than as a clean negative. The throughput scan matched a number adjacent to a throughput unit on four cards, which is its positive control, and none of those four is on Apple hardware. The machine scan matched an Apple machine name on exactly one card — stefanprodan/Apodex-1.1-mini-oQ4e-mtp, created today, which says it was "Built for and tested with oMLX 0.6.4 on an M2 Max" and publishes no number at all. So: one card names an Apple machine and carries no measurement; four cards carry measurements and name no Apple machine; the intersection is empty. Measurements of this model do exist — just not on this silicon, and not in a form that transfers. kingjones777 publishes a ladder topping out at 63.30 tok/s on a Ryzen AI MAX+ 395 under a third-party ROCm fork of llama.cpp; ghazni101 reports 212–235 tok/s on an RX 7900 XTX — gfx1100 — under hipfire, a separate Rust engine reading its own .mq4r container rather than a GGUF. Different vendor, different silicon, different runtimes, different quant families: neither bounds this page in either direction, and neither is quoted as if it did. Treat the repo count as a snapshot — it moved by one during the writing of this page — and re-run the enumeration rather than trusting the number. If you run this, please contribute the measurement — mlx_lm.benchmark ships with mlx-lm and produces exactly the numbers /check/apodex-1-1-mini/m3-max is missing. A generation figure would be the more useful one: this bin is 400 GB/s against the M4 Max's 546, and nobody has measured either.
• Unified memory usage: two numbers, and the difference between them is this page's argument. Resident at the 131,072-token window is 20.729 GiB against a 36.000 GiB pool (19.479 GiB at 65,536; 23.229 GiB at the full 262,144) — derived from published artifact bytes and the model's config.json, not observed. The peak during a long prefill is larger and is what sets the ceiling: about 31.7 GB against a 38.65 GB pool at 131,072, rising past the pool somewhere around 198K–210K tokens, extrapolated from the community ladder in mlx-lm #1480 and its raw per-rung data. That ceiling is a per-request figure for a single-sequence workload and it is contested by about a factor of seven: a second comment on the same issue, from an agentic session on a different quantisation through a different runtime, reports a transient that would put this model's floor at a 96 GB Mac. The peak-memory section gives both, the bracket, and the three verified differences between them — none of which has been shown to cause the gap. Nobody has measured either figure on this pair, and the honest summary is that 131K is the best-supported target for the workload this page documents, not a guaranteed one.
• Quality notes: Apodex publishes agentic-evaluation scores for this model (50.2 on FrontierFinance, 27.7 on APEX-Agent). Those are the vendor's own figures from the vendor's own harness, and they say nothing about this machine or about any quantisation of it — treat them as a vendor claim, not a measurement. The effect of 4-bit versus 5-bit on them is unmeasured by anyone, which is why the tier argument above is decided on memory and bandwidth rather than on a quality gain nobody has shown.

For the full benchmark data, see /check/apodex-1-1-mini/m3-max.

Troubleshooting

Answers are short, flat, or stop mid-thought

Three server defaults, all corrected on the command line above. --max-tokens defaults to 512 and this model spends most of a turn inside <think>, so a truncated reasoning block is the common cause of a "wrong" answer. --temp defaults to 0.0 and --top-p to 1.0, against the card's recommended 1.0 / 0.95 — greedy decoding on a reasoning model tends to flatten and loop. And repetition_penalty has no CLI flag at all: pass it in the request body (the card recommends 1.05; the server's default is 0.0, i.e. off).

The model insists it is Apodex, or ignores your persona

That is the chat template, not the weights. chat_template.jinja sets identity_mode to 'fallback' when the caller does not define it, and its third line computes inject_identity = (identity_mode == 'force') or (identity_mode == 'fallback' and not has_caller_system) — so a # Role Definition block naming Apodex is injected whenever you send no system message. Send your own and the injection stops; pass identity_mode: 'force' through --chat-template-args if you want it regardless.

A prefill OOMs, or the process dies partway through a long prompt

This is the failure this page's budget is about, and it is the one you are most likely to see on this machine, because the resident set is comfortable and the peak is not.

Do not reach for --prefill-step-size. It is the intuitive lever, both published Apple recipes recommend it, and it has now been tested: lueghausen on mlx-lm #1480 reports that with prefill "chunked, at a fixed 2048 tokens per forward pass", the spike happens anyway, and a matching -ub sweep on llama.cpp moved peak memory 9.47 → 10.79 GiB across an 8× change in batch width. Lower your prompt length instead, keep --prompt-cache-size small, and serve one request at a time.

Measure it with an OS-level tool, not with MLX's counter. mx.get_peak_memory() is a high-water mark of active allocation and excludes the buffer pool; closing mlx #3896, a maintainer writes that for "longtime and parallel model serving it is not going to accurately report the actual footprint" and suggests reading get_active_memory() + get_cache_memory() instead. On a long-lived server, footprint <pid> is the number that matches what the OS will kill you for — and note how far wrong the ordinary tools go: on the machine in mlx #3896, ps -o rss reported 15 GB and ollama ps 20 GB at a moment when footprint --pid read 93 GB. ps "does not see Metal/IOSurface-resident memory on Apple Silicon at all", so anything you build monitoring on will show a healthy process seconds before the machine dies.

If you hit this well below the ~198K–210K ceiling derived above, that is a datapoint nobody has — it would be the first Apple measurement of this model either way, so please report it.

Memory climbs across a few different conversations

The prompt cache, almost certainly. At 4-bit this pool holds six caches at the 131,072-token window resident-only — fewer once a prefill is in flight — against a default of ten. The server keeps up to --prompt-cache-size distinct KV caches and applies no byte cap unless --prompt-cache-bytes is set, so several long conversations accumulate at up to 2.560 GiB each at this window, or 5.060 GiB each at the full one. The server logs Prompt Cache: N sequences, X GB at INFO — read that before blaming the weights. Batched decode is the other multiplier: --decode-concurrency defaults to 32 (server.py:1856), and every figure in this recipe is for a single sequence. One flag that will not help, and it is worth knowing before you reach for it: --max-kv-size is "a server-level --max-kv-size would NOT have helped here, and is silently ignored" for "make_cache architectures like the qwen3_5 hybrids" — which is this one — per the reporter who traced the panic below.

If it keeps climbing after you have capped the cache, that is a known open report and not your configuration. mlx-lm #1807, filed 2026-08-30 against a hybrid-attention qwen3_5 model on mlx-lm 0.31.3, describes a server on a 128 GB Mac Studio whose footprint passes 30 GB for a ~16 GB model under an 8 GB cache cap, over multi-day serving, reclaimed only by restarting the process. No maintainer has replied and no workaround is offered, so on a long-lived agent endpoint the practical mitigation today is to watch the footprint and restart on a schedule. That report also notes that RSS underreports the true footprint on macOS because of compressed memory, which is worth knowing before you decide from Activity Monitor that everything is fine.

The machine kernel-panics or hard-freezes while serving

This is not your memory budget and it is not an OOM — it is a driver bug, it has an identified trigger, and it has a mitigation that has been soak-tested by two people on two machines.

mlx #3186 is open, labelled bug, and filed with Apple as FB22091885. The signature is a panic in IOGPUMemory.cpp:550, "completeMemory() prepare count underflow". It matters here because of who reproduced it: ronm92130 on a Mac mini M4 base (32 GB), mlx-lm 0.31.3 — the release this page installs — serving mlx-community/Qwen3.6-35B-A3B-4bit, the same architecture and the same 4-bit affine build class this recipe uses, through mlx_lm.server, panicking about eight minutes after start with resident memory around 20 GB "well under the 32 GB ceiling". The original reporter, on an M4 Max with 36 GB, is equally explicit that "Memory capacity is not the issue".

The trigger is concurrency and cache churn, not prompt length. Isolated by single-variable testing in the same thread: a faithful replay of sequential requests up to 23k-token prefills ran 111 minutes without a panic, while adding two concurrent request streams plus prompt-cache eviction churn panicked in 102–108 seconds, 3 of 3 cold boots. That is the strongest available argument for the two things the run command above already does — one request at a time, and --prompt-cache-size 2 rather than the default ten. It also means a bigger Mac does not buy you out of it: a 96 GB M3 Ultra appears in the same thread.

The mitigation, and its cost. The same investigator attributes it to the per-buffer residency-set traffic that mx.set_wired_limit enables, and reports that never calling it gives "no panic, 10/10 runs, 4.2 h" against a stock control that panicked in about 100 seconds — repeated on a 6-bit build of the same model, so it is not quant-specific. The workaround is a launcher shim that no-ops that call before mlx_lm.server starts, at a measured cost of "≤3 % sequential throughput" because the weights become pageable. A second operator, BrunoCerberus, reports it still holding on a much newer driver (IOGPUFamily 162.11, macOS 27) as a permanent production deployment on mlx 0.32.2 / mlx-lm 0.31.3.

Three cautions before you copy any of that. The shim's MLX_LM_DISABLE_WIRED_LIMIT environment variable is not an mlx-lm feature — that operator says so himself; it is his own script's toggle, and setting it on a stock install does nothing. --prompt-cache-bytes was tried in the thread and did not prevent the panic. And an open-sourced version exists as Harperbot/metal-guard (MIT), named here because the thread names it, not because this page has run it. If you are serving one request at a time — which is what this recipe documents — you are already in the arm that ran for 111 minutes clean, and the shim is something to reach for only if you start serving concurrently.

Speculative decoding requires a trimmable prompt cache

Speculative decoding does not work for this architecture on mlx-lm, and the reason is structural rather than a misconfiguration. mlx_lm/generate.py:532 refuses when any layer's cache reports itself non-trimmable — "Speculative decoding requires a trimmable prompt cache" — and this model's 30 linear layers use ArraysCache (models/cache.py:594), which defines neither is_trimmable nor trim and therefore inherits _BaseCache's return False, while KVCache at :325 defines both. A recurrent state cannot be rolled back the way a KV cache can. The open issue is mlx-lm #1446; the pull request that would have made the recurrent cache trimmable, #1730, was closed unmerged.

Multi-token prediction is not an alternative route on this path: the first-party checkpoint ships an MTP head, mlx-lm drops it on load, and every MLX build listed above carries zero mtp. tensors, so there is no head left to draft with. Even a checkpoint that kept one would not help through this server — mlx-lm #1462 is open on mlx_lm.server --draft-model failing to load any Qwen3.6 MTP checkpoint at all, because model_type qwen3_5_mtp is not registered. One thing here changed on the day this page was written, and it is worth knowing about even though it is not a recommendation: an MTP-retaining Apple build of this model now exists — stefanprodan/Apodex-1.1-mini-oQ4e-mtp, created 2026-08-30. Read out of its safetensors headers rather than off its card, it carries 2,052 tensors of which 42 are language_model.mtp.* — the head really is there — in 21,612,875,019 B across five shards. It is built for oMLX, a third-party runtime this page does not document, not for mlx-lm; its own card says "Other MLX runtimes have not been tested with this build."; it carries no published measurement, and it is hours old.

It does not open a speculative-decoding path on mlx-lm, and that is checkable rather than a guess. At the release this page installs, the load-time filter is still there: qwen3_5.py:313 reads weights = {k: v for k, v in weights.items() if "mtp." not in k}, and qwen3_5_moe.Model.sanitize routes into it at qwen3_5_moe.py:52. So mlx_lm.server discards all 42 of those tensors before the model is built, and you are left with the double-shift hazard of the next entry and none of the drafting. The head being present in the artifact and the head being usable by your runtime are different questions; this build changes the first and not the second.

Garbage output from an MLX build whose name ends in -mtp

Avoid those builds on the release this page installs, and note that this is no longer a hypothetical. Two repos of this model retain the head, and neither claim here comes from the repo name — a name is a label, the tensor list is the fact. Reading their safetensors headers directly, stefanprodan/…-oQ4e-mtp and felkf/…-oQ6e-fp16-mtp each carry 2,052 tensors, of which 42 are mtp. and 333 are vision — against 1,757 / 0 / 0 in every build this page tabulates. Their tags disagree about what they are (one is tagged mlx, the other transformers and image-text-to-text) and their config.json files agree: both declare MLX's own mode: affine quantiser at group_size: 64, at 4 and 6 bits, with 317 and 236 per-tensor overrides. The oQ in those names is a publisher's label for that per-tensor map, not a format either config declares — so if you load one, you load it into the same sanitize path as everything else. That path is qwen3_5_moe.Model.sanitize, which delegates to qwen3_5.Model.sanitize at models/qwen3_5_moe.py:52, and at v0.31.3 it decides whether a checkpoint still needs its RMSNorm shift with should_shift_norm_weights = has_mtp_weights or has_unsanitized_conv1d (models/qwen3_5.py:308,312), using the presence of MTP tensors as a proxy for "this is a raw export". For an already-converted checkpoint that keeps its head the proxy is wrong and every norm is shifted a second time at :328, with no error raised anywhere.

Two claims there, and only one of them is measured — keep them apart. That these two repos retain 42 mtp. tensors is a fact read out of their own headers. That loading one into mlx-lm v0.31.3 therefore produces token salad is an inference from a code path: the mechanism is read from source and the same proxy is what the merged fix below removes, but nobody has published a run of either of these two repos through this release, and this page has not run one either. Treat it as a well-sourced reason to avoid them here, not as a reported failure. The count is a snapshot too: it was one build a day ago and two today.

This is fixed upstream but not in a release yet, and the dates are the whole story. mlx-lm #1623 — "Fix Qwen3.6 converted RMSNorm double shift" — was merged on 2026-08-18 at 02:20:00Z, and it drops the MTP proxy in favour of the raw Conv1D layout alone; reading qwen3_5.py on main confirms should_shift_norm_weights and has_mtp_weights no longer occur, while the "mtp." not in k filter remains. Two earlier attempts at the same fix did not merge: #1442 is still open, and #1735 was closed unmerged five minutes before #1623 landed, which is what supersession looks like rather than a rejection. But v0.31.3 shipped on 2026-04-22, four months before the merge, so the release you install above still has the hazard and will until the next one. The 4-bit and 5-bit builds this page tabulates all carry zero mtp. tensors and are unaffected either way.

A build's card warns of a Metal OOM below 48 GB

Every nicolasembleton tier carries the same one-sentence Known Issues section: "Metal OOM during long-context prefill on less than 48 GB unified memory. Keep context under 32k tokens." Read the qualifier — less than 48 GB — and then check what the sentence is attached to. Diffing the four cards, the only differences are the tier name, the bits-per-weight and the size line, so this caution is byte-identical on an 18.169 GiB build and on a 34.299 GiB one. A warning that does not change across a 16 GiB spread in artifact size is not a measurement of any of them, and it names no chip, no context length it was observed at, and no figure.

It is still worth taking seriously in one respect, and that is the word prefill, because there is a real report underneath the boilerplate — and this page's entire ceiling is built on it. What the card gets wrong is the quantity. mlx-lm #1480 documents exactly this failure on this architecture shape, and it does so on an "Apple Silicon Mac Studio, 128 GB unified memory", with the reporter recording that "System memory is not close to exhausted". So the card's "less than 48 GB" threshold is not where the wall is; on a machine with nearly three times this one's pool, it was still hit.

The thread does not settle where the wall is, either, which is worth knowing before you plan around it. It is a 128 GB machine failing at 176k; its first comment is a different 128 GB machine completing 176k and 192k; its second, from a third 128 GB machine, reports a transient roughly seven times larger than the first comment's. Same capacity, three outcomes — but none of it is a controlled comparison, because the runs differ in mlx version, in quantisation, in runner, and in whether the process had served anything before. So the wall is not a fixed function of how much memory the machine has, and the card is directionally right about the failure mode and wrong about the number — which is what you would expect from a sentence that is byte-identical across four builds.

Tried to install FlashAttention, a CUDA toolkit, or the NVFP4 build

None of those apply. There is no CUDA on macOS, no FP8 or NVFP4 tensor hardware on Apple Silicon, and no GPTQ/AWQ/bitsandbytes GPU kernels — MLX uses its own Metal kernels and its own affine quantisation. The first-party Apodex-1.1-mini-NVFP4 and -GPTQ-Int4 repos are for NVIDIA deployments. Note what does not disqualify them, because it is the thing people repeat: their quick-starts are single-GPU — both launch SGLang at --tp-size 1, and the FP8 repo at --tp 1 — so they are not datacenter-scale recipes you can dismiss on size. It is the unquantised base repo that runs eight-way (--tp 8, --tensor-parallel-size 8). The quantised three fail here for the only reason that matters on this machine: NVFP4, GPTQ and FP8 are CUDA-side formats with no Apple Silicon implementation, at any tensor-parallel degree. If a generic tutorial tells you to pass --tensor-parallel-size 8, -DGGML_CUDA=ON, or to pip install flash-attn, skip it — the commands above are the complete Apple path.

Report anything else via the submission form.