How much VRAM does Mistral Nemo 12B need?

About 16 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Mistral Nemo 12B on Apple M2 Pro (16GB): Local Private Assistant via llama.cpp / Ollama (Metal)

What You'll Build

A fully local, private general assistant: Mistral Nemo 12B — Mistral AI and NVIDIA's Apache-2.0 generalist (Instruct, release 2407) — served as an OpenAI-compatible endpoint by llama.cpp or Ollama on an Apple M2 Pro with 16GB unified memory, then used from a chat UI (Open WebUI is a good local front-end) or directly via the API. This is a text-only chat/reasoning/writing model: general Q&A, drafting and editing, multi-step reasoning, function calling, and strong multilingual support. Positioned as a drop-in upgrade to Mistral 7B, it's a capable 12B that runs on modest hardware — the M2 Pro's Metal GPU serves it entirely from unified memory, and this is the entry Apple tier for Nemo. Everything runs on your own Mac, so prompts and documents never leave the machine.

Hardware data: Apple M2 Pro (16GB unified memory, Metal) · Mistral Nemo 12B, GGUF Q6_K (10.06GB, recommended) — or Q5_K_M (8.73GB) / Q4_K_M (7.48GB) for more context headroom · See benchmark data

ℹ️ This is a dense, text-only 12B generalist — no MoE, no vision. Mistral Nemo is a MistralForCausalLM (model_type: mistral) — 40 layers, hidden size 5120, GQA with 32 query / 8 KV heads, head_dim 128. Because it is dense, its footprint is simply the quant file you load plus the KV cache; there is no "active-parameters" shortcut. It is a pure text model — there is no vision tower and no image input. Context window is 128K (max_position_embeddings 131072). It was the first model to use Mistral's Tekken tokenizer (tekken.json), which needs mistral-common on the Python serving paths — but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there. Nemo was trained with quantization awareness for FP8 inference and tuned for function calling and multilingual use.

ℹ️ Runs on current llama.cpp out of the box. Mistral Nemo shipped in July 2024 and has been long supported — there is no special patch or PR gate. Just use a recent llama.cpp (or Ollama) build. On Apple Silicon the Metal backend is on by default. Pass --jinja so the embedded chat template applies.

⚠️ Use a low sampling temperature (~0.3). Mistral recommends a low temperature (~0.3) for Nemo; the usual default of 0.7 noticeably degrades output quality on this model. Set it explicitly — this is a real, easy-to-miss gotcha.

Requirements

Component	Minimum	Tested target
GPU	Apple Silicon with Metal (unified memory)	Apple M2 Pro (16GB unified, Metal)
Memory	16GB unified memory	16GB unified (shared with the OS)
Storage	~8GB (Q4_K_M) up to ~13GB (Q8_0)	~10GB for Q6_K
Software	Recent llama.cpp (Metal) or Ollama; optional Open WebUI chat client	`llama-server`, Open WebUI

Model weights (community GGUF — there is NO first-party GGUF). Mistral publishes only the full-precision weights (mistralai/Mistral-Nemo-Instruct-2407); the model is quantized to GGUF by the community. Primary source is bartowski/Mistral-Nemo-Instruct-2407-GGUF; unsloth/Mistral-Nemo-Instruct-2407-GGUF is a good alternative that also ships smaller Q2_K / Q3_K_M quants. Byte-verified on-disk sizes (bartowski):

Quant	On-disk size	Fit on M2 Pro (16GB unified)
Q4_K_M	7.48GB	Tiny footprint — most room for a large KV cache / long context
Q5_K_M	8.73GB	Comfortable — more quality than Q4 with plenty of context room
Q6_K	10.06GB	Recommended — near-lossless-feeling and sits comfortably under the ~11-12GB GPU-usable ceiling on a 16GB Mac
Q8_0	13.02GB	Tight — may not fit. On a 16GB Mac only ~11-12GB is usable by the GPU once the OS takes its share, and 13.02GB exceeds that; expect it to fall back to the CPU or fail to load. Prefer Q6_K here
f16	24.50GB	Full precision — does not fit 16GB; needs a much larger-memory Mac

Not model weights — don't count this in the memory math:

The .imatrix (~7 MB) is calibration data used to produce the quants — never load it as a model.

Unified memory, honestly. On Apple Silicon the GPU shares the same physical RAM as the OS and apps. On a 16GB Mac only roughly ~11-12GB is realistically usable by the GPU once macOS reserves memory — so plan around that ceiling, not the full 16GB. That's why Q6_K (10.06GB) is the recommended quant here: it leaves a little room for the KV cache and the rest of the system, while Q8_0 (13.02GB) is tight and may not fit. If a large model refuses to stay on the GPU you can raise the wired-memory limit with sudo sysctl iogpu.wired_limit_mb=<MB>, but don't starve the OS — leave a few GB free.

Licensing. Mistral Nemo 12B is Apache-2.0 — free for commercial and non-commercial use, no revenue caps (model card).

Installation

You have two GGUF runtimes; pick one. Both are fine for this model — there is no patch requirement — so choose Ollama for the fastest start, or llama.cpp for the most control over context and KV-cache quantization. Both use Apple's Metal GPU backend; there is no CUDA on a Mac.

Option A — llama.cpp with Metal

Build a recent llama.cpp with the Metal backend, per the official build guide. On Apple Silicon Metal is enabled by default, so a plain build already targets the M2 Pro GPU:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# Metal is ON by default on macOS/Apple Silicon; the flag is shown here explicitly
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j 8

A recent release is all you need — Mistral Nemo has been mainline in llama.cpp since its July 2024 launch. If you prefer a prebuilt binary, grab a current macOS build from the releases page. No CUDA toolkit is involved on a Mac — the Metal backend ships with the build.

Option B — Ollama

Ollama is built on llama.cpp and is the fastest way to stand this model up. On Apple Silicon it uses Metal automatically. Either use the curated tag (ollama run mistral-nemo) or pull the community GGUF straight from Hugging Face (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K

Swap the :Q6_K tag for :Q5_K_M or :Q4_K_M if you want an even smaller footprint. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Running

With llama.cpp

Serve an OpenAI-compatible API on port 8000. The -hf flag pulls the GGUF from Hugging Face; append :Q6_K (case-insensitive) to pick the quant (llama-server docs):

# Q6_K (recommended on 16GB), offload all layers to the M2 Pro GPU, low temperature per Mistral's guidance
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K \
    --port 8000 \
    -ngl 99 \
    -c 8192 \
    --temp 0.3 \
    --jinja

-ngl 99 (--n-gpu-layers) offloads every layer to the Metal GPU — the dense 12B Q6_K file (10.06GB) sits in unified memory under the ~11-12GB GPU-usable ceiling.
-c 8192 sets an 8K context. Unified memory is shared with the OS, so start conservative and raise it while watching memory; quantize the KV cache (below) to push context further.
--temp 0.3 sets the low sampling temperature Mistral recommends for Nemo — leaving it at the usual 0.7 noticeably degrades output. Set it explicitly (many clients default higher).
--jinja applies the GGUF's built-in chat template so the assistant format parses correctly.

Push toward a longer context window. Mistral Nemo advertises a 128K context (max_position_embeddings 131072), but on 16GB unified memory the KV cache is the constraint, not the model. Quantize the KV cache to fit more: add -fa on (Flash Attention, required for a quantized cache) and -ctk q8_0 -ctv q8_0, which roughly halves KV-cache memory versus f16 with minimal quality impact:

# Longer context by 8-bit-quantizing the KV cache (raise -c only as memory allows)
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K \
    --port 8000 -ngl 99 -c 16384 --temp 0.3 --jinja \
    -fa on -ctk q8_0 -ctv q8_0

Because Nemo is only 12B, Q6_K leaves headroom for a useful context on a 16GB Mac. If you need more context, drop to Q5_K_M (8.73GB) or Q4_K_M (7.48GB) to free memory for a larger KV cache.

With Ollama

Pull and run the community GGUF directly from Hugging Face; append a :quant tag to choose the quant (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K

Remember to set a low temperature (~0.3) in your client or Modelfile — Ollama's default sampling can be higher, and Nemo degrades at 0.7. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Use it as a chat assistant

Point any OpenAI-compatible chat client at your local endpoint by setting its base URL and a dummy API key — no cloud, no per-token cost.

Open WebUI (optional local chat front-end). A self-hosted, ChatGPT-style UI that talks to any OpenAI-compatible server. Run it and point it at your local endpoint:

# Point Open WebUI at your local llama-server (or Ollama on :11434)
docker run -d -p 3000:8080 \
    -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
    -e OPENAI_API_KEY=EMPTY \
    ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 and chat. (Open WebUI also autodetects a local Ollama install, so with the Ollama path you can skip the base-URL wiring entirely.) Set the temperature to ~0.3 in the model's parameters.

Directly via the API. Any OpenAI SDK or curl works against the same endpoint — use it for scripts, writing tools, or your own app:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "mistral-nemo-12b",
      "temperature": 0.3,
      "messages": [{"role": "user", "content": "Summarize this in three bullet points: ..."}]
    }'

Local servers don't check the key, so any non-empty string (e.g. EMPTY) works where a client requires one.

Results

Memory usage: The dense 12B loads entirely as its GGUF file — Q6_K is 10.06GB on disk (byte-verified from the bartowski GGUF tree). On the M2 Pro's 16GB unified memory only ~11-12GB is usable by the GPU once the OS takes its share, so Q6_K sits comfortably with a little left for the KV cache — and more with an 8-bit-quantized cache (see Running). Q5_K_M (8.73GB) and Q4_K_M (7.48GB) shrink the footprint further for a larger context. Q8_0 (13.02GB) is tight and may not fit the ~11-12GB usable ceiling, and the full-precision f16 GGUF (24.50GB) does not fit 16GB at all.
Model capability (vendor evals — Mistral's own, NOT hardware throughput): Mistral reports MMLU 68.0% and HellaSwag (0-shot) 83.5%, with strong multilingual results — MMLU French 62.3%, German 62.7%, Spanish 64.6%. These are the vendor's benchmarks, not measurements on this GPU.
Speed: No community throughput benchmark for Mistral Nemo 12B on the Apple M2 Pro exists yet — we would rather omit a tok/s figure than invent one or borrow it from different hardware. Live measurements will appear at /check/mistral-nemo-12b/m2-pro once contributed.

For the full benchmark data, see /check/mistral-nemo-12b/m2-pro.

Troubleshooting

Output quality is poor / rambling / incoherent — check the temperature

Mistral recommends a low sampling temperature of ~0.3 for Nemo. The common default of 0.7 noticeably degrades this model's output — if responses feel off, this is the first thing to fix. Set --temp 0.3 on llama-server, or the equivalent temperature parameter in your client / Ollama Modelfile.

The chat template looks wrong / responses are malformed

Pass --jinja to llama-server so the GGUF's built-in chat template is applied — without it the assistant format won't parse. Mistral Nemo uses Mistral's Tekken tokenizer (tekken.json) — it was the first Tekken model. On the Python serving paths that needs mistral-common, but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there.

Out of memory, or when raising the context

On a 16GB Mac the GPU can use only ~11-12GB, so Q6_K (10.06GB) is the safe recommended quant and Q8_0 (13.02GB) is likely too tight — it can fail to load or spill to the CPU. Options, in order: keep Q6_K and quantize the KV cache with -fa on -ctk q8_0 -ctv q8_0 (roughly halves cache memory); lower -c; or drop to Q5_K_M (8.73GB) or Q4_K_M (7.48GB) for more room. If a model won't stay on the GPU you can raise sudo sysctl iogpu.wired_limit_mb=<MB>, but leave the OS a few GB. The f16 GGUF (24.50GB) does not fit 16GB.

There's no `nvidia-smi` — this is Apple Metal, not CUDA

On a Mac there is no nvidia-smi and no CUDA — the GPU is Apple's, driven by the Metal backend (default on Apple Silicon). To watch GPU and memory pressure use Activity Monitor (Window → GPU History) or sudo powermetrics --samplers gpu_power in a terminal. Serving Mistral Nemo via llama.cpp or Ollama does not require PyTorch or a Python ML stack; if the model won't use the GPU, confirm you built (or downloaded) a Metal-enabled llama.cpp (Option A, -DGGML_METAL=ON). At 12B, Q6_K is already near-lossless.

Model or GPU 404 on /check

Mistral Nemo 12B is a new addition; if the /check/mistral-nemo-12b/m2-pro link 404s, the catalogue row is still being registered. The recipe's install and run steps are independent of the benchmark endpoint.