How much VRAM does Mistral Nemo 12B need?

About 16 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Mistral Nemo 12B on RX 7800 XT: Local Private Assistant via llama.cpp-HIP / Ollama (ROCm, 16GB)

What You'll Build

A fully local, private general assistant on a 16GB Radeon RX 7800 XT (RDNA3, Navi 32, gfx1101): Mistral Nemo 12B — Mistral AI and NVIDIA's Apache-2.0 generalist (Instruct, release 2407) — served as an OpenAI-compatible endpoint by llama.cpp (built against AMD's HIP/ROCm backend) or Ollama, then used from a chat UI (Open WebUI is a good local front-end) or directly via the API. This is a text-only chat/reasoning/writing model: general Q&A, drafting and editing, multi-step reasoning, function calling, and strong multilingual support. Positioned as a drop-in upgrade to Mistral 7B, it's a capable 12B that fits a 16GB card comfortably at near-lossless quality. Everything runs on your own hardware, so prompts and documents never leave the machine.

Hardware data: RX 7800 XT (16GB VRAM) · Mistral Nemo 12B, GGUF Q8_0 (13.02GB, recommended — near-lossless) — or Q6_K (10.06GB) / Q5_K_M (8.73GB) for more KV-cache / context headroom · ROCm 7 · See benchmark data

⚠️ This is a ROCm recipe, not CUDA. The RX 7800 XT runs on AMD's ROCm/HIP stack — there is no cu124/cu128 wheel and no FlashAttention prebuilt-wheel step here. For this model the reliable path is GGUF via llama.cpp-HIP (or Ollama, which bundles llama.cpp). Do not follow a guide that tells you to pip install flash-attn, pick a cu12x wheel, or use ExLlamaV2/Marlin for this card — those are NVIDIA-only.

ℹ️ This is a dense, text-only 12B generalist — no MoE, no vision. Mistral Nemo is a MistralForCausalLM (model_type: mistral) — 40 layers, hidden size 5120, GQA with 32 query / 8 KV heads, head_dim 128. Because it is dense, its footprint is simply the quant file you load plus the KV cache; there is no "active-parameters" shortcut. It is a pure text model — there is no vision tower and no image input. Context window is 128K (max_position_embeddings 131072). It was the first model to use Mistral's Tekken tokenizer (tekken.json), which needs mistral-common on the Python serving paths — but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there. Nemo was trained with quantization awareness for FP8 inference and tuned for function calling and multilingual use.

ℹ️ Runs on current llama.cpp out of the box. Mistral Nemo shipped in July 2024 and has been long supported — there is no special patch or PR gate. Just use a recent llama.cpp (built with the HIP backend, below) or Ollama. Pass --jinja so the embedded chat template applies.

⚠️ Use a low sampling temperature (~0.3). Mistral recommends a low temperature (~0.3) for Nemo; the usual default of 0.7 noticeably degrades output quality on this model. Set it explicitly — this is a real, easy-to-miss gotcha.

Requirements

Component	Minimum	Tested target
GPU	16GB VRAM (Q8_0 / Q6_K path)	RX 7800 XT (16GB, RDNA3 Navi 32, gfx1101)
RAM	16GB system RAM	32GB comfortable
Storage	~8.7GB (Q5_K_M) up to ~13GB (Q8_0)	~13GB for Q8_0
Driver	AMD ROCm v7 (installed via `amdgpu-install`) on Linux	—
Software	llama.cpp (HIP build) or Ollama; optional Open WebUI chat client	`llama-server`, Open WebUI

Model weights (community GGUF — there is NO first-party GGUF). Mistral publishes only the full-precision weights (mistralai/Mistral-Nemo-Instruct-2407); the model is quantized to GGUF by the community. Primary source is bartowski/Mistral-Nemo-Instruct-2407-GGUF; unsloth/Mistral-Nemo-Instruct-2407-GGUF is a good alternative that also ships smaller Q2_K / Q3_K_M quants. Byte-verified on-disk sizes (bartowski):

Quant	On-disk size	Fit on RX 7800 XT (16GB)
Q5_K_M	8.73GB	Fits with lots of context room — good if you want the largest KV cache
Q6_K	10.06GB	Comfortable — near-lossless-feeling with room for a large KV cache
Q8_0	13.02GB	Recommended — near-lossless weights with ~3GB left for the KV cache; the practical best-quality choice on 16GB
f16	24.50GB	Full precision — does not fit 16GB (needs a 24GB+ card)

Not model weights — don't count this in the VRAM math:

The .imatrix (~7 MB) is calibration data used to produce the quants — never load it as a model.

Licensing. Mistral Nemo 12B is Apache-2.0 — free for commercial and non-commercial use, no revenue caps (model card).

Installation

Prerequisite — install the AMD ROCm v7 driver

The RX 7800 XT (gfx1101) is an officially ROCm-supported GPU — it is listed with LLVM target gfx1101 in AMD's ROCm Linux system-requirements matrix — but ROCm is not bundled with Ollama or the llama.cpp release binaries; you install it once at the OS level. Per the Ollama AMD GPU docs, Ollama requires the AMD ROCm v7 driver on Linux, installed or upgraded with the amdgpu-install utility. On Ubuntu 24.04 (Noble), install ROCm 7.2.1 via the standard amdgpu-install flow (AMD's Radeon ROCm install docs cover the current packages):

# 1. Add the amdgpu-install package and install ROCm
wget https://repo.radeon.com/amdgpu-install/7.2.1/ubuntu/noble/amdgpu-install_7.2.1.70201-1_all.deb
sudo apt install ./amdgpu-install_7.2.1.70201-1_all.deb
sudo apt update
sudo amdgpu-install -y --usecase=graphics,rocm

# 2. Add yourself to the render/video groups (log out/in afterward)
sudo usermod -a -G render,video $LOGNAME

After logging back in, confirm the driver sees the card — rocm-smi should list the RX 7800 XT (this is the ROCm equivalent of nvidia-smi; there is no nvidia-smi on an AMD box). Because the RX 7800 XT is on the supported-GPU matrix as gfx1101, you should not normally need an HSA_OVERRIDE_GFX_VERSION masquerade. If a tool ships only gfx1100 kernels and refuses to start on your card, the documented Linux fallback is to export HSA_OVERRIDE_GFX_VERSION=11.0.0 so the gfx1101 card presents as gfx1100 — treat that as a fallback, not a default.

You have two GGUF runtimes; pick one. Both are fine for this model — there is no patch requirement — so choose Ollama for the fastest start, or llama.cpp for the most control over context and KV-cache quantization.

Option A — llama.cpp built with HIP/ROCm (recommended for full control)

1. Build llama.cpp with the HIP backend. Per the llama.cpp build docs, the Linux HIP build pattern is the same as for any RDNA3 card — only the GPU_TARGETS value changes. For the RX 7800 XT, pin it to gfx1101 (the card's LLVM target per the AMD ROCm system-requirements matrix):

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# RX 7800 XT is RDNA3 Navi 32 = gfx1101
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release -- -j 16

-DGGML_HIP=ON selects the ROCm backend; -DGPU_TARGETS=gfx1101 pins the kernels to the RX 7800 XT's architecture (Navi 32). Do not copy a gfx1100 value from a 7900-series guide — that is the wrong target for this card. A recent release is all you need: Mistral Nemo has been mainline in llama.cpp since its July 2024 launch, so there is no PR branch to check out. The GGUF quants are integer formats, so RDNA3's lack of FP8 tensor hardware is irrelevant here.

2. That's it for install — llama.cpp pulls the GGUF straight from Hugging Face at launch (next section). No separate download step.

Option B — Ollama

Ollama is built on llama.cpp and is the fastest way to stand this model up. Install it with the Linux one-liner; Ollama detects the ROCm runtime you installed above and runs the gfx1101 card without any manual architecture flag:

curl -fsSL https://ollama.com/install.sh | sh

Per the Ollama AMD preview blog, all of Ollama's features can be accelerated by AMD graphics cards on Linux. Then either use the curated tag (ollama run mistral-nemo) or pull the community GGUF straight from Hugging Face (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0

Swap the :Q8_0 tag for :Q6_K or :Q5_K_M if you want a smaller footprint. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients and uses the ROCm runtime on the gfx1101 card automatically.

Running

With llama.cpp

Serve an OpenAI-compatible API on port 8000. The -hf flag pulls the GGUF from Hugging Face; append :Q8_0 (case-insensitive) to pick the quant (llama-server docs):

# Q8_0 (recommended), offload all layers to the 7800 XT, low temperature per Mistral's guidance
./build/bin/llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0 \
    --port 8000 \
    -ngl 99 \
    -c 16384 \
    --temp 0.3 \
    --jinja

-ngl 99 (--n-gpu-layers) offloads every layer to the GPU — the dense 12B quant file (13.02GB at Q8_0) sits entirely in the 16GB VRAM.
-c 16384 sets a 16K context. At Q8_0 you have ~3GB free after the weights, which is enough for a solid context; quantize the KV cache (below) to push further.
--temp 0.3 sets the low sampling temperature Mistral recommends for Nemo — leaving it at the usual 0.7 noticeably degrades output. Set it explicitly (many clients default higher).
--jinja applies the GGUF's built-in chat template so the assistant format parses correctly.

Push toward a larger context window. Mistral Nemo advertises a 128K context (max_position_embeddings 131072). At Q8_0 on 16GB the ~3GB left after the weights limits how much f16 KV cache you can hold, so to raise the context you either quantize the KV cache or drop the weight quant. To quantize the cache, add -fa on (Flash Attention, required for a quantized cache) and -ctk q8_0 -ctv q8_0, which roughly halves KV-cache VRAM versus f16 with minimal quality impact:

# Larger context by 8-bit-quantizing the KV cache
./build/bin/llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0 \
    --port 8000 -ngl 99 -c 32768 --temp 0.3 --jinja \
    -fa on -ctk q8_0 -ctv q8_0

A note on Flash Attention on RDNA3. Flash Attention on the ROCm/HIP backend is less mature than on CUDA — if -fa on misbehaves or a quantized KV cache errors on your ROCm version, fall back to an un-quantized cache and a smaller -c, or use the lower-weight-quant route below (which frees VRAM for an f16 cache without needing -fa).

If you want more context headroom without touching the KV cache, drop to Q6_K (10.06GB) or Q5_K_M (8.73GB) — a smaller weight file frees more of the 16GB for the f16 KV cache while staying near-lossless:

./build/bin/llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K \
    --port 8000 -ngl 99 -c 32768 --temp 0.3 --jinja

With Ollama

Pull and run the community GGUF directly from Hugging Face; append a :quant tag to choose the quant (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0

Remember to set a low temperature (~0.3) in your client or Modelfile — Ollama's default sampling can be higher, and Nemo degrades at 0.7. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients and uses the ROCm runtime on the gfx1101 card automatically.

Use it as a chat assistant

Point any OpenAI-compatible chat client at your local endpoint by setting its base URL and a dummy API key — no cloud, no per-token cost.

Open WebUI (optional local chat front-end). A self-hosted, ChatGPT-style UI that talks to any OpenAI-compatible server. Run it and point it at your local endpoint:

# Point Open WebUI at your local llama-server (or Ollama on :11434)
docker run -d -p 3000:8080 \
    -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
    -e OPENAI_API_KEY=EMPTY \
    ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 and chat. (Open WebUI also autodetects a local Ollama install, so with the Ollama path you can skip the base-URL wiring entirely.) Set the temperature to ~0.3 in the model's parameters.

Directly via the API. Any OpenAI SDK or curl works against the same endpoint — use it for scripts, writing tools, or your own app:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "mistral-nemo-12b",
      "temperature": 0.3,
      "messages": [{"role": "user", "content": "Summarize this in three bullet points: ..."}]
    }'

Local servers don't check the key, so any non-empty string (e.g. EMPTY) works where a client requires one.

Results

VRAM usage: The dense 12B loads entirely as its GGUF file — Q8_0 is 13.02GB on disk (byte-verified from the bartowski GGUF tree). On the RX 7800 XT's 16GB that leaves ~3GB for the KV cache — enough for a solid context, and more with an 8-bit-quantized cache (see Running). Q6_K (10.06GB) and Q5_K_M (8.73GB) shrink the footprint further for even larger context. The full-precision f16 GGUF (24.50GB) does not fit 16GB — it needs a 24GB+ card.
Model capability (vendor evals — Mistral's own, NOT hardware throughput): Mistral reports MMLU 68.0% and HellaSwag (0-shot) 83.5%, with strong multilingual results — MMLU French 62.3%, German 62.7%, Spanish 64.6%. These are the vendor's benchmarks, not measurements on this GPU.
Speed: No community throughput benchmark for Mistral Nemo 12B on the RX 7800 XT exists yet — we would rather omit a tok/s figure than invent one or borrow it from different hardware. Live measurements will appear at /check/mistral-nemo-12b/rx-7800-xt once contributed.

For the full benchmark data, see /check/mistral-nemo-12b/rx-7800-xt.

Troubleshooting

Output quality is poor / rambling / incoherent — check the temperature

Mistral recommends a low sampling temperature of ~0.3 for Nemo. The common default of 0.7 noticeably degrades this model's output — if responses feel off, this is the first thing to fix. Set --temp 0.3 on llama-server, or the equivalent temperature parameter in your client / Ollama Modelfile.

The chat template looks wrong / responses are malformed

Pass --jinja to llama-server so the GGUF's built-in chat template is applied — without it the assistant format won't parse. Mistral Nemo uses Mistral's Tekken tokenizer (tekken.json) — it was the first Tekken model. On the Python serving paths that needs mistral-common, but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there.

Ollama or llama.cpp runs on the CPU instead of the GPU

Confirm the ROCm v7 driver is installed (rocm-smi should list the 7800 XT) and that your user is in the render and video groups (groups should show both — log out and back in after the usermod step). Per the Ollama AMD GPU docs, ROCm is a separate install from Ollama; if it's missing, Ollama silently falls back to CPU. For a source llama.cpp build, confirm you compiled with -DGGML_HIP=ON -DGPU_TARGETS=gfx1101. The RX 7800 XT (gfx1101) is on the supported-GPU matrix, so you should not normally need HSA_OVERRIDE_GFX_VERSION — reach for the HSA_OVERRIDE_GFX_VERSION=11.0.0 masquerade only if a specific tool ships gfx1100-only kernels and refuses to start.

Out of memory, or when raising the context

Q8_0 weights (13.02GB) leave ~3GB on a 16GB 7800 XT for the KV cache, so a very long f16 context can OOM. Options, in order: quantize the KV cache with -fa on -ctk q8_0 -ctv q8_0 (roughly halves cache VRAM — note Flash Attention is less mature on ROCm/RDNA3, so verify it behaves on your version); lower -c; or drop to Q6_K (10.06GB) or Q5_K_M (8.73GB) for more headroom. The f16 GGUF (24.50GB) does not fit 16GB at all — it needs a 24GB+ card.

Token generation feels slower than expected — try the Vulkan backend

On RDNA3 the ROCm/HIP backend can be slower at token generation than llama.cpp's Vulkan backend. Per llama.cpp issue #20934, measured on a 7900-series RDNA3 card, the Vulkan (RADV) backend outpaced ROCm for pure generation on a small model across ROCm 6.4.4–7.x. If your generation rate disappoints under ROCm on the 7800 XT, build llama.cpp with -DGGML_VULKAN=ON instead of -DGGML_HIP=ON and re-benchmark with llama-bench — Vulkan often wins for pure generation on RDNA3.

`torch` / CUDA errors — this is llama.cpp on ROCm, not a Python ML stack

Serving Mistral Nemo via llama.cpp or Ollama does not require PyTorch, flash-attn wheels, or a Python ML stack — and it does not use CUDA at all on this card. If you hit a CUDA error, you almost certainly grabbed a CUDA/CPU binary instead of the ROCm build; rebuild with -DGGML_HIP=ON -DGPU_TARGETS=gfx1101 (Option A) or reinstall Ollama with the ROCm driver present. At 12B, Q8_0 is already near-lossless, so there is no reason to reach for the full-precision weights on this card.

Model or GPU 404 on /check

Mistral Nemo 12B is a new addition; if the /check/mistral-nemo-12b/rx-7800-xt link 404s, the catalogue row is still being registered. The recipe's install and run steps are independent of the benchmark endpoint.