Mistral Nemo 12B on RTX 4080: Local Private Assistant via llama.cpp / Ollama (16GB)

What You'll Build

A fully local, private general assistant: Mistral Nemo 12B — Mistral AI and NVIDIA's Apache-2.0 generalist (Instruct, release 2407) — served as an OpenAI-compatible endpoint by llama.cpp or Ollama on a single 16GB RTX 4080, then used from a chat UI (Open WebUI is a good local front-end) or directly via the API. This is a text-only chat/reasoning/writing model: general Q&A, drafting and editing, multi-step reasoning, function calling, and strong multilingual support. Positioned as a drop-in upgrade to Mistral 7B, it's a capable 12B that runs on modest hardware — on a 16GB RTX 4080 it reaches the quality tier, running a near-lossless Q8_0 with ~3GB left for the KV cache. Everything runs on your own hardware, so prompts and documents never leave the machine.

Hardware data: RTX 4080 (16GB VRAM) · Mistral Nemo 12B, GGUF Q8_0 (13.02GB, recommended — near-lossless) — or Q6_K (10.06GB) / Q4_K_M (7.48GB) for even more KV-cache / context headroom · See benchmark data

ℹ️ This is a dense, text-only 12B generalist — no MoE, no vision. Mistral Nemo is a MistralForCausalLM (model_type: mistral) — 40 layers, hidden size 5120, GQA with 32 query / 8 KV heads, head_dim 128. Because it is dense, its footprint is simply the quant file you load plus the KV cache; there is no "active-parameters" shortcut. It is a pure text model — there is no vision tower and no image input. Context window is 128K (max_position_embeddings 131072). It was the first model to use Mistral's Tekken tokenizer (tekken.json), which needs mistral-common on the Python serving paths — but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there. Nemo was trained with quantization awareness for FP8 inference and tuned for function calling and multilingual use.

ℹ️ Runs on current llama.cpp out of the box. Mistral Nemo shipped in July 2024 and has been long supported — there is no special patch or PR gate. Just use a recent llama.cpp (or Ollama) build. Pass --jinja so the embedded chat template applies.

⚠️ Use a low sampling temperature (~0.3). Mistral recommends a low temperature (~0.3) for Nemo; the usual default of 0.7 noticeably degrades output quality on this model. Set it explicitly — this is a real, easy-to-miss gotcha.

Requirements

Component	Minimum	Tested target
GPU	8GB VRAM (Q4_K_M floor — the matrix reaches down this far)	RTX 4080 (16GB, Ada Lovelace AD103, sm_89)
RAM	16GB system RAM	32GB comfortable
Storage	~7.5GB (Q4_K_M) up to ~13GB (Q8_0)	~13GB for Q8_0
Software	Recent llama.cpp (CUDA) or Ollama; optional Open WebUI chat client	llama-server, Open WebUI

Model weights (community GGUF — there is NO first-party GGUF). Mistral publishes only the full-precision weights (mistralai/Mistral-Nemo-Instruct-2407); the model is quantized to GGUF by the community. Primary source is bartowski/Mistral-Nemo-Instruct-2407-GGUF; unsloth/Mistral-Nemo-Instruct-2407-GGUF is a good alternative that also ships smaller Q2_K / Q3_K_M quants. Byte-verified on-disk sizes (bartowski):

Quant	On-disk size	Fit on RTX 4080 (16GB)
Q4_K_M	7.48GB	Roomy — lots of KV-cache / context headroom; also the quant that fits an 8GB card
Q6_K	10.06GB	Comfortable — ~6GB left for a large KV cache
Q8_0	13.02GB	Recommended — near-lossless weights with ~3GB left for the KV cache; the practical best-quality choice on 16GB
f16	24.50GB	Full precision — does NOT fit 16GB; at 12B, Q8_0 is already near-lossless, so f16 buys nothing here and needs a much larger card

Not model weights — don't count this in the VRAM math:

• The .imatrix (~7 MB) is calibration data used to produce the quants — never load it as a model.

Licensing. Mistral Nemo 12B is Apache-2.0 — free for commercial and non-commercial use, no revenue caps (model card).

Installation

You have two GGUF runtimes; pick one. Both are fine for this model — there is no patch requirement — so choose Ollama for the fastest start, or llama.cpp for the most control over context and KV-cache quantization.

Option A — llama.cpp with CUDA

The RTX 4080 is Ada Lovelace (AD103, sm_89). Build a recent llama.cpp and compile for sm_89, per the official build guide:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# RTX 4080 is Ada Lovelace = compute capability 8.9 (sm_89)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j 8

A recent release is all you need — Mistral Nemo has been mainline in llama.cpp since its July 2024 launch. If you prefer a prebuilt binary, grab a current one from the releases page. The CUDA backend flag is -DGGML_CUDA=ON on current llama.cpp (the old LLAMA_CUDA name was retired in late 2024); install the NVIDIA CUDA toolkit first.

Option B — Ollama

Ollama is built on llama.cpp and is the fastest way to stand this model up. Either use the curated tag (ollama run mistral-nemo) or pull the community GGUF straight from Hugging Face (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0

Swap the :Q8_0 tag for :Q6_K or :Q4_K_M if you want an even smaller footprint. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Running

With llama.cpp

Serve an OpenAI-compatible API on port 8000. The -hf flag pulls the GGUF from Hugging Face; append :Q8_0 (case-insensitive) to pick the quant (llama-server docs):

# Q8_0 (recommended, near-lossless), offload all layers to the 4080, low temperature per Mistral's guidance
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0 \
    --port 8000 \
    -ngl 99 \
    -c 8192 \
    --temp 0.3 \
    --jinja

• -ngl 99 (--n-gpu-layers) offloads every layer to the GPU — the dense 12B quant file (13.02GB at Q8_0) sits entirely in VRAM.
• -c 8192 sets an 8K context. At Q8_0 you have ~3GB free after the weights; raise it as your prompts need, and quantize the KV cache (below) to push further.
• --temp 0.3 sets the low sampling temperature Mistral recommends for Nemo — leaving it at the usual 0.7 noticeably degrades output. Set it explicitly (many clients default higher).
• --jinja applies the GGUF's built-in chat template so the assistant format parses correctly.

Longer context. Mistral Nemo advertises a 128K context (max_position_embeddings 131072). At Q8_0 on 16GB the ~3GB of headroom supports a healthy context, and you can stretch it by quantizing the KV cache: add -fa on (Flash Attention, required for a quantized cache) and -ctk q8_0 -ctv q8_0, which roughly halves KV-cache VRAM versus f16 with minimal quality impact. For a lot more context room, drop to Q6_K (10.06GB) or Q4_K_M (7.48GB):

# Longer context by 8-bit-quantizing the KV cache (optionally on a smaller quant for even more room)
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0 \
    --port 8000 -ngl 99 -c 32768 --temp 0.3 --jinja \
    -fa on -ctk q8_0 -ctv q8_0

Because Nemo is only 12B, the 16GB RTX 4080 comfortably runs the near-lossless Q8_0 — the quality tier for this model — while the same model also fits far smaller GPUs (Q4_K_M at 7.48GB runs on an 8GB card).

With Ollama

Pull and run the community GGUF directly from Hugging Face; append a :quant tag to choose the quant (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q8_0

Remember to set a low temperature (~0.3) in your client or Modelfile — Ollama's default sampling can be higher, and Nemo degrades at 0.7. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Use it as a chat assistant

Point any OpenAI-compatible chat client at your local endpoint by setting its base URL and a dummy API key — no cloud, no per-token cost.

Open WebUI (optional local chat front-end). A self-hosted, ChatGPT-style UI that talks to any OpenAI-compatible server. Run it and point it at your local endpoint:

# Point Open WebUI at your local llama-server (or Ollama on :11434)
docker run -d -p 3000:8080 \
    -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
    -e OPENAI_API_KEY=EMPTY \
    ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 and chat. (Open WebUI also autodetects a local Ollama install, so with the Ollama path you can skip the base-URL wiring entirely.) Set the temperature to ~0.3 in the model's parameters.

Directly via the API. Any OpenAI SDK or curl works against the same endpoint — use it for scripts, writing tools, or your own app:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "mistral-nemo-12b",
      "temperature": 0.3,
      "messages": [{"role": "user", "content": "Summarize this in three bullet points: ..."}]
    }'

Local servers don't check the key, so any non-empty string (e.g. EMPTY) works where a client requires one.

Results

• VRAM usage: The dense 12B loads entirely as its GGUF file — Q8_0 is 13.02GB on disk (byte-verified from the bartowski GGUF tree). On the RTX 4080's 16GB that leaves ~3GB for the KV cache — enough for a healthy context, and more with an 8-bit-quantized cache (see Running). Q6_K (10.06GB) and Q4_K_M (7.48GB) shrink the footprint further for larger context. The full-precision f16 GGUF (24.50GB) does not fit 16GB — and at 12B, Q8_0 is already near-lossless, so f16 buys nothing on this card.
• Model capability (vendor evals — Mistral's own, NOT hardware throughput): Mistral reports MMLU 68.0% and HellaSwag (0-shot) 83.5%, with strong multilingual results — MMLU French 62.3%, German 62.7%, Spanish 64.6%. These are the vendor's benchmarks, not measurements on this GPU.
• Speed: No community throughput benchmark for Mistral Nemo 12B on the RTX 4080 exists yet — we would rather omit a tok/s figure than invent one or borrow it from different hardware. Live measurements will appear at /check/mistral-nemo-12b/rtx-4080 once contributed.

For the full benchmark data, see /check/mistral-nemo-12b/rtx-4080.

Troubleshooting

Output quality is poor / rambling / incoherent — check the temperature

Mistral recommends a low sampling temperature of ~0.3 for Nemo. The common default of 0.7 noticeably degrades this model's output — if responses feel off, this is the first thing to fix. Set --temp 0.3 on llama-server, or the equivalent temperature parameter in your client / Ollama Modelfile.

The chat template looks wrong / responses are malformed

Pass --jinja to llama-server so the GGUF's built-in chat template is applied — without it the assistant format won't parse. Mistral Nemo uses Mistral's Tekken tokenizer (tekken.json) — it was the first Tekken model. On the Python serving paths that needs mistral-common, but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there.

Out of memory, or when raising the context

Q8_0 weights (13.02GB) leave ~3GB on a 16GB 4080 for the KV cache, so OOM is most likely when you push -c high. Options, in order: quantize the KV cache with -fa on -ctk q8_0 -ctv q8_0 (roughly halves cache VRAM); lower -c; or drop to Q6_K (10.06GB) or Q4_K_M (7.48GB) for a lot more headroom. Avoid the f16 GGUF (24.50GB) — it does not fit 16GB, and Q8_0 is already near-lossless.

torch / CUDA errors — this is llama.cpp, not a Python ML stack

Serving Mistral Nemo via llama.cpp or Ollama does not require PyTorch, flash-attn wheels, or a Python ML stack. If you hit a CUDA error, confirm you built (or downloaded) the CUDA-enabled llama.cpp (Option A, -DGGML_CUDA=ON) rather than a CPU-only binary. For large-VRAM or multi-GPU production serving you could instead run the full-precision weights under a server like vLLM, but on a single 4080 the GGUF + llama.cpp path is the right one — and at 12B, Q8_0 is already near-lossless.

Model or GPU 404 on /check

Mistral Nemo 12B is a new addition; if the /check/mistral-nemo-12b/rtx-4080 link 404s, the catalogue row is still being registered. The recipe's install and run steps are independent of the benchmark endpoint.