Mistral Nemo 12B on RTX 4070: Local Private Assistant via llama.cpp / Ollama (12GB)

What You'll Build

A fully local, private general assistant: Mistral Nemo 12B — Mistral AI and NVIDIA's Apache-2.0 generalist (Instruct, release 2407) — served as an OpenAI-compatible endpoint by llama.cpp or Ollama on a single 12GB RTX 4070, then used from a chat UI (Open WebUI is a good local front-end) or directly via the API. This is a text-only chat/reasoning/writing model: general Q&A, drafting and editing, multi-step reasoning, function calling, and strong multilingual support. Positioned as a drop-in upgrade to Mistral 7B, it's a capable 12B that runs on modest hardware — on a 12GB RTX 4070 it sits comfortably in the mid tier, taking a near-lossless-feeling Q6_K with ~2GB left for the KV cache. Everything runs on your own hardware, so prompts and documents never leave the machine.

Hardware data: RTX 4070 (12GB VRAM) · Mistral Nemo 12B, GGUF Q6_K (10.06GB, recommended) — or Q5_K_M (8.73GB) / Q4_K_M (7.48GB) for even more KV-cache / context headroom · See benchmark data

ℹ️ This is a dense, text-only 12B generalist — no MoE, no vision. Mistral Nemo is a MistralForCausalLM (model_type: mistral) — 40 layers, hidden size 5120, GQA with 32 query / 8 KV heads, head_dim 128. Because it is dense, its footprint is simply the quant file you load plus the KV cache; there is no "active-parameters" shortcut. It is a pure text model — there is no vision tower and no image input. Context window is 128K (max_position_embeddings 131072). It was the first model to use Mistral's Tekken tokenizer (tekken.json), which needs mistral-common on the Python serving paths — but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there. Nemo was trained with quantization awareness for FP8 inference and tuned for function calling and multilingual use.

ℹ️ Runs on current llama.cpp out of the box. Mistral Nemo shipped in July 2024 and has been long supported — there is no special patch or PR gate. Just use a recent llama.cpp (or Ollama) build. Pass --jinja so the embedded chat template applies.

⚠️ Use a low sampling temperature (~0.3). Mistral recommends a low temperature (~0.3) for Nemo; the usual default of 0.7 noticeably degrades output quality on this model. Set it explicitly — this is a real, easy-to-miss gotcha.

Requirements

Component	Minimum	Tested target
GPU	8GB VRAM (Q4_K_M floor — the matrix reaches down this far)	RTX 4070 (12GB, Ada Lovelace AD104, sm_89)
RAM	16GB system RAM	32GB comfortable
Storage	~7.5GB (Q4_K_M) up to ~10GB (Q6_K)	~10GB for Q6_K
Software	Recent llama.cpp (CUDA) or Ollama; optional Open WebUI chat client	llama-server, Open WebUI

Model weights (community GGUF — there is NO first-party GGUF). Mistral publishes only the full-precision weights (mistralai/Mistral-Nemo-Instruct-2407); the model is quantized to GGUF by the community. Primary source is bartowski/Mistral-Nemo-Instruct-2407-GGUF; unsloth/Mistral-Nemo-Instruct-2407-GGUF is a good alternative that also ships smaller Q2_K / Q3_K_M quants. Byte-verified on-disk sizes (bartowski):

Quant	On-disk size	Fit on RTX 4070 (12GB)
Q4_K_M	7.48GB	Roomy — lots of KV-cache / context headroom; also the quant that fits an 8GB card
Q5_K_M	8.73GB	Comfortable — ~3GB left for a generous KV cache
Q6_K	10.06GB	Recommended — near-lossless-feeling quality with ~2GB left for the KV cache; the practical best-quality choice on 12GB
Q8_0	13.02GB	Does NOT fit 12GB — its 13.02GB of weights alone exceed the card; use Q6_K here, or step up to a 16GB card for Q8_0

Not model weights — don't count this in the VRAM math:

• The .imatrix (~7 MB) is calibration data used to produce the quants — never load it as a model.

Licensing. Mistral Nemo 12B is Apache-2.0 — free for commercial and non-commercial use, no revenue caps (model card).

Installation

You have two GGUF runtimes; pick one. Both are fine for this model — there is no patch requirement — so choose Ollama for the fastest start, or llama.cpp for the most control over context and KV-cache quantization.

Option A — llama.cpp with CUDA

The RTX 4070 is Ada Lovelace (AD104, sm_89). Build a recent llama.cpp and compile for sm_89, per the official build guide:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# RTX 4070 is Ada Lovelace = compute capability 8.9 (sm_89)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j 8

A recent release is all you need — Mistral Nemo has been mainline in llama.cpp since its July 2024 launch. If you prefer a prebuilt binary, grab a current one from the releases page. The CUDA backend flag is -DGGML_CUDA=ON on current llama.cpp (the old LLAMA_CUDA name was retired in late 2024); install the NVIDIA CUDA toolkit first.

Option B — Ollama

Ollama is built on llama.cpp and is the fastest way to stand this model up. Either use the curated tag (ollama run mistral-nemo) or pull the community GGUF straight from Hugging Face (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K

Swap the :Q6_K tag for :Q5_K_M or :Q4_K_M if you want an even smaller footprint. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Running

With llama.cpp

Serve an OpenAI-compatible API on port 8000. The -hf flag pulls the GGUF from Hugging Face; append :Q6_K (case-insensitive) to pick the quant (llama-server docs):

# Q6_K (recommended), offload all layers to the 4070, low temperature per Mistral's guidance
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K \
    --port 8000 \
    -ngl 99 \
    -c 8192 \
    --temp 0.3 \
    --jinja

• -ngl 99 (--n-gpu-layers) offloads every layer to the GPU — the dense 12B quant file (10.06GB at Q6_K) sits entirely in VRAM.
• -c 8192 sets an 8K context. At Q6_K you have ~2GB free after the weights; raise it as your prompts need, and quantize the KV cache (below) to push further.
• --temp 0.3 sets the low sampling temperature Mistral recommends for Nemo — leaving it at the usual 0.7 noticeably degrades output. Set it explicitly (many clients default higher).
• --jinja applies the GGUF's built-in chat template so the assistant format parses correctly.

Longer context. Mistral Nemo advertises a 128K context (max_position_embeddings 131072). At Q6_K on 12GB the ~2GB of headroom limits how far you can go, but you can stretch it by quantizing the KV cache: add -fa on (Flash Attention, required for a quantized cache) and -ctk q8_0 -ctv q8_0, which roughly halves KV-cache VRAM versus f16 with minimal quality impact. For a lot more context room, drop to Q5_K_M (8.73GB) or Q4_K_M (7.48GB):

# More context headroom on 12GB via a smaller quant + quantized KV cache
llama-server -hf bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q5_K_M \
    --port 8000 -ngl 99 -c 32768 --temp 0.3 --jinja \
    -fa on -ctk q8_0 -ctv q8_0

Because Nemo is only 12B, the 12GB RTX 4070 is a comfortable mid tier — enough for a near-lossless-feeling Q6_K, while the same model also fits far smaller GPUs (Q4_K_M at 7.48GB runs on an 8GB card) and takes a full Q8_0 on 16GB+.

With Ollama

Pull and run the community GGUF directly from Hugging Face; append a :quant tag to choose the quant (HF × Ollama docs):

ollama run hf.co/bartowski/Mistral-Nemo-Instruct-2407-GGUF:Q6_K

Remember to set a low temperature (~0.3) in your client or Modelfile — Ollama's default sampling can be higher, and Nemo degrades at 0.7. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Use it as a chat assistant

Point any OpenAI-compatible chat client at your local endpoint by setting its base URL and a dummy API key — no cloud, no per-token cost.

Open WebUI (optional local chat front-end). A self-hosted, ChatGPT-style UI that talks to any OpenAI-compatible server. Run it and point it at your local endpoint:

# Point Open WebUI at your local llama-server (or Ollama on :11434)
docker run -d -p 3000:8080 \
    -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
    -e OPENAI_API_KEY=EMPTY \
    ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 and chat. (Open WebUI also autodetects a local Ollama install, so with the Ollama path you can skip the base-URL wiring entirely.) Set the temperature to ~0.3 in the model's parameters.

Directly via the API. Any OpenAI SDK or curl works against the same endpoint — use it for scripts, writing tools, or your own app:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "mistral-nemo-12b",
      "temperature": 0.3,
      "messages": [{"role": "user", "content": "Summarize this in three bullet points: ..."}]
    }'

Local servers don't check the key, so any non-empty string (e.g. EMPTY) works where a client requires one.

Results

• VRAM usage: The dense 12B loads entirely as its GGUF file — Q6_K is 10.06GB on disk (byte-verified from the bartowski GGUF tree). On the RTX 4070's 12GB that leaves ~2GB for the KV cache — enough for a solid context, and more with an 8-bit-quantized cache (see Running). Q5_K_M (8.73GB) and Q4_K_M (7.48GB) shrink the footprint further for larger context. Q8_0 (13.02GB) does not fit 12GB — its weights alone exceed the card; that quant needs a 16GB+ GPU.
• Model capability (vendor evals — Mistral's own, NOT hardware throughput): Mistral reports MMLU 68.0% and HellaSwag (0-shot) 83.5%, with strong multilingual results — MMLU French 62.3%, German 62.7%, Spanish 64.6%. These are the vendor's benchmarks, not measurements on this GPU.
• Speed: No community throughput benchmark for Mistral Nemo 12B on the RTX 4070 exists yet — we would rather omit a tok/s figure than invent one or borrow it from different hardware. Live measurements will appear at /check/mistral-nemo-12b/rtx-4070 once contributed.

For the full benchmark data, see /check/mistral-nemo-12b/rtx-4070.

Troubleshooting

Output quality is poor / rambling / incoherent — check the temperature

Mistral recommends a low sampling temperature of ~0.3 for Nemo. The common default of 0.7 noticeably degrades this model's output — if responses feel off, this is the first thing to fix. Set --temp 0.3 on llama-server, or the equivalent temperature parameter in your client / Ollama Modelfile.

The chat template looks wrong / responses are malformed

Pass --jinja to llama-server so the GGUF's built-in chat template is applied — without it the assistant format won't parse. Mistral Nemo uses Mistral's Tekken tokenizer (tekken.json) — it was the first Tekken model. On the Python serving paths that needs mistral-common, but the GGUF / llama.cpp path uses the embedded tokenizer, so no extra install is required there.

Out of memory, or when raising the context

Q6_K weights (10.06GB) leave ~2GB on a 12GB 4070 for the KV cache, so OOM is most likely when you push -c high. Options, in order: quantize the KV cache with -fa on -ctk q8_0 -ctv q8_0 (roughly halves cache VRAM); lower -c; or drop to Q5_K_M (8.73GB) or Q4_K_M (7.48GB) for a lot more headroom. Q8_0 (13.02GB) will not load on 12GB — the weights exceed the card before any KV cache.

torch / CUDA errors — this is llama.cpp, not a Python ML stack

Serving Mistral Nemo via llama.cpp or Ollama does not require PyTorch, flash-attn wheels, or a Python ML stack. If you hit a CUDA error, confirm you built (or downloaded) the CUDA-enabled llama.cpp (Option A, -DGGML_CUDA=ON) rather than a CPU-only binary. On a 12GB card the GGUF + llama.cpp path is the right one — and at 12B, Q6_K is already near-lossless-feeling.

Model or GPU 404 on /check

Mistral Nemo 12B is a new addition; if the /check/mistral-nemo-12b/rtx-4070 link 404s, the catalogue row is still being registered. The recipe's install and run steps are independent of the benchmark endpoint.