How much VRAM does Gemma 4 12B need?

About 24 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Gemma 4 12B on RTX 4090: Local Private Assistant via llama.cpp / Ollama (24GB)

What You'll Build

A fully local, private general assistant: Gemma 4 12B — Google DeepMind's open-weight multimodal generalist (Instruct, release 2026) — served as an OpenAI-compatible endpoint by llama.cpp or Ollama on a single 24GB RTX 4090, then used from a chat UI (Open WebUI is a good local front-end) or directly via the API. This is a general assistant: Q&A, drafting and editing, multi-step reasoning, and — optionally — understanding images and audio you feed it. It's a reasoning-strong 12B that runs on modest hardware; on a 24GB RTX 4090 it's very comfortable, and the same quants reach all the way down to 8GB cards. Everything runs on your own hardware, so prompts, documents, images and audio never leave the machine.

Hardware data: RTX 4090 (24GB VRAM) · Gemma 4 12B, GGUF Q8_0 (12.67GB, recommended) — or Q6_K (9.79GB) / Q4_K_M (7.12GB) / Google's own QAT Q4_0 (6.98GB) for more KV-cache / context headroom · See benchmark data

ℹ️ This is a dense ~12B multimodal generalist — no MoE. Gemma 4 12B is a Gemma4UnifiedForConditionalGeneration (model_type: gemma4_unified) — ~11.95B dense parameters, 48 layers, hidden size 3840, GQA with 16 query / 8 KV heads, head_dim 256. Because it is dense, its footprint is simply the quant file you load plus the KV cache; there is no "active-parameters" shortcut. It uses a unified, encoder-free design: images (raw patches) and audio (waveforms) are projected directly into the decoder rather than through a separate vision/audio encoder. Positioned and used as a general assistant, so we file it under llm.

ℹ️ Multimodal input is optional and needs a separate projector. Gemma 4 accepts text, image, and audio in, text out. The LLM GGUF you load for chat is text-only on its own — to feed it images or audio you also pass a separate mmproj projector GGUF with --mmproj (and use llama-mtmd-cli / the multimodal server path). The mmproj-* file is not the LLM and is excluded from the weight/VRAM math below — if you only need text chat, you don't need it at all.

ℹ️ Very long 256K context, made affordable by sliding-window attention. Gemma 4 advertises a 256K context window (max_position_embeddings 262,144). It uses hybrid attention: interleaved local sliding-window (window 1024) layers plus periodic full global attention (the final layer is always global). Sliding-window attention keeps the KV cache far smaller than a full-attention model at the same length — long context is genuinely cheap here. Even so, the full 256K won't fit on small cards; bound the context (-c) on modest VRAM. On a 24GB 4090 you have ample room for a long context.

ℹ️ Runs on current llama.cpp out of the box. Gemma 4 support landed at the model's launch (~April 2026) and ggml-org ships official GGUFs — there is no special patch or PR gate. Just use a recent llama.cpp (or Ollama) build. Pass --jinja so the embedded chat template applies (it's a complex template that includes a reasoning/thought channel).

Requirements

Component	Minimum	Tested target
GPU	8GB VRAM (QAT Q4_0 / Q4_K_M floor — the matrix reaches down this far)	RTX 4090 (24GB, Ada Lovelace AD102, sm_89)
RAM	16GB system RAM	32GB comfortable
Storage	~7GB (QAT Q4_0) up to ~13GB (Q8_0); +~1GB for the optional mmproj	~13GB for Q8_0
Software	Recent llama.cpp (CUDA) or Ollama; optional Open WebUI chat client	`llama-server`, Open WebUI

Model weights (first-party GGUF available). Unlike many open models, Gemma 4 ships official GGUFs. There are three good sources:

Google's own QAT Q4_0 — google/gemma-4-12b-it-qat-q4_0-gguf is a quantization-aware-trained Q4_0 (6.98GB). Because the model was fine-tuned for this quantization, it delivers noticeably better quality-per-byte than a naive Q4_0 — this is the low-VRAM hero (fits an 8GB card). (The mmproj-* file in that repo is the vision/audio projector, not the LLM.)
ggml-org first-party GGUF — ggml-org/gemma-4-12B-it-GGUF ships Q4_K_M (7.38GB, marginally larger than unsloth's 7.12GB in the table), Q8_0 (12.67GB) and bf16 (23.83GB), plus the mmproj.
Community K_M ladder — unsloth/gemma-4-12b-it-GGUF provides the conventional ladder used in the fit table below.

Byte-verified on-disk sizes (unsloth K_M ladder, plus Google's QAT):

Quant	On-disk size	Fit on RTX 4090 (24GB)
QAT Q4_0 (Google)	6.98GB	Quality-per-byte low-VRAM option — quantization-aware-trained; also fits an 8GB card
Q4_K_M	7.12GB	Tiny footprint — huge KV-cache / context headroom; small enough for an 8GB card
Q5_K_M	8.41GB	Small footprint with a quality bump over Q4
Q6_K	9.79GB	Comfortable — near-lossless-feeling with lots of room for a large KV cache
Q8_0	12.67GB	Recommended — near-lossless weights with ~11GB left for a very large KV cache / long context; the practical best-quality choice on 24GB
bf16	23.83GB	Full precision — fits a 24GB card only tightly (barely, needs a bounded context); not recommended on 24GB

Not model weights — don't count these in the VRAM math:

The mmproj-* file is the multimodal (image/audio) projector, loaded separately with --mmproj only if you want image/audio input. It is not part of the text-chat weights.
Any *-MTP* / mtp-* file is a multi-token-prediction / speculative-decode draft head — not the model weights either.

Licensing. Gemma 4 is Apache-2.0 — free for commercial and non-commercial use, no revenue caps (model card). This is a notable change: earlier Gemma generations (1–3) shipped under the custom "Gemma Terms of Use", and Gemma 4 moved to standard Apache-2.0. Google layers a separate Prohibited Use Policy on top (disallowed use cases apply regardless of the license), but the weights themselves are Apache-2.0.

Installation

You have two GGUF runtimes; pick one. Both are fine for this model — there is no patch requirement — so choose Ollama for the fastest start, or llama.cpp for the most control over context, KV-cache quantization, and multimodal input.

Option A — llama.cpp with CUDA

The RTX 4090 is Ada Lovelace (AD102, sm_89). Build a recent llama.cpp and compile for sm_89, per the official build guide:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# RTX 4090 is Ada Lovelace = compute capability 8.9 (sm_89)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j 8

A recent release is all you need — Gemma 4 has been mainline in llama.cpp since its launch. If you prefer a prebuilt binary, grab a current one from the releases page. The CUDA backend flag is -DGGML_CUDA=ON on current llama.cpp (the old LLAMA_CUDA name was retired in late 2024); install the NVIDIA CUDA toolkit first.

Option B — Ollama

Ollama is built on llama.cpp and is the fastest way to stand this model up. Either use the curated tag (ollama run gemma4:12b, if listed) or pull the community GGUF straight from Hugging Face (HF × Ollama docs):

ollama run hf.co/unsloth/gemma-4-12b-it-GGUF:Q8_0

Swap the :Q8_0 tag for :Q6_K, :Q5_K_M or :Q4_K_M if you want an even smaller footprint. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Running

With llama.cpp

Serve an OpenAI-compatible API on port 8000. The -hf flag pulls the GGUF from Hugging Face; append :Q8_0 (case-insensitive) to pick the quant (llama-server docs):

# Q8_0 (recommended), offload all layers to the 4090
llama-server -hf ggml-org/gemma-4-12B-it-GGUF:Q8_0 \
    --port 8000 \
    -ngl 99 \
    -c 16384 \
    --jinja

-ngl 99 (--n-gpu-layers) offloads every layer to the GPU — the dense 12B quant file (12.67GB at Q8_0) sits entirely in VRAM with room to spare.
-c 16384 sets a 16K context. At Q8_0 you have ~11GB free after the weights, and Gemma's sliding-window attention keeps the KV cache modest, so you can raise this a lot.
--jinja applies the GGUF's built-in chat template so the assistant format parses correctly — Gemma 4's template is complex (it includes a reasoning/thought channel), so this flag matters.

Push toward the 256K context window. Gemma 4 advertises a 256K context (max_position_embeddings 262,144), and its interleaved sliding-window attention (window 1024) + periodic global attention makes long context far cheaper in KV cache than a full-attention model of the same size. At Q8_0 on 24GB you have plenty of room; you can go further by quantizing the KV cache — add -fa on (Flash Attention, required for a quantized cache) and -ctk q8_0 -ctv q8_0, which roughly halves KV-cache VRAM versus f16 with minimal quality impact:

# Longer context by 8-bit-quantizing the KV cache
llama-server -hf ggml-org/gemma-4-12B-it-GGUF:Q8_0 \
    --port 8000 -ngl 99 -c 131072 --jinja \
    -fa on -ctk q8_0 -ctv q8_0

The full 256K won't fit on small cards even with SWA — bound -c there — but on a 24GB 4090 a very long context is comfortable. Because Gemma 4 12B is only ~12B, you have generous headroom on a 24GB card; this same model also fits far smaller GPUs (QAT Q4_0 at 6.98GB or Q4_K_M at 7.12GB run on an 8GB card), so the matrix reaches well below this starter tier.

Optional: image and audio input. To use Gemma 4's multimodal side, add the projector with --mmproj (download the mmproj-* file from the same GGUF repo) and serve via the multimodal path — for the CLI, llama-mtmd-cli is the multimodal front-end:

# Multimodal: LLM weights + the separate projector (mmproj)
llama-mtmd-cli -hf ggml-org/gemma-4-12B-it-GGUF:Q8_0 \
    --mmproj <path-to-mmproj-gguf> \
    -ngl 99 --jinja

The mmproj is a small extra file (~1GB) on top of the quant sizes above — only load it if you actually want to pass images or audio; text chat doesn't need it.

With Ollama

Pull and run the community GGUF directly from Hugging Face; append a :quant tag to choose the quant (HF × Ollama docs):

ollama run hf.co/unsloth/gemma-4-12b-it-GGUF:Q8_0

Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients.

Use it as a chat assistant

Point any OpenAI-compatible chat client at your local endpoint by setting its base URL and a dummy API key — no cloud, no per-token cost.

Open WebUI (optional local chat front-end). A self-hosted, ChatGPT-style UI that talks to any OpenAI-compatible server. Run it and point it at your local endpoint:

# Point Open WebUI at your local llama-server (or Ollama on :11434)
docker run -d -p 3000:8080 \
    -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
    -e OPENAI_API_KEY=EMPTY \
    ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 and chat. (Open WebUI also autodetects a local Ollama install, so with the Ollama path you can skip the base-URL wiring entirely.)

Directly via the API. Any OpenAI SDK or curl works against the same endpoint — use it for scripts, writing tools, or your own app:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "gemma-4-12b",
      "messages": [{"role": "user", "content": "Summarize this in three bullet points: ..."}]
    }'

Local servers don't check the key, so any non-empty string (e.g. EMPTY) works where a client requires one.

Results

VRAM usage: The dense ~12B loads entirely as its GGUF file — Q8_0 is 12.67GB on disk (byte-verified from the ggml-org GGUF tree). On the RTX 4090's 24GB that leaves ~11GB for the KV cache — plenty for a long context, and even more thanks to Gemma's sliding-window attention (and further with an 8-bit-quantized cache; see Running). Q6_K (9.79GB), Q5_K_M (8.41GB), Q4_K_M (7.12GB) and Google's QAT Q4_0 (6.98GB) shrink the footprint further for even larger context or smaller cards. The full-precision bf16 GGUF (23.83GB) fits a 24GB card only tightly, with almost no room for the KV cache — not recommended on 24GB.
Model capability (vendor evals — Google's own, NOT hardware throughput): Google reports MMLU Pro 77.2%, MMMLU 83.4%, GPQA Diamond 78.8%, AIME 2026 77.5%, LiveCodeBench v6 72.0%, and MMMU Pro (vision) 69.1% — a reasoning-strong card for its size. These are the vendor's benchmarks, not measurements on this GPU.
Speed: No community throughput benchmark for Gemma 4 12B on the RTX 4090 exists yet — we would rather omit a tok/s figure than invent one or borrow it from different hardware. Live measurements will appear at /check/gemma-4-12b/rtx-4090 once contributed.

For the full benchmark data, see /check/gemma-4-12b/rtx-4090.

Troubleshooting

The chat template looks wrong / responses are malformed

Pass --jinja to llama-server so the GGUF's built-in chat template is applied — without it the assistant format won't parse. Gemma 4's chat template is complex (it includes a reasoning/thought channel), so applying it correctly matters more than for a plain instruct model. Use a recent llama.cpp build so the template is fully supported.

Images or audio aren't recognized

The plain LLM GGUF is text-only. To pass images or audio you must also load the separate mmproj projector with --mmproj and use the multimodal path (llama-mtmd-cli, or the multimodal server). Download the mmproj-* file from the same GGUF repo — it is a distinct file from the quant, and text chat works fine without it.

Out of memory, or when raising the context

Q8_0 weights (12.67GB) leave ~11GB on a 24GB 4090 for the KV cache, and Gemma's sliding-window attention keeps that cache smaller than a full-attention model would — so OOM is unlikely at sane context sizes. But the full 256K can still be large. Options, in order: quantize the KV cache with -fa on -ctk q8_0 -ctv q8_0 (roughly halves cache VRAM); lower -c; or drop to Q6_K (9.79GB), Q4_K_M (7.12GB) or Google's QAT Q4_0 (6.98GB) for even more headroom. Avoid the bf16 GGUF (23.83GB) on 24GB — it barely fits the weights with almost no room for a KV cache.

`torch` / CUDA errors — this is llama.cpp, not a Python ML stack

Serving Gemma 4 via llama.cpp or Ollama does not require PyTorch, flash-attn wheels, or a Python ML stack. If you hit a CUDA error, confirm you built (or downloaded) the CUDA-enabled llama.cpp (Option A, -DGGML_CUDA=ON) rather than a CPU-only binary. For large-VRAM or multi-GPU production serving you could instead run the full-precision weights under a server like vLLM, but on a single 4090 the GGUF + llama.cpp path is the right one — and at 12B, Q8_0 is already near-lossless.

Model or GPU 404 on /check

Gemma 4 12B is a new addition; if the /check/gemma-4-12b/rtx-4090 link 404s, the catalogue row is still being registered. The recipe's install and run steps are independent of the benchmark endpoint.