How much VRAM does Phi-4 need?

About 16 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Phi-4 (14B) on RTX 4080: Local Private Assistant via llama.cpp / Ollama (16GB)

What You'll Build

A fully local, private general assistant: Phi-4 — Microsoft's flagship 14B of the Phi-4 family (released December 2024) — served as an OpenAI-compatible endpoint by llama.cpp or Ollama on a single 16GB RTX 4080, then used from a chat UI (Open WebUI is a good local front-end) or directly via the API. This is a text-only chat/reasoning/writing model with a reputation for strong STEM, math, and multi-step reasoning for its size — the result of heavy training on curated and synthetic data. On a 16GB RTX 4080 it runs comfortably at Q6_K with room for the full-context KV cache; Q8_0 also fits but is tight. Everything runs on your own hardware, so prompts and documents never leave the machine.

Hardware data: RTX 4080 (16GB VRAM) · Phi-4, GGUF Q6_K (12.03GB, recommended) — or Q8_0 (15.58GB, tight) / Q5_K_M (10.41GB) / Q4_K_M (8.89GB) for more headroom · See benchmark data

ℹ️ This is a dense, text-only 14B generalist — no MoE, no vision. Phi-4 is a Phi3ForCausalLM (model_type: phi3) — 40 layers, hidden size 5120, GQA with 40 query / 10 KV heads. Because it is dense, its footprint is simply the quant file you load plus the KV cache; there is no "active-parameters" shortcut. It is a pure text model — there is no vision tower and no image input. This is the generalist Phi-4 — not Phi-4-reasoning / -reasoning-plus (separate reasoning-RL variants) and not Phi-4-mini; don't conflate them.

⚠️ Context window is only 16K tokens (16,384). Phi-4's max_position_embeddings is 16,384 — notably shorter than most current peers, which often reach 128K. This caps long-document and long-conversation use: keep inputs within ~16K tokens. On a 16GB card the KV cache (~3.3GB at f16 for full 16K) is a real budget line — with Q6_K weights you have room for it at f16, but with Q8_0 you'll want to quantize it (see the fit note below).

ℹ️ Runs on current llama.cpp out of the box. Phi-4 uses the long-supported phi3 architecture — there is no special patch or PR gate. Just use a recent llama.cpp (or Ollama) build together with a recent GGUF (see the template/EOS note below), and pass --jinja so the embedded chat template applies. Phi-4 uses the <|im_start|>role<|im_sep|>…<|im_end|> chat template (not Tekken).

⚠️ Early-2025 Phi-4 GGUFs had a chat-template / EOS bug. The first GGUFs (January 2025) shipped with the wrong end-of-turn token — EOS was set to <|endoftext|> instead of <|im_end|>, which causes runaway / garbled generation that won't stop cleanly. It is fixed in current llama.cpp and current GGUFs — unsloth documented the fix and re-uploaded corrected files. If you grabbed an early build, re-download a current GGUF and update llama.cpp. This is the single most common Phi-4 gotcha; see Troubleshooting.

Requirements

Component	Minimum	Tested target
GPU	12GB VRAM (Q4_K_M is 8.89GB — the 14B does not fit an 8GB card)	RTX 4080 (16GB, Ada Lovelace AD103, sm_89)
RAM	16GB system RAM	32GB comfortable
Storage	~12GB (Q6_K) up to ~15.6GB (Q8_0)	~12GB for Q6_K
Software	Recent llama.cpp (CUDA) or Ollama + a recent GGUF; optional Open WebUI chat client	`llama-server`, Open WebUI

Model weights (first-party GGUF exists). Microsoft publishes both the full-precision weights (microsoft/phi-4) and an official GGUF (microsoft/phi-4-gguf, MIT). Note a naming quirk in Microsoft's repo: it names its K-quants Q4_K / Q5_K (not Q4_K_M / Q5_K_M) — sizes there are Q4_K 9.05GB, Q4_K_S 8.44GB, Q5_K 10.60GB, Q6_K 12.03GB, Q8_0 15.58GB, BF16 29.32GB. For most users we recommend the community unsloth/phi-4-GGUF (MIT), which ships the conventional K_M ladder with the documented chat-template fix already applied; bartowski/phi-4-GGUF offers imatrix quants too. Byte-verified on-disk sizes (unsloth K_M ladder):

Quant	On-disk size	Fit on RTX 4080 (16GB)
Q4_K_M	8.89GB	Small footprint, lots of headroom — also the practical floor for a 12GB card (too big for 8GB)
Q5_K_M	10.41GB	Comfortable — plenty of room for a full-16K f16 KV cache
Q6_K	12.03GB	Recommended — 12.03GB + full-16K f16 KV ~3.3GB ≈ 15.4GB fits 16GB with a little room; near-lossless-feeling
Q8_0	15.58GB	Tight — 15.58GB + ~3.3GB f16 KV would overflow 16GB. It fits only with a quantized KV cache (`-fa on -ctk q8_0 -ctv q8_0` → ~1.7GB) and possibly reduced context. Prefer Q6_K here unless you specifically want Q8_0
F16 / BF16	29.32GB	Full precision — does not fit a 16GB card (needs 32GB+)

Licensing. Phi-4 is MIT — free for commercial and non-commercial use, no revenue caps (model card).

Installation

You have two GGUF runtimes; pick one. Both are fine for this model — there is no patch requirement — so choose Ollama for the fastest start, or llama.cpp for the most control. Either way, use a recent build and a recent GGUF so the chat-template / EOS fix is in place.

Option A — llama.cpp with CUDA

The RTX 4080 is Ada Lovelace (AD103, sm_89). Build a recent llama.cpp and compile for sm_89, per the official build guide:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# RTX 4080 is Ada Lovelace = compute capability 8.9 (sm_89)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j 8

A recent release matters here: it carries the corrected Phi-4 chat template / EOS handling. If you prefer a prebuilt binary, grab a current one from the releases page. The CUDA backend flag is -DGGML_CUDA=ON on current llama.cpp (the old LLAMA_CUDA name was retired in late 2024); install the NVIDIA CUDA toolkit first.

Option B — Ollama

Ollama is built on llama.cpp and is the fastest way to stand this model up. Either use the curated tag (ollama run phi4) or pull the community GGUF straight from Hugging Face (HF × Ollama docs):

ollama run hf.co/unsloth/phi-4-GGUF:Q6_K

Swap the :Q6_K tag for :Q5_K_M or :Q4_K_M if you want a smaller footprint, or :Q8_0 if you want maximum weight fidelity (tight on 16GB — see the fit note). Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients. Prefer a recent Ollama version so the current, fixed template is used.

Running

With llama.cpp

Serve an OpenAI-compatible API on port 8000. The -hf flag pulls the GGUF from Hugging Face; append :Q6_K (case-insensitive) to pick the quant (llama-server docs):

# Q6_K (recommended), offload all layers to the 4080, full 16K context
llama-server -hf unsloth/phi-4-GGUF:Q6_K \
    --port 8000 \
    -ngl 99 \
    -c 16384 \
    --jinja

-ngl 99 (--n-gpu-layers) offloads every layer to the GPU — the dense 14B quant file (12.03GB at Q6_K) sits entirely in VRAM.
-c 16384 sets the full 16K context — that's Phi-4's ceiling (max_position_embeddings 16384), so there's no benefit to requesting more. At Q6_K, the ~3.3GB full-context f16 KV cache fits in the ~4GB left after weights.
--jinja applies the GGUF's built-in <|im_start|>…<|im_sep|>…<|im_end|> chat template so the assistant format parses correctly — required for clean turn boundaries.

If you want Q8_0 on this 16GB card, the weights (15.58GB) leave almost nothing for the KV cache, so you must quantize it and probably trim context:

# Q8_0 on 16GB is tight — quantize the KV cache; reduce -c if you still OOM
llama-server -hf unsloth/phi-4-GGUF:Q8_0 \
    --port 8000 \
    -ngl 99 \
    -c 8192 \
    -fa on -ctk q8_0 -ctv q8_0 \
    --jinja

For most people on a 16GB 4080, Q6_K at full context is the better default — the quality gap to Q8_0 at 14B is small, and Q6_K leaves comfortable KV headroom.

With Ollama

Pull and run the community GGUF directly from Hugging Face; append a :quant tag to choose the quant (HF × Ollama docs):

ollama run hf.co/unsloth/phi-4-GGUF:Q6_K

Ollama serves an OpenAI-compatible API at http://localhost:11434/v1 for chat clients. Use a recent Ollama build so the corrected Phi-4 template is applied.

Use it as a chat assistant

Point any OpenAI-compatible chat client at your local endpoint by setting its base URL and a dummy API key — no cloud, no per-token cost.

Open WebUI (optional local chat front-end). A self-hosted, ChatGPT-style UI that talks to any OpenAI-compatible server. Run it and point it at your local endpoint:

# Point Open WebUI at your local llama-server (or Ollama on :11434)
docker run -d -p 3000:8080 \
    -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
    -e OPENAI_API_KEY=EMPTY \
    ghcr.io/open-webui/open-webui:main

Then open http://localhost:3000 and chat. (Open WebUI also autodetects a local Ollama install, so with the Ollama path you can skip the base-URL wiring entirely.)

Directly via the API. Any OpenAI SDK or curl works against the same endpoint — use it for scripts, writing tools, or your own app:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "phi-4",
      "messages": [{"role": "user", "content": "Summarize this in three bullet points: ..."}]
    }'

Local servers don't check the key, so any non-empty string (e.g. EMPTY) works where a client requires one.

Results

VRAM usage: The dense 14B loads entirely as its GGUF file — Q6_K is 12.03GB on disk (byte-verified from the unsloth GGUF tree). On the RTX 4080's 16GB that leaves ~4GB for the KV cache, and at full 16K context the f16 KV cache is ~3.3GB — so Q6_K + full context fits with a little room. Q8_0 (15.58GB) is tight: it overflows 16GB at full f16 KV and needs KV-cache quantization (~1.7GB) plus possibly reduced context. Q5_K_M (10.41GB) and Q4_K_M (8.89GB) leave more headroom. The full-precision F16/BF16 GGUF (29.32GB) does not fit a 16GB card.
Model capability (vendor evals — Microsoft's own, NOT hardware throughput): Microsoft reports MMLU 84.8, GPQA 56.1, MATH 80.4, HumanEval 82.6, MGSM 80.6, DROP 75.5 — strong STEM/math/reasoning for a 14B. (Its SimpleQA score is low by design — that's a factuality / hallucination-resistance probe, not a capability score.) These are the vendor's benchmarks, not measurements on this GPU.
Speed: No community throughput benchmark for Phi-4 on the RTX 4080 exists yet — we would rather omit a tok/s figure than invent one or borrow it from different hardware. Live measurements will appear at /check/phi-4/rtx-4080 once contributed.

For the full benchmark data, see /check/phi-4/rtx-4080.

Troubleshooting

Runaway / garbled generation that won't stop — old GGUF template bug

This is the headline Phi-4 gotcha. Early-2025 (January) Phi-4 GGUFs shipped with the wrong end-of-turn token — EOS was <|endoftext|> instead of <|im_end|> — so the model never stops cleanly and produces runaway or garbled output. Fix it by using a current GGUF (the unsloth GGUF ships the corrected template — unsloth's write-up) and a recent llama.cpp / Ollama build. If you downloaded weights in early 2025, re-download them. Also make sure you pass --jinja so the embedded template is applied.

The chat template looks wrong / responses are malformed

Pass --jinja to llama-server so the GGUF's built-in chat template is applied — without it the assistant format won't parse. Phi-4 uses the <|im_start|>role<|im_sep|>…<|im_end|> template (the phi3 architecture's format, not Mistral's Tekken). The GGUF / llama.cpp path uses the embedded tokenizer and template, so there's no extra Python install required.

Out of memory — usually Q8_0 at full context

On a 16GB 4080, Q6_K at full 16K context fits with room to spare. If you hit OOM it's almost always because you loaded Q8_0 (15.58GB): its weights leave under ~0.5GB, so a full f16 KV cache (~3.3GB) overflows. Fixes in order: (1) drop to Q6_K (12.03GB) — the recommended choice here; (2) if you insist on Q8_0, quantize the KV cache with -fa on -ctk q8_0 -ctv q8_0 (~1.7GB) and reduce context (-c 8192). Don't try the F16/BF16 GGUF (29.32GB) — it doesn't fit 16GB.

`torch` / CUDA errors — this is llama.cpp, not a Python ML stack

Serving Phi-4 via llama.cpp or Ollama does not require PyTorch, flash-attn wheels, or a Python ML stack. If you hit a CUDA error, confirm you built (or downloaded) the CUDA-enabled llama.cpp (Option A, -DGGML_CUDA=ON) rather than a CPU-only binary. On a 16GB 4080 the GGUF + llama.cpp path is the right one; at 14B, Q6_K is already near-lossless-feeling.

Model or GPU 404 on /check

Phi-4 is a new addition; if the /check/phi-4/rtx-4080 link 404s, the catalogue row is still being registered. The recipe's install and run steps are independent of the benchmark endpoint.