What You'll Build
A local large-language-model assistant: you chat with Qwen2.5 7B entirely on your own machine — ask questions, draft and refactor code, do math, summarize long text — and it answers as a chatbot, no cloud and no API key. The whole thing runs on an 8 GB RTX 3060 Ti through Ollama at the Q4 quant.
Hardware data: RTX 3060 Ti (8GB VRAM) · ~58 tokens/s generation (Q4, Ollama 0.5.4) · sits right at the 8 GB ceiling · See benchmark data
Requirements
| Component | Minimum | Tested |
|---|---|---|
| GPU | 8GB VRAM | RTX 3060 Ti (8GB) |
| RAM | 16GB | — |
| Storage | ~5 GB (Q4 weights) | 4.7 GB model pull |
| Software | Ollama, NVIDIA driver + CUDA | Ollama 0.5.4 |
Qwen2.5 7B is a text-only causal language model from Qwen (Alibaba Cloud), released under the Apache-2.0 license. Ollama describes it as the latest Qwen series: "Qwen2.5 is the latest series of Qwen large language models" (ollama.com/library/qwen2.5).
⚠️ Right at the 8 GB wall. The cited benchmark peaks at 8.0 GB on this 8 GB card — there is no headroom. Close other GPU consumers before you run: browsers with hardware acceleration, other models, even a second monitor's compositor can tip you into an out-of-memory error or force a slow CPU fallback. This recipe documents the Q4 quant specifically because heavier quants do not fit 8 GB.
Installation
1. Install Ollama
Download and install Ollama for your OS from ollama.com/download. On Linux:
curl -fsSL https://ollama.com/install.sh | sh
Confirm it sees your GPU:
ollama --version
nvidia-smi
2. Pull the Qwen2.5 7B weights
The qwen2.5:7b tag is a 4.7 GB Q4 download (ollama.com/library/qwen2.5):
ollama pull qwen2.5:7b
Running
Start an interactive chat session:
ollama run qwen2.5:7b
>>> Write a Python function that returns the nth Fibonacci number.
Or pass a single prompt non-interactively:
ollama run qwen2.5:7b "Summarize the plot of Hamlet in three sentences."
The model streams its answer token by token. In an interactive session you can keep asking follow-up questions in the same context window.
Results
- Speed: ~58 tokens/s generation at Q4 on the RTX 3060 Ti, measured by DatabaseMart under their "Eval Rate(tokens/s)" column (Ollama 0.5.4). This is the rate at which the model writes its answer.
- VRAM usage: The backend records an 8.0 GB peak on this 8 GB card — i.e. effectively full. Plan for no spare VRAM. See /check
- Quality notes: This is a single commercial benchmark source. Numbers will vary with your Ollama version, driver, and context length. If you measure your own throughput or peak VRAM on a 3060 Ti, please contribute it via /contribute so the next reader gets a corroborating datapoint.
For the full benchmark data, see /check/qwen2-5-7b/rtx-3060-ti.
Troubleshooting
Out of memory / model falls back to CPU
At 8.0 GB peak on an 8 GB card there is no margin. If you see an OOM error or generation suddenly crawls, something else is holding VRAM. Run nvidia-smi to see what is resident, close it, and retry. Keep the context length modest — a very long prompt grows the KV cache and can push you over the 8 GB ceiling. If you find a stable context length on your own 3060 Ti, share it via /contribute.
Don't reach for a larger quant or a bigger model on this card
The Q4 qwen2.5:7b tag is chosen because it fits 8 GB; heavier quants of the same 7B model and the larger qwen2.5:14b / qwen2.5:32b tags (ollama.com/library/qwen2.5) do not leave room to run on an 8 GB card. Stay on the 7B Q4 tag for this GPU.
Which Qwen2.5 variant am I running?
The site catalogues this model against the canonical Qwen2.5-7B-Instruct repo (Apache-2.0, by Qwen / Alibaba Cloud). The Ollama qwen2.5:7b tag tracks the instruction-tuned 7B line; pin a specific tag if you need exact reproducibility.
No other widely-reported issues for this pair. Report problems via the submission form.