Qwen3.8-27B on RTX 3090: 128K-context vision chat via llama.cpp GGUF

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — Qwen's 27B dense vision-language model — on a single RTX 3090, answering both text and image prompts at a 131,072-token context window. The install is llama.cpp with a 4-bit GGUF plus its separate vision projector.

Hardware data: RTX 3090 (24GB VRAM) · ~20.5 GiB derived working set at 128K context · See benchmark data

⚠️ Known issue: a 4-bit K or V cache (q4_0/q4_1) silently moves attention to the CPU on default CUDA builds, dropping prefill from ~1000 t/s to ~35 t/s on this card. Use q8_0 for both, as the command below does. Details in Troubleshooting.

Requirements

Why a 27B model leaves room for 128K of context

Qwen3.8-27B is a hybrid: the model card gives its stack as "Hidden Layout: 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))". Only 16 of the 64 blocks are full-attention layers; the other 48 are Gated DeltaNet linear-attention layers that carry a fixed-size recurrent state instead of a growing KV cache. llama.cpp reproduces exactly that split — src/models/qwen35.cpp marks a layer recurrent when (i + 1) % full_attention_interval != 0, which for full_attention_interval = 4 leaves attention at blocks 3, 7, … 63.

That is the whole reason this fits. With head_count_kv = 4 and key_length = value_length = 256 (read from the GGUF header), one token costs 2 × 4 × 256 × 2 bytes = 4 KiB per attention layer, so 64 KiB per token at f16 across 16 layers — a quarter of what a conventional 64-layer 27B would charge. A community reader reached the same conclusion on the model's discussions tab, noting "KV usage is much lower than a conventional 64-layer full-attention 27B model".

The 48 recurrent layers cost a constant instead. llama-hparams.cpp sizes them at (d_conv − 1) × (d_inner + 2 × n_group × d_state) for the convolution state and d_state × d_inner for the recurrent state — 30,720 + 786,432 elements per layer, held in F32 regardless of --cache-type-k/v, and multiplied by the number of parallel sequences. At --parallel 1 that is 156,893,184 B total, about 0.15 GiB.

Derived budget at -c 131072 with q8_0 K and V:

That leaves 3.521 GiB of the card's 24 GiB for compute buffers and CUDA context. This is a derived envelope from cited file sizes and the runtime's own allocation formulas, not a measured peak — see /check/qwen3-8-27b/rtx-3090 for measured data as it lands.

Two limits worth knowing before you pick a context size. The same weights with an unquantized f16 KV at 131072 need 24.229 GiB and do not fit; drop to -c 65536 and f16 KV fits at 20.229 GiB. And the model's full native window — the card advertises "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." — needs 24.729 GiB even at q8_0, so 262K does not fit on a 24 GB card by this path.

Installation

1. Get a CUDA llama.cpp build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -S . -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

On build numbers. The two things this recipe structurally needs are the qwen35 architecture in src/llama-arch.cpp and the qwen3vl_merger projector in tools/mtmd/clip-impl.h. Both are present at least as far back as b8001 — thousands of builds before this model was released — so the support floor is old, and any remotely current build clears it. Do not read the b10178 mentioned elsewhere on this page as a minimum: it is simply the build in the one public RTX 3090 report of this model, an observation about what someone happened to run. What actually matters is the opposite end — the model is days old, several bugs against it are still open, so prefer a recent build and pin the one you tested. If you intend to use a 4-bit KV cache, add -DGGML_CUDA_FA_ALL_QUANTS=ON here — and note that this is a build flag, not a build number, so no version bump substitutes for it. See Troubleshooting.

2. Download the weights and the vision projector

Vision is a separate file. The main GGUF alone gives you a text-only model.

pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-IQ4_NL.gguf mmproj-BF16.gguf \
  --local-dir ./qwen3.8-27b

mmproj-BF16.gguf declares clip.projector_type = qwen3vl_merger, which llama.cpp maps to its PROJECTOR_TYPE_QWEN3VL handler — the projector is structurally loadable by this runtime, not merely published alongside it.

Running

./build/bin/llama-server \
  -m ./qwen3.8-27b/Qwen3.8-27B-IQ4_NL.gguf \
  --mmproj ./qwen3.8-27b/mmproj-BF16.gguf \
  -ngl 99 -c 131072 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on -b 2048 -ub 512 \
  --jinja --reasoning-format deepseek \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --host 127.0.0.1 --port 8080

The server listens on http://127.0.0.1:8080 with an OpenAI-compatible API and a built-in web UI that accepts image uploads. On load it logs loaded multimodal model once the projector is accepted.

The sampler values above are the model card's own recommended settings for thinking mode.

--parallel 1 is load-bearing — do not trim it. The 48 linear-attention layers hold their recurrent state per sequence, and llama.cpp sizes that allocation as max(1, n_seq_max), so the cost scales with the slot count. It is easy to assume the flag is redundant, because common/common.h declares n_parallel = 1 — but that is not what llama-server runs. common/arg.cpp overrides it to -1 ("auto") for the server example before your arguments are parsed, and tools/server/server.cpp then resolves a negative value to n_parallel = 4 with kv_unified = true. Leaving the flag off therefore allocates the recurrent state four times over — 627,572,736 B instead of 156,893,184 B, an extra 0.438 GiB off your headroom for slots you did not ask for. (llama-cli and llama-mtmd-cli never take that branch, so they genuinely do default to one sequence.) The KV cache itself is unchanged either way, because the auto path also turns on a unified cache, which keeps the full context available to a single sequence rather than dividing it.

Thinking is on by default, at the highest effort setting. The card documents reasoning_effort levels of xhigh, medium and low with xhigh as the default, which is why short prompts can produce long reasoning traces. Pass --reasoning-effort low (or medium) to trade depth for latency, and --reasoning-budget 0 to disable thinking entirely.

Images

curl -s http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role":"user","content":[
    {"type":"text","text":"What is in this image?"},
    {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,'"$(base64 -w0 photo.jpg)"'"}}
  ]}]
}'

If grounding or OCR accuracy looks poor, llama.cpp itself suggests raising the image token floor with --image-min-tokens 1024; it prints that hint on load for Qwen-VL models (example log).

Video, if you need it

The model card describes Qwen3.8-27B as "a native vision-language model that understands images and videos", and llama.cpp does have a video path — but it is a build-time option, not a given. mtmd-helper.h documents the feature as "video input helpers (requires ffmpeg/ffprobe installed on the system)", gated on the MTMD_VIDEO compile flag (default ON, force-disabled when LLAMA_SUBPROCESS is off). With such a build, llama-mtmd-cli exposes a /video <path> command and llama-server accepts an input_video content part. Confirm your binary reports video support before planning around it — a build without ffmpeg on PATH will refuse the input.

Results

• Prompt processing, short prompts: 991–1276 t/s on 4–10K prompts, community-reported, single source, on this exact card — the reporter of llama.cpp issue #27109 running Qwen3.8-27B-IQ4_NL.gguf with the BF16 projector at -c 130000, q8_0 K and V, full offload, build 10178. Read the prompt length as part of the number: that is the column header in the source table, and it is not the regime this recipe is built for.
• Prompt processing at long context: expect substantially less. The same reporter published a decay curve on the same card and build — 1094 t/s at ~20K falling to 659 t/s at ~120K, about a 40% drop. Treat that as the shape only, not as values you can expect: it was measured under q4_1/q8_0 with a GGML_CUDA_FA_ALL_QUANTS=ON build, which is a different cache configuration from the q8_0/q8_0 this recipe documents. No published figure covers q8_0/q8_0 at 128K on this card. The honest planning assumption is that a 128K prefill costs materially more per token than the 4–10K headline suggests.
• Generation speed: omitted. The report's decode figures were taken with --spec-type draft-mtp --spec-draft-n-max 4 active while this recipe recommends no speculation, and — the deciding reason — neither of the two speculation datasets available is on this model and card, so nothing anchors a correction to this pair. (Both do point the same way, ≤ neutral, which would make the omitted figure a floor rather than an unknown; that is still not a measurement of the configuration documented here.) No first-party benchmark exists for this pair; if you measure your own, please send it via /contribute so it lands on /check/qwen3-8-27b/rtx-3090.
• VRAM usage: ~20.5 GiB derived working set at 128K context (table above), leaving ~3.5 GiB headroom on a 24 GB card. Derived from cited file sizes and llama.cpp's allocation formulas, not measured.
• Quality notes: this is a thinking model with xhigh reasoning effort by default; expect long traces on short questions unless you lower the effort level. The card's non-thinking preset is temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5.

For the full benchmark data, see /check/qwen3-8-27b/rtx-3090.

Troubleshooting

Prefill collapses to ~35 tokens/s after switching to a 4-bit KV cache

This is the single most expensive trap on this card. On a default CUDA build, llama.cpp's flash-attention kernel supports neither q4_1 KV nor a K type that differs from the V type. When the requested combination is unsupported the graph scheduler quietly relocates the attention op to the CPU backend — no error is printed. The reporter of issue #27109 measured prefill falling from 991–1276 t/s to 34–106 t/s on an RTX 3090, and traced it to that fallback: "Prefill then runs on CPU at ~34 t/s." Generation speed is unaffected, which is what makes it hard to spot.

Two fixes. Either keep K and V both at q8_0 (what the command above does), or rebuild with the extra kernels: "This compiles the additional FA quant kernels (q4_1/q5_0/q5_1) and allows K != V type mixes."

cmake -S . -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j

Ollama's default tag turns on speculative decoding

qwen3.8:27b and qwen3.8:27b-mtp-q4_K_M resolve to byte-identical manifests on the Ollama registry — same weights blob, same projector, same params blob carrying "draft_num_predict":4. The only no-speculation tag is qwen3.8:27b-q4_K_M, whose params blob is identical except that the draft_num_predict key is absent. So ollama run qwen3.8:27b is running multi-token prediction whether or not you asked for it.

Whether that helps you is not settled, and neither datapoint below is on an RTX 3090 — so treat this as a reason to measure, not as a verdict. A reader benchmarking on an Apple M4 Pro in discussion #80 concluded "So under Ollama, speculation repays its own overhead in the best case and never more." — 11.78 tok/s with speculation off, 11.40 when drafts are accepted, 5.14 when they are rejected. Separately, llama.cpp issue #26750 reports draft-mtp acceptance falling to 35–41% on CUDA against ~92% on Vulkan — but read its scope before applying it here: its CUDA measurements are on a workstation RTX PRO 4000 (Blackwell), not consumer Ampere; its target is Qwen3.5-9B, a previous-generation model, not this one; and a follow-up in the same thread records 84% CUDA acceptance at build b10261, placing the regression after that and inside b10290. That window is newer than the b10178 build the RTX 3090 report used, so none of it has been shown to apply to this card. The one acceptance figure that is from an RTX 3090 on this model is a parenthetical in issue #27109 — 41–62% draft acceptance at build 10178 — recorded without complaint, under that report's q4_1/q8_0 build rather than this recipe's cache settings. It is closer to the low CUDA figures than to Vulkan's ~92%, which is a reason to measure rather than a verdict either way. If you want speculation here, benchmark qwen3.8:27b against qwen3.8:27b-q4_K_M on your own machine before keeping it.

Multi-token prediction: which file, and whether to bother

The two GGUF publishers package the MTP head differently, and the header settles it. unsloth's Qwen3.8-27B-IQ4_NL.gguf reports block_count = 65 with nextn_predict_layers = 1 — the head is inside the main file, so --spec-type draft-mtp needs no extra download. ggml-org's Qwen3.8-27B-Q4_K_M.gguf reports block_count = 64 and no such key; that repo ships the head separately as mtp-Qwen3.8-27B-Q4_0.gguf. Do not mix a target from one publisher with a drafter from the other.

Given the CUDA acceptance-rate reports above, this recipe leaves speculation off. Note also that enabling it adds a 17th attention layer's worth of KV, since llama.cpp gives the MTP block a plain attention cache rather than the hybrid one.

The server freezes mid-generation on a multi-GPU box

llama.cpp issue #27122 reports reproducible CUDA lockups with this model when --split-mode tensor and --spec-type draft-mtp are combined, on a two-card setup; the reporter notes it does not occur with --split-mode layer. This does not apply to a single RTX 3090 — --split-mode is a multi-GPU setting — and is listed only so you can rule it out if you later add a second card.

An image request crashes the server

There is one open report of silent crashes on image input, issue #27124, but it is on a Vulkan build on an AMD Ryzen AI MAX+ 395 APU, not a CUDA build on an NVIDIA card, so its failure mode should not be assumed to transfer here. If you do hit a crash on CUDA, note that the vision path for this architecture is weeks old: pin your build, and report it upstream with the build number.

Where the model came from, if you are hunting for a different quant

Qwen does not publish a GGUF of this model. The Qwen organisation's Hugging Face account carries exactly four Qwen3.8 repositories — Qwen3.8-27B, Qwen3.8-27B-FP8, Qwen3.8-2.4T-A95B and Qwen3.8-2.4T-A95B-FP8 — and none is a GGUF conversion, even though the same organisation ships first-party GGUFs for many older models. Every GGUF referenced here is a community conversion. Note also that a nominal quant name is not a size: Q4_K_M ranges from 16.81 GB (lmstudio-community) to 18.97 GB (ggml-org) across publishers, so check the byte count of the specific file you are downloading rather than trusting the label.