Qwen3.8-27B on RTX 4090: 128K-context vision chat via llama.cpp Q4_K_M

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — Qwen's 27B vision-language model — on a single RTX 4090, answering both text and image prompts at a 131,072-token context window. The install is llama.cpp with a 4-bit GGUF plus its separate vision projector.

Hardware data: RTX 4090 (24GB VRAM) · ~21.2 GiB derived working set at 128K context · See benchmark data

⚠️ Known issue: a 4-bit K or V cache (q4_1, or a K type that differs from the V type) silently moves attention to the CPU on default CUDA builds, collapsing prefill by roughly 20×. Use q8_0 for both, as the command below does. Details in Troubleshooting.

Requirements

Why a 27B model leaves room for 128K of context

Qwen3.8-27B is a hybrid: the model card gives its stack as "Hidden Layout: 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))". Only 16 of the 64 blocks are full-attention layers; the other 48 are Gated DeltaNet linear-attention layers that carry a fixed-size recurrent state instead of a growing KV cache. llama.cpp reproduces exactly that split — src/models/qwen35.cpp marks a layer recurrent when (i + 1) % full_attention_interval != 0, which for full_attention_interval = 4 leaves attention at blocks 3, 7, … 63.

That is the whole reason this fits. With head_count_kv = 4 and key_length = value_length = 256 (read from the GGUF header), one token costs 2 × 4 × 256 × 2 bytes = 4 KiB per attention layer, so 64 KiB per token at f16 across 16 layers — a quarter of what a conventional 64-layer 27B would charge. A community reader reached the same conclusion on the model's discussions tab, noting "KV usage is much lower than a conventional 64-layer full-attention 27B model".

The 48 recurrent layers cost a constant instead. llama-hparams.cpp sizes them at (d_conv − 1) × (d_inner + 2 × n_group × d_state) for the convolution state and d_state × d_inner for the recurrent state — 30,720 + 786,432 elements per layer, held in F32 regardless of --cache-type-k/v, and multiplied by the number of parallel sequences. At --parallel 1 that is 156,893,184 B total, about 0.15 GiB.

Derived budget at -c 131072 with q8_0 K and V:

That leaves 2.805 GiB of the card's 24 GiB for graph and compute buffers, the CUDA context and the vision graph, none of which is derived here. This is a derived envelope from cited file sizes and the runtime's own allocation formulas, not a measured peak — see /check/qwen3-8-27b/rtx-4090 for measured data as it lands. Two practical deductions come out of that 2.805 GiB rather than out of the components: cudaDeviceProp::totalGlobalMem sits slightly under the nominal 24,576 MiB, and the display driver reserves more on top if a monitor is attached. Check yours with nvidia-smi --query-gpu=memory.total,memory.used --format=csv before committing to a context size.

Two limits worth knowing before you pick one. The same weights with an unquantized f16 KV at 131072 need 24.945 GiB and do not fit; drop to -c 65536 and f16 KV fits at 20.945 GiB. And the model's full native window — the card advertises "Context Length: 262,144 natively and extensible up to 1,000,000 tokens." — needs 25.445 GiB even at q8_0, so 262K does not fit on a 24 GB card by this path.

Which GGUF, and why not the one the RTX 3090 recipe leads with

The sibling recipe on this site, for the RTX 3090, leads with unsloth's IQ4_NL build, and that choice does not transfer here even though the two cards hold the same 24 GiB. Its reason was evidential rather than architectural: the only public prefill figures for this model on any 24 GB card were measured on that exact file, in llama.cpp issue #27109, on an RTX 3090. Leading with the file someone had measured was right there. No such measurement exists on an RTX 4090 — see Results for the space searched — so the anchor for IQ4_NL is absent here, and a different and better-matched one is available in its place.

A reader publishing serving configs in discussion #34 runs this model on an RTX 4090 under llama-server with Qwen3.8-27B-Q4_K_M.gguf (aliased in the config to unsloth/Qwen3.8-27B-GGUF:Q4_K_M) at --ctx-size 131072, --cache-type-k q8_0 --cache-type-v q8_0, --parallel 1, --flash-attn on and -b 2048 -ub 512. That is the same file, window, cache configuration and batch shape this page recommends, chosen independently by someone running it on this card. It is a published configuration and not a benchmark — it carries no timings and no peak-VRAM figure, and it runs without --mmproj, so it says nothing about the vision path or about the 0.867 GiB the projector adds to the budget above. Treat it as evidence about the settings, not as a measurement.

The arithmetic agrees with it. Both builds fit the same 131,072-token window on this card: Q4_K_M at 21.195 GiB and IQ4_NL at 20.479 GiB (16,337,628,128 B of weights in place of the row above, same projector, same KV, same recurrent state). The difference is 0.716 GiB of headroom against roughly half a bit per weight, and with 2.8 GiB spare the less lossy build is the better default. Nothing else separates them here, and the runtime is unusually clear about it. ggml_cuda_should_use_mmq in ggml/src/ggml-cuda/mmq.cu has exactly one type-dependent gate — a switch of MMQ-supported types that lists both GGML_TYPE_Q4_K and GGML_TYPE_IQ4_NL, each with its own mul_mat_q_case instantiation in the same file — and then, past a shared-memory guard that is not type-dependent either, returns true outright at if (turing_mma_available(cc)). That predicate (common.cuh) holds for any NVIDIA card at Turing or above, and this one is compute capability 8.9, so the decision never reaches the batch-size heuristic below it. Both builds take the same kernel path on this card.

If 2.805 GiB proves too tight once your driver's reservation is accounted for, IQ4_NL is the documented step down and buys back 0.716 GiB with no other change to the command. Do not climb instead: Q5_K_M (19,834,055,648 B) totals 23.735 GiB in the same configuration, which is arithmetically inside 24 GiB with 0.265 GiB left over — not a margin to plan on.

What crossing a GPU generation does and does not affect. The sibling recipe targets an Ampere card and this one is Ada, so the inherited reasoning is worth checking rather than assuming. Nothing in the budget above moves: both are 24 GiB parts, and the memory subsystem is the same shape and the same speed — NVIDIA's Ada architecture whitepaper Table 1 lists the RTX 4090 at 384-bit, 21 Gbps GDDR6X and 1008 GB/sec, where the GA102 whitepaper gives the RTX 3090 the same 384-bit interface at 19.5 Gbps and 936 GB/sec. What the generation actually buys is compute and cache: the same table puts this card at 82.6 FP32 TFLOPS and 73,728 KB of L2 against 40 TFLOPS and 6,144 KB for the RTX 3090 Ti. Neither figure enters the VRAM budget, and neither licenses a speed claim on a pair nobody has measured. In the runtime, no arch-dependent branch resolves differently: llama.cpp's CUDA kernel selection keys on "fp16 MMA hardware available", true for both generations, and the flash-attention trap described in Troubleshooting is gated on a compile flag, not on a compute capability, so it fires here exactly as it does on Ampere.

One thing the generation does add is an FP8 tensor path, and it is worth saying plainly why this recipe still leads with a GGUF: Qwen's first-party Qwen3.8-27B-FP8 repository totals 30,866,866,928 B of safetensors — 28.747 GiB of weights alone, over the card before a single byte of KV cache. The FP8 hardware is real; the artifact does not fit. The author of discussion #34 makes the same split in practice, running an NVFP4 build under vLLM on an NVIDIA DGX Spark and the GGUF under llama.cpp on the 4090.

Installation

1. Get a CUDA llama.cpp build

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -S . -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

On build numbers. The two things this recipe structurally needs are the qwen35 architecture in src/llama-arch.cpp and the qwen3vl_merger projector in tools/mtmd/clip-impl.h. Both are present at least as far back as b8001 — thousands of builds before this model was released — so the support floor is old and any remotely current build clears it. What actually matters is the opposite end: the model is days old and several bugs against it are still open, so prefer a recent build and pin the one you tested. If you intend to use a 4-bit KV cache, add -DGGML_CUDA_FA_ALL_QUANTS=ON here — note that this is a build flag, not a build number, so no version bump substitutes for it, and the prebuilt ghcr.io/ggml-org/llama.cpp images do not carry it. See Troubleshooting.

2. Download the weights and the vision projector

Vision is a separate file. The main GGUF alone gives you a text-only model.

pip install -U "huggingface_hub[cli]"
hf download unsloth/Qwen3.8-27B-GGUF \
  Qwen3.8-27B-Q4_K_M.gguf mmproj-BF16.gguf \
  --local-dir ./qwen3.8-27b

mmproj-BF16.gguf declares clip.projector_type = qwen3vl_merger, which llama.cpp maps to its PROJECTOR_TYPE_QWEN3VL handler — the projector is structurally loadable by this runtime, not merely published alongside it.

Running

./build/bin/llama-server \
  -m ./qwen3.8-27b/Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj ./qwen3.8-27b/mmproj-BF16.gguf \
  -ngl 99 -c 131072 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on -b 2048 -ub 512 \
  --jinja --reasoning-format deepseek \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --host 127.0.0.1 --port 8080

The server listens on http://127.0.0.1:8080 with an OpenAI-compatible API and a built-in web UI that accepts image uploads. On load it logs loaded multimodal model once the projector is accepted.

The sampler values above are the model card's own recommended settings for thinking mode, and are the same values the RTX 4090 config in discussion #34 passes.

--parallel 1 is load-bearing — do not trim it. The 48 linear-attention layers hold their recurrent state per sequence, and llama.cpp sizes that allocation as max(1, n_seq_max), so the cost scales with the slot count. It is easy to assume the flag is redundant, because common/common.h declares n_parallel = 1 — but that is not what llama-server runs. common/arg.cpp overrides it to -1 ("auto by default") for the server example before your arguments are parsed, and tools/server/server.cpp then resolves a negative value to n_parallel = 4 with kv_unified = true. Leaving the flag off therefore allocates the recurrent state four times over — 627,572,736 B instead of 156,893,184 B, an extra 0.438 GiB off headroom you do not have much of — for slots you did not ask for. (llama-cli and llama-mtmd-cli never take that branch, so they genuinely do default to one sequence.) The KV cache itself is unchanged either way, because the auto path also turns on a unified cache, which keeps the full context available to a single sequence rather than dividing it.

Thinking is on by default, at the highest effort setting. The card documents reasoning_effort levels of xhigh, medium and low with xhigh as the default, which is why short prompts can produce long reasoning traces. Pass --reasoning-effort low (or medium) to trade depth for latency, and --reasoning-budget 0 to disable thinking entirely. --reasoning-effort needs a build of b10434 or newer — the flag does not exist before that, and on an older build the effort level is ignored even when sent over HTTP. Troubleshooting has the exact boundary and the workaround for older builds.

Images

curl -s http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role":"user","content":[
    {"type":"text","text":"What is in this image?"},
    {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,'"$(base64 -w0 photo.jpg)"'"}}
  ]}]
}'

If grounding or OCR accuracy looks poor, llama.cpp itself suggests raising the image token floor with --image-min-tokens 1024; it prints that hint on load for Qwen-VL models (example log).

Video, if you need it

The model card describes Qwen3.8-27B as "a native vision-language model that understands images and videos", and llama.cpp does have a video path — but it is a build-time option, not a given. mtmd-helper.h documents the feature as "video input helpers (requires ffmpeg/ffprobe installed on the system)", gated on the MTMD_VIDEO compile flag (default ON, force-disabled when LLAMA_SUBPROCESS is off). With such a build, llama-mtmd-cli exposes a /video <path> command and llama-server accepts an input_video content part. Confirm your binary reports video support before planning around it — a build without ffmpeg on PATH will refuse the input.

Results

• Speed: omitted, because nothing has been measured on this card. The absence is a searched one, not an assumed one. All 98 discussions on the model's HF repo were enumerated and fetched individually: exactly one names this card, and it is the serving config quoted above, which carries no timings. GitHub's issue search returns 24 issues mentioning Qwen3.8 in the llama.cpp repository at the time of writing and none of them mentions this card; the same index returns 495 hits for the card's name across that repository overall. It also indexes comment bodies and not just opening posts — issue #27023 carries Qwen3.8 in neither its title nor its body, only in a reply, and the search returns it anyway — so an empty result is a real negative rather than a thread the index could not see. The backend has no benchmark for this pair either. If you measure your own, please send it via /contribute so it lands on /check/qwen3-8-27b/rtx-4090.
• Do not carry the RTX 3090's published figures across. Prefill numbers for this model do exist on that card — quoted in Troubleshooting below, because they are the evidence for the KV-cache trap — but they were taken on a different build of the weights, under a different cache configuration, at a prompt length this recipe is not built around, and on the previous GPU generation. The two cards have identical memory bandwidth and very different compute, which is exactly the combination that makes a cross-card projection unsafe in both directions.
• VRAM usage: ~21.2 GiB derived working set at 128K context (table above), leaving ~2.8 GiB on a 24 GB card. Derived from cited file sizes and llama.cpp's allocation formulas, not measured.
• Quality notes: this is a thinking model with xhigh reasoning effort by default; expect long traces on short questions. The card's non-thinking preset is temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5.

For the full benchmark data, see /check/qwen3-8-27b/rtx-4090.

Troubleshooting

Prefill collapses to a few dozen tokens/s after switching to a 4-bit KV cache

This is the most expensive trap on any CUDA build of this model. On a default build, llama.cpp's flash-attention kernel supports neither a q4_1 KV cache nor a K type that differs from the V type, and when the requested combination is unsupported the graph scheduler quietly relocates the attention op to the CPU backend — no error is printed. The gate is visible in ggml/src/ggml-cuda/fattn.cu: ggml_cuda_fattn_kv_type_supported returns false for Q4_1/Q5_0/Q5_1 under #ifndef GGML_CUDA_FA_ALL_QUANTS, and a few lines above, the same guard returns BEST_FATTN_KERNEL_NONE whenever K->type != V->type. Both conditions are compile-flag gated and neither is architecture-specific, so crossing from Ampere to Ada changes nothing about them — and the prebuilt ghcr.io/ggml-org/llama.cpp image used by the RTX 4090 config in discussion #34 is a default build, which is presumably why that config also runs matched q8_0 K and V.

It was reported on an RTX 3090 in issue #27109, where prefill on 4–10K prompts fell from 991–1276 t/s at q8_0/q8_0 to 34–106 t/s with K=q4_1, V=q8_0, and the reporter traced it to the fallback: "Prefill then runs on CPU at ~34 t/s." Generation speed was unaffected, which is what makes it hard to spot. Read those numbers as the size of the cliff, not as throughput to expect here — they are another card's, from the previous generation, on the IQ4_NL build.

Two fixes. Either keep K and V both at q8_0 (what the command above does), or rebuild with the extra kernels: "This compiles the additional FA quant kernels (q4_1/q5_0/q5_1) and allows K != V type mixes."

cmake -S . -B build -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j

Note that the issue's title condemns q4_0 and its own thread walks that back: the same reporter records that K=q8_0, V=q8_0 and K=q4_0, V=q4_0 are "supported, stays on GPU", and both of the thread's q4_0 datapoints are mixed K/V configurations. The rule that survives is match the two types, not "avoid 4-bit". A community pull request, #27140, proposes making all the small KV quants fast without the compile flag; as of writing it is open and unmerged, its author discloses the kernel code was AI-written, and it was tested only on the reporter's own hardware. Nothing in this recipe depends on it.

Ollama's default tag turns on speculative decoding

qwen3.8:27b and qwen3.8:27b-mtp-q4_K_M resolve to byte-identical manifests on the Ollama registry — same 16,810,714,464 B weights blob, same 931,146,016 B projector, and the same 114-byte params blob, which reads {"draft_num_predict":4,…}. The only no-speculation tag is qwen3.8:27b-q4_K_M, whose 92-byte params blob is identical except that the draft_num_predict key is absent. So ollama run qwen3.8:27b is running multi-token prediction whether or not you asked for it. Both tags fit this card, but Ollama will not give you 131,072 tokens of context by default — raise num_ctx explicitly.

Whether speculation helps is not settled, and no datapoint below is on this card. A reader benchmarking on an Apple M4 Pro in discussion #80 concluded "So under Ollama, speculation repays its own overhead in the best case and never more." — 11.78 tok/s with speculation off, 11.40 when drafts are accepted, 5.14 when they are rejected. Separately, llama.cpp issue #26750 reports draft-mtp acceptance falling to 35–41% on CUDA against ~92% on Vulkan — but its CUDA measurements are on a workstation Blackwell card and its target is a previous-generation 9B model, and a follow-up in the same thread records 84% CUDA acceptance at build b10261, placing the regression in a narrow and much newer window. Benchmark qwen3.8:27b against qwen3.8:27b-q4_K_M on your own machine before keeping it.

Multi-token prediction: which file, and whether to bother

The two GGUF publishers package the MTP head differently, and the header settles it. unsloth's build reports block_count = 65 with nextn_predict_layers = 1 — the head is inside the main file, so --spec-type draft-mtp needs no extra download. ggml-org's Qwen3.8-27B-Q4_K_M.gguf reports block_count = 64 and no such key; that repo ships the head separately as mtp-Qwen3.8-27B-Q4_0.gguf. Do not mix a target from one publisher with a drafter from the other. Note also that enabling speculation adds a 17th attention layer's worth of KV, since llama.cpp gives the MTP block a plain attention cache rather than the hybrid one — on a budget with 2.8 GiB spare that is worth measuring before you commit to it.

Reasoning effort does nothing on a build older than b10434

Issue #27023 is worth reading before you try to shorten those traces, because this model is in it. A reader running Qwen3.8-27B-Q6_K.gguf with --mmproj mmproj-F16.gguf reports "Same problem here with Qwen3.8-27B." and pins the behaviour down precisely: a top-level reasoning_effort field sent over HTTP has no effect, while the same value passed inside chat_template_kwargs does work, with /apply-template confirming that the injected system instruction changes from xhigh to low. A project contributor replies "@kidultff this is fixed on master."

"Fixed on master" is a statement made on a date, not a version, so here is the version — read out of the source at each tag rather than taken on trust. The fix is commit 7e4c0a9, "chat : pass reasoning_effort to template", merging PR #26941. The symbol it introduces, caps_apply_reasoning_effort in common/jinja/caps.cpp, is absent at tag b10433 and present at b10434 — both files fetched whole at HTTP 200, so that boundary is measured rather than inferred. The same commit is what adds the --reasoning-effort flag to common/arg.cpp in the first place: at b10433 that flag does not exist, at b10434 it does. And before it, the server honoured only reasoning_effort: none — the line the commit deletes from the server's own README says so outright: "Other values (e.g., low, max) have no effect on reasoning."

So on b10434 or newer, --reasoning-effort low behaves as the Running section describes. On anything older the flag is not recognised at all, and the workaround is the one from the thread — either --chat-template-kwargs '{"reasoning_effort": "low"}' at startup, or the same key per request:

"chat_template_kwargs": {"reasoning_effort": "low"}

--reasoning-budget 0 is unaffected either way; that flag predates the fix. Finally, read the issue's title — which sounds like a blanket "reasoning effort is broken" — against its own thread: the original reporter's two models cannot exercise the feature at all, as the first responder points out (Gemma 4 has no effort levels, Muse Glimmer uses reasoning_strength) and the reporter then concedes. The evidence that bears on this page is the later Qwen3.8-27B report, not the opening one.

The server freezes, or asserts on startup, on a multi-GPU box

Two open reports pair this model with --split-mode tensor: #27122 records reproducible CUDA lockups when tensor split is combined with --spec-type draft-mtp (the reporter notes --split-mode layer avoids it), and #27116 records a startup assertion with tensor split and an iq4_nl KV cache. Neither applies to a single RTX 4090 — --split-mode is a multi-GPU setting — and they are listed only so you can rule them out if you later add a second card. A third report, #26901, describes discrete GPUs being misclassified as integrated and breaking tensor split; it names RTX 5080 and RTX 5070 Ti specifically, and this card is neither Blackwell nor part of a multi-GPU split, so it does not reach this recipe on either count.

An image request crashes the server

There is one open report of silent crashes on image input, issue #27124, but it is on a Vulkan build on an AMD Ryzen AI MAX+ 395 APU, not a CUDA build on an NVIDIA card, so its failure mode should not be assumed to transfer here. If you do hit a crash on CUDA, note that the vision path for this architecture is weeks old: pin your build, and report it upstream with the build number.

Where the model came from, if you are hunting for a different quant

Qwen does not publish a GGUF of this model. The Qwen organisation's Hugging Face account carries exactly four Qwen3.8 repositories — Qwen3.8-27B, Qwen3.8-27B-FP8, Qwen3.8-2.4T-A95B and Qwen3.8-2.4T-A95B-FP8 — and none is a GGUF conversion, even though the same organisation ships first-party GGUFs for many older models. Every GGUF referenced here is a community conversion. Note also that a nominal quant name is not a size: Q4_K_M ranges from 16.81 GB (lmstudio-community) to 18.97 GB (ggml-org) across publishers, so check the byte count of the specific file you are downloading rather than trusting the label. Quantised non-GGUF builds have started to appear too — unsloth published an NVFP4 conversion on 2026-08-13 — but it is a safetensors build for vLLM-class runtimes rather than a GGUF, and as noted above the only published serving config for it names a DGX Spark rather than this card. Nothing among the sources checked here establishes a path for that build here.