Qwen3.8-27B on RTX 5090: the full 262K context window on one card

What You'll Build

A single-card llama-server deployment of Qwen3.8-27B on an RTX 5090 that runs the model's entire native 262,144-token context window with the vision projector loaded — not a truncated 32K or 64K window. The model card states that Qwen3.8-27B "natively supports context lengths of up to 262,144 tokens." On most 24 GB consumer cards that number is aspirational; on a 32 GB card it is reachable, and the reason is architectural rather than a matter of brute capacity.

Hardware data: RTX 5090 (32GB VRAM) · derived working set 25.445 GiB at the full 262,144-token window · See benchmark data

ℹ️ Why the window is affordable here. Qwen3.8-27B is a hybrid: config.json gives 64 layers of which layer_types marks 48 linear_attention (Gated DeltaNet) and only 16 full_attention, at indices 3, 7, 11 … 63. Only those 16 layers hold a KV cache — llama.cpp's hybrid memory filter for this architecture admits exactly the non-recurrent layers (llama-model.cpp). The 48 GDN layers instead hold a fixed-size recurrent state that does not grow with context at all. Had all 64 layers been full-attention, the same 262,144-token window would cost 34 GiB of KV at q8_0 and would not fit on this card.

Requirements

Three separate claims live in that cell, so take them one at a time.

The structural requirement is old and any current build clears it. Running this model at all needs two things in llama.cpp: the qwen35 architecture and the qwen3vl_merger projector. Both predate the model by months — qwen35 appears in src/llama-arch.cpp and qwen3vl_merger in tools/mtmd/clip-impl.h as far back as b8001, which is merely the oldest tag anyone has checked, not the build they landed in. So no build number on this page is a support floor, and you should not read one as a minimum for getting the model to load.

b10434 is a functional floor for one capability: long-context prefill. A user running Qwen3.8-27B at long context on Blackwell reports in llama.cpp issue #27090 that prompts past the 90–100K range crashed llama-server on b10430 and were fixed by b10434, the fix being a recurrent-state rollback in ggml_ssm_scan — precisely the GDN path this model leans on for 48 of its 64 layers. Text generation works on much older builds; the 262K window this recipe is about does not.

And the commit this page was authored against is ad1de39e (b10442-era) — every line of llama.cpp source quoted below was read there. That is a provenance note for the quotes, not a requirement on you.

Installation

1. Get a recent llama.cpp CUDA build

The GGUF architecture string for this family is qwen35, not qwen3.8 — read straight out of the file header (general.architecture = qwen35). You are not hunting for a build that supports it; as noted above, support long predates the model. Take b10434 or newer for the long-context fix; the release tarballs are on the llama.cpp releases page.

# Prebuilt CUDA 13 server image
docker pull ghcr.io/ggml-org/llama.cpp:server-cuda13

That is the same image a community member reports using for a Qwen3.8-27B GGUF server on an RTX 4090 in discussion #34. It runs this recipe as written.

Build from source instead only if you need a compile-time option the prebuilt images do not carry — GGML_CUDA_FA_ALL_QUANTS being the one this page mentions. No image tag and no version bump substitutes for a flag:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120
cmake --build build --config Release -j

CMAKE_CUDA_ARCHITECTURES=120 is the RTX 5090's compute capability, reported as 12.0 in the environment table of discussion #51.

2. Download the weights and the vision projector

There is no first-party GGUF. Qwen ships this model as Transformers safetensors plus an FP8 build; the org's own repo listing returns 54 GGUF repositories and not one of them is a Qwen3.8 (https://huggingface.co/api/models?author=Qwen&search=GGUF), and the model card names only SGLang, vLLM and TokenSpeed as deployment targets. The GGUF conversions come from ggml-org, unsloth, bartowski and lmstudio-community.

Pick a publisher deliberately, because a nominally identical quant is not one file size: Q4_K_M is 16.81 GB from lmstudio-community, 17.11 GB from unsloth and 18.97 GB from ggml-org. This recipe uses unsloth's build.

Units, once, so the two tables below reconcile. File sizes are quoted in decimal GB, exactly as HuggingFace prints them. Everything that has to fit in VRAM is quoted in binary GiB, because that is the unit a 32 GB card's capacity is actually in. The same unsloth Q4_K_M file is therefore 17.11 GB on the Files tab and 15.932 GiB in the budget.

pip install -U huggingface_hub

hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-Q4_K_M.gguf --local-dir ./models
hf download unsloth/Qwen3.8-27B-GGUF mmproj-BF16.gguf          --local-dir ./models

3. Know which file you got

The two publisher families differ in a way that matters for what you can enable later. ggml-org's main GGUF declares qwen35.block_count = 64 and ships the multi-token-prediction head separately as mtp-Qwen3.8-27B-*.gguf. The unsloth and lmstudio-community builds declare qwen35.block_count = 65 and qwen35.nextn_predict_layers = 1 — the MTP head is inside the single file. Both count 64 effective layers for KV purposes, because llama.cpp's n_layer() is defined as n_layer_all - n_layer_nextn (llama-hparams.cpp), so the budget below is the same either way.

Running

./build/bin/llama-server \
  --model  ./models/Qwen3.8-27B-Q4_K_M.gguf \
  --mmproj ./models/mmproj-BF16.gguf \
  --n-gpu-layers 999 \
  --ctx-size 262144 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --image-min-tokens 1024 \
  --batch-size 2048 \
  --ubatch-size 512 \
  --jinja \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0 \
  --host 127.0.0.1 --port 8080

The sampler values are the model card's own recommendation for thinking mode (temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0). The server exposes an OpenAI-compatible /v1/chat/completions on port 8080 and a web UI at http://127.0.0.1:8080.

Four flags in that command are load-bearing rather than habit:

• --cache-type-k q8_0 and --cache-type-v q8_0 must match. llama.cpp's CUDA flash-attention kernel selector returns BEST_FATTN_KERNEL_NONE when K->type != V->type in a default build, and rejects q4_1/q5_0/q5_1 outright unless compiled with GGML_CUDA_FA_ALL_QUANTS (fattn.cu). Falling out of that selector does not error — it silently moves the attention op to the CPU backend. See Troubleshooting.
• --parallel 1 is required, not defensive. llama-server does not default to one slot. The struct default in common.h is 1, but arg.cpp overwrites it to -1 ("auto") for the server example before your command line is parsed, and server.cpp resolves any negative value to n_parallel = 4 with kv_unified = true. A server started with no -np at all therefore comes up with four slots — you can see it in its own startup line, n_slots = 4. Since llama.cpp sizes the GDN state pool at max(1, n_seq_max) slots, dropping this flag quadruples the recurrent state from 0.146 GiB to 0.585 GiB. That 0.438 GiB is the whole cost: the same auto branch sets kv_unified = true, and under it llama-context.cpp takes n_ctx_seq = n_ctx rather than n_ctx / n_seq_max, so the four slots share one full-size attention pool instead of quartering it. Your context window is not divided — only the recurrent state multiplies. Note also that the recurrent state is constructed with a hardcoded GGML_TYPE_F32, so --cache-type-k/--cache-type-v reach only the attention cache and no KV quantization shrinks it.
• --image-min-tokens 1024 is llama.cpp's own advice for this projector family; the loader prints a warning recommending it for grounding tasks, quoted in issue #27124.
• --ctx-size 262144 is the native window. Anything beyond it needs YaRN, and YaRN on this model has a documented ceiling — see Troubleshooting.

The 60-second alternative

ollama pull qwen3.8:27b-q4_K_M     # note the explicit tag — see below

That tag is Q4_K_M and carries the vision projector as its own layer: the registry manifest lists an application/vnd.ollama.image.model blob of 16,810,714,464 bytes plus a projector blob of 931,146,016 bytes. Raise num_ctx explicitly; Ollama will not give you 262K by default.

⚠️ Pull the explicit tag, not the bare one. qwen3.8:27b and qwen3.8:latest are byte-identical to qwen3.8:27b-mtp-q4_K_M — the same manifest sha256, the same weights and projector blobs — and their params blob carries draft_num_predict: 4. So the obvious ollama pull qwen3.8:27b starts you with MTP speculative decoding on, which is the setting Troubleshooting below tells you to measure before adopting, and which the one published Ollama measurement of this model found net-negative. 27b-q4_K_M is a different manifest whose params omit that line; it is the speculation-off baseline you want first.

The VRAM budget, derived

Every figure below comes from bytes, not from a run. KV cost per token is fixed by the architecture: 16 full-attention layers × 2 (K and V) × 4 KV heads × 256 head dimension = 32,768 elements per token — 64 KiB/token at f16, 34 KiB/token at q8_0 (a q8_0 block is 34 bytes per 32 elements).

The Gated DeltaNet layers add a constant 0.146 GiB per sequence, at any context length. That comes from llama.cpp's own sizing: conv state (ssm.conv_kernel − 1) × (ssm.inner_size + 2 × ssm.group_count × ssm.state_size) = 3 × (6,144 + 4,096) = 30,720 elements, plus recurrent state ssm.state_size × ssm.inner_size = 786,432 elements, both held as F32, across 48 layers.

The lead configuration:

That leaves 6.555 GiB of the card's 32 GiB for graph and compute buffers, which are additional and are not derived here. If you would rather spend the surplus on weights than on headroom, Q5_K_M (19.83 GB) lands at 27.985 GiB at the same 262K window, and Q6_K (22.88 GB) reaches 30.826 GiB — arithmetically inside 32 GiB but with under 1.2 GiB left, which is not a margin to plan on.

Results

• Speed: omitted. No first-party or community measurement of this model on a single RTX 5090 exists on any surface searched — all 97 discussions on the model card were fetched individually, and the only two that involve 5090 hardware (#51 and #65) both run 4× RTX 5090 under vLLM, not one card under llama.cpp. If you measure this pair, please post it via /contribute and it will land on /check/qwen3-8-27b/rtx-5090.
• VRAM usage: 25.445 GiB derived working set at the full 262,144-token window, per the table above.
• Quality notes: in discussion #65 a quantizer scored 36 files — their own 16 plus 20 community builds from unsloth, lmstudio-community and ggml-org — against one BF16 reference on a shared harness, and reports that below 10 GB every quant of this model degrades fast, with their 8.5 GB IQ1_M at 76.3% top-1. A 32 GB card never needs to go near that tier, which is the point of the card.

For the full benchmark data, see /check/qwen3-8-27b/rtx-5090.

Is NVFP4 a real path on this card?

Partly, and it is worth stating precisely because the pieces look more ready than they are.

The kernel is real, and it is genuinely consumer-Blackwell. ggml carries GGML_TYPE_NVFP4 as a first-class tensor type, and mmq.cu gates its native FP4 matrix-multiply path on blackwell_mma_available(cc), which admits compute capability ≥ GGML_CUDA_CC_BLACKWELL, defined as 1200 — i.e. sm_120, this card exactly (mmq.cu, common.cuh). This is worth contrasting with vLLM, where a reporter in discussion #51 found the fused GDN decode kernel gated behind an SM100-family check that excluded SM120 until it was relaxed to a capability-80 floor.

The artifact is not there yet. Across the four established GGUF repositories for this model — 12 files in ggml-org, 29 in unsloth, 35 in bartowski, 6 in lmstudio-community, 82 in all — none is NVFP4. The NVFP4 GGUFs that do exist come from individual accounts with three- and four-digit download counts, and none has been cross-checked here.

Ollama's 27b-nvfp4 tag is not what its name suggests. Its registry manifest is byte-identical to 27b-mlx — same config digest, same 1,209 layer digests — a safetensors build of 18.17 GB declaring "file_type":"nvfp4". One artifact, two names, and Ollama's own merged NVFP4 prefill optimisation was benchmarked on an M5 Max. Pulling the nvfp4 tag on a 5090 does not select a Blackwell-specific GGUF path; it selects Ollama's new-engine safetensors build.

Verdict for today: stay on the GGUF path above. The NVFP4 route is not architecturally closed on this card — it is waiting on a publisher.

Troubleshooting

Prefill collapses to tens of tokens per second

If you give K and V different types, or pick a KV type the default build compiles no kernels for (q4_1, q5_0, q5_1), CUDA flash-attention silently declines the op and the graph scheduler runs attention on the CPU. There is no error message — only a prefill that has fallen off a cliff. Note it is the mismatch and the unsupported type that bite, not 4-bit KV as such.

On an RTX 3090, issue #27109 measures prefill on this model architecture dropping from 991–1,276 tokens/s with q8_0/q8_0 to 34–106 tokens/s with --cache-type-k q4_1 --cache-type-v q8_0. Attribution matters here, because the thread says two different things. The issue body attributes the collapse to "kernel selection for 4-bit KV on this hybrid architecture, independent of the shared-memory guard", and its title names q4_1/q4_0 together. A later comment from the same reporter narrows it to two specific guards in fattn.cu — q4_1 being unsupported unless built with GGML_CUDA_FA_ALL_QUANTS, and a separate K->type != V->type rejection — and concludes that matched q4_0/q4_0 stays on the GPU. Reading the source at the pinned commit agrees with the comment rather than the title: ggml_cuda_fattn_kv_type_supported returns true for GGML_TYPE_Q4_0 unconditionally, and the K != V check is a separate guard a few lines above (fattn.cu).

So: use q8_0 for both K and V, and never mix types. That is the only quantized-KV configuration anyone in that thread actually measured on this architecture, and it is what the command above uses. Matched q4_0/q4_0 is permitted by the source but was not measured there — both of the thread's q4_0 datapoints pair it against a q8_0 on the other side, which the K != V guard already explains — so it is not this recipe's recommendation. If you need mixed or q4_1 types, rebuild with -DGGML_CUDA_FA_ALL_QUANTS=ON. That is a compile-time flag, not a version — ggml/CMakeLists.txt declares it OFF by default, and the cuda.Dockerfile that builds the published images never passes it. Upgrading llama.cpp will therefore never enable it, and on the Docker path these types are unavailable no matter how new the tag.

The server exits silently on a very long prompt

Two distinct cliffs, both from issue #27090. The first sits in the 90–100K prefill range and is fixed in b10434 — if you see it, your build is too old. The second appears only under YaRN ×4 pushing toward a 1M window: the process dies at n_tokens = 520,192, just under 2× the 262,144 native context, with no assert and no message. YaRN ×2 to a 524,288-token window completed a 444K-token prompt for the same reporter. The native 262K window this recipe configures is below both cliffs.

Vision requests crash the server

Reported on issue #27124 against a Vulkan build on an AMD Ryzen AI MAX+ 395, with text-only generation unaffected. The reporter states in the same thread that the identical setup works on an NVIDIA 4080 — so this is a Vulkan/AMD problem, not a defect in the model's projector. Note the projector declares clip.projector_type = qwen3vl_merger with all 27 entries of clip.vision.is_deepstack_layers set false, so llama.cpp drives it through the existing Qwen3-VL graph rather than anything model-specific.

MTP speculative decoding makes things slower, or hangs

Multi-token prediction is available in llama.cpp as --spec-type draft-mtp (add --spec-draft-model pointing at mtp-Qwen3.8-27B-Q8_0.gguf if your weights came from ggml-org; unsloth and lmstudio-community builds carry the head inline). Enabling it also allocates a second KV cache holding only the MTP layer, which adds 0.531 GiB at a 262,144-token window. Whether it pays is genuinely unsettled, and the current evidence leans negative:

• Under vLLM on 4× RTX 5090, the author of discussion #51 reports that "MTP buys ~1.8 tokens/step at 80.4% acceptance, but costs ~4× per step."
• Under Ollama the 27b-mtp-q4_K_M and 27b-q4_K_M tags share the same weights blob and differ only by a draft_num_predict: 4 line in their params — a fact confirmable from the registry manifests, and the basis of the measurement in discussion #80, whose author notes "Ollama does activate the MTP heads." and measured the speculative path landing below the non-speculative baseline on unpredictable content.
• Two open llama.cpp reports touch it directly: #27122 describes reproducible CUDA lockups when --spec-type draft-mtp is combined with --split-mode tensor (not applicable to a single card, and absent with --split-mode layer), and #27106 reports draft acceptance falling to ~0.5 on builds b10430–b10435.

Start without it, measure, and add it only if your own numbers justify it.

The model thinks for a very long time

Thinking mode is on by default. The model card documents reasoning_effort for tuning depth and per-request disabling; several threads on the card report long reasoning traces on short prompts. Budget KV accordingly — reasoning traces consume context like any other tokens, which on this model is 34 KiB per 1,024 tokens at q8_0.