Qwen3.8-27B on RTX 5080: a vision-capable 27B inside 16 GB with llama.cpp

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — a 27B dense vision-language model — on a single 16 GB RTX 5080, with the vision projector loaded and a 32,768-token context. The install leads with llama-server from llama.cpp, a 3-bit dynamic GGUF from unsloth, and an 8-bit KV cache.

Hardware data: RTX 5080 (16 GB VRAM) · derived working set 14.59 GiB of 16 GiB · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5080. /check/qwen3-8-27b/rtx-5080 returns verdict: unknown with zero benchmarks, and no measurement of this model on this card surfaced anywhere this research reached — see Results for exactly which spaces were searched and how. Every number below is either a byte count read from the artifact you download or arithmetic derived from llama.cpp's own source, shown in full so you can check it. If you run this, please contribute your numbers — that is how the /check page gets real data.

ℹ️ 16 GB is the tight tier for this model, and the fit is not comfortable. Plenty of 4-bit builds are small enough to load on this card — ten of the twenty surveyed below are under 15.6 GiB. None of them leaves room for the vision projector and a usable KV cache at the same time, which is the constraint that actually decides the quant. The arithmetic is under Picking a quant; the 3-bit tier it forces is a genuine quality cost and is discussed there rather than glossed over.

What this card changes

The RTX 5080 is the fastest of the catalogue's seven NVIDIA 16 GB cards, and it holds exactly as much of this model as the slowest of them. NVIDIA's RTX Blackwell architecture whitepaper puts it at 10,752 CUDA cores on a GB203 die at compute capability 12.0 (sm_120), with 16 GB of GDDR7 on a 256-bit interface — the same bus width and the same capacity as the RTX 5070 Ti this recipe was cloned from, which is why the entire VRAM budget below transfers unchanged.

Of the seven NVIDIA 16 GB cards it is the only one faster than that sibling on both axes. Table 4 (GeForce RTX 5080 vs GeForce RTX 4080 vs GeForce RTX 3080 Specs) gives it 960 GB/sec of peak memory bandwidth on 30 Gbps GDDR7 — the whitepaper states the same figure in prose — against 896 GB/sec for the RTX 5070 Ti in Table 5 (GeForce RTX 5070 Ti vs GeForce RTX 4070 Ti vs GeForce RTX 3070 Ti), i.e. 7.1% more. The same tables give peak FP16 tensor throughput with FP32 accumulate as 112.6 TFLOPS against 87.9 TFLOPS, i.e. 28% more.

Why generation is the phase that cares. Producing one token means reading the whole weight set — 13,441,059,904 bytes of it — out of VRAM, so token generation is bounded by memory bandwidth, not by shader count. Prompt processing and the vision encoder are the other half of the workload and they are compute-bound. Neither half has been measured on this card, or on the RTX 5070 Ti sibling, so this is a statement about which spec governs which phase and not a prediction with a number attached.

And this is why 16 GB cards need separate pages at all. Across the catalogue's seven NVIDIA 16 GB cards — RTX 4060 Ti 16GB, RTX 5060 Ti, RTX 4070 Ti Super, RTX 4080, RTX 4080 Super, RTX 5070 Ti, RTX 5080 — peak memory bandwidth spans 288 GB/s to 960 GB/s, a factor of 3.33, at identical capacity. The floor is the RTX 4060 Ti 16GB: NVIDIA lists it on a 128-bit bus and ASUS gives its memory speed as 18 Gbps, which at 16 bytes per clock is 288 GB/s; the ceiling is the RTX 5080 at 960 GB/sec in the whitepaper's Table 4 (GeForce RTX 5080 vs GeForce RTX 4080 vs GeForce RTX 3080 Specs). Every one of them fits this model at exactly the same quant and context, because capacity is what decides the fit; none of them generates at the same speed, because bandwidth is what decides that. (The tier has two further 16 GB members, the Apple M2 Pro and the Radeon RX 7800 XT; neither is characterised here, so every bandwidth superlative on this page ranges over the seven NVIDIA cards only.)

More of both does not buy context. Every number in the budget below is identical to every other 16 GB card's and so is the quant choice, because the binding constraint is capacity and capacity did not move. What the extra bandwidth changes is how fast tokens come out once the model fits — not whether it fits, and not how much context you get. If you came here hoping the 5080 would let you run 4-bit with vision, the arithmetic under Picking a quant is the same arithmetic and the same answer.

This card has one capability the Ada 16 GB cards lack — and nothing in llama.cpp to spend it on yet. llama.cpp's CUDA build file annotates compute capability 12.0 as "Blackwell, needs CUDA v12.8, FP4 tensor cores" — hardware FP4, which the RTX 4080, RTX 4080 Super and RTX 4070 Ti Super do not have. It is not unique to this card, and the sibling page should not be read as lacking it: the RTX 5070 Ti is the same GB203 generation at the same compute capability and has the same FP4 hardware. You will find unsloth/Qwen3.8-27B-NVFP4 while looking for a way to use it — 90,924 downloads when this page was written — so here is what it is and is not. Its file tree is two .safetensors, eight .json, a chat template and a README: no GGUF at all. It is a compressed-tensors checkpoint for vLLM-class runtimes, not something llama-server can open, and nothing found here establishes a llama.cpp path for it. It would not fit in any case — model.safetensors alone is 22,568,192,096 bytes, 21.019 GiB of weights before the MTP head, the projector or a single KV cell. Within the GGUF world the picture is simpler: across the five publishers surveyed below, 84 GGUF files enumerated between them, not one is NVFP4. Community NVFP4 GGUF repacks do exist outside that set and land in the same size band as the 4-bit builds below — the smallest found, esatapedico's -LOW, is 14.468 GiB — so they fail the same fixed-load arithmetic for the same reason. FP4 silicon is a real advantage on this card; this model is not currently where you collect it.

Requirements

Why a 27B fits at all: the architecture

Qwen3.8-27B is not a conventional 64-layer dense transformer, and that is the whole reason a 16 GB card is even in the conversation. Its config.json declares 64 layers of which 48 are linear_attention and 16 are full_attention, at indices 3, 7, 11 … 63 — the model card describes the pattern as "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))".

llama.cpp implements this as architecture qwen35 and derives the same split independently. In src/models/qwen35.cpp it marks recurrent layers as is_recr_impl[i] = (i < n_layer()) && ((i + 1) % full_attn_interval != 0), with full_attn_interval read from the GGUF key qwen35.full_attention_interval (which is 4 in every published build). Layers 3, 7, 11 … 63 are therefore the only ones that get a KV cache; src/llama-model.cpp confirms this with a layer filter of il < hparams.n_layer() && !hparams.is_recr(il) when it builds the hybrid memory.

The consequence: only 16 layers hold a KV cache instead of 64, so KV costs a quarter of what a conventional 27B would charge. The other 48 layers hold a fixed-size recurrent state that does not grow with context at all.

Count the layers from 64, not from the GGUF's block_count. Publishers disagree here: ggml-org's files declare block_count = 64 and ship the multi-token-prediction head as a separate mtp-*.gguf, while unsloth, bartowski and lmstudio-community declare 65 with nextn_predict_layers = 1 and the head inline. Both describe the same model. llama.cpp reconciles them in llama-hparams.cpp with n_layer() { return n_layer_all - n_layer_nextn; } — commented in the header as "number of effective layers (excludes nextn layers)" — and the MTP block is excluded from the attention cache by the same il < hparams.n_layer() test. Sizing KV off a raw block_count of 65 would give 16.25 layers and overstate the cache by about 6%.

VRAM budget

Every figure here is either a byte count from the HuggingFace tree API, re-fetched for this page rather than inherited, or arithmetic over values read out of the GGUF header and llama.cpp's source.

On units, because this is where fit calculations go wrong. Everything below is GiB (2^30 bytes) on both sides of the comparison. A "16 GB" graphics card is 16 GiB = 16,384 MiB — GPU memory is quoted in binary units, unlike disk capacity, and the whole stack agrees: ggml-cuda.cu prints device memory as prop.totalGlobalMem / (1024 * 1024) labelled MiB, and nvidia-smi reports the same. Do not convert the card's "16 GB" as if it were a decimal figure — that would understate it by 7% and turn a fit into a miss. Two real deductions do apply and both come out of the headroom line rather than the components: cudaDeviceProp::totalGlobalMem sits slightly under the nominal 16,384 MiB, and the display driver reserves more on top — noticeably more on Windows with a monitor attached than on a headless Linux box. Check yours with nvidia-smi --query-gpu=memory.total,memory.used --format=csv before you size the context.

KV cache, per token. 16 full-attention layers × 4 KV heads × 256 head dim × 2 (K and V) = 32,768 elements per token. The GGUF header confirms the shape: qwen35.attention.head_count_kv = 4, qwen35.attention.key_length = 256, qwen35.attention.value_length = 256. At fp16 that is 65,536 bytes per token; at q8_0 (34 bytes per 32-element block) it is 34,816 bytes per token.

Recurrent state, per sequence. llama_hparams::n_embd_r() returns (ssm_d_conv - 1) * (ssm_d_inner + 2 * ssm_n_group * ssm_d_state) = 3 × (6144 + 2×16×128) = 30,720 elements of convolution state, and n_embd_s() returns ssm_d_state * ssm_d_inner = 128 × 6144 = 786,432 elements of recurrent state, per layer. Both are allocated as GGML_TYPE_F32 — the constructor call in llama-model.cpp passes GGML_TYPE_F32 for both recurrent_type_k and recurrent_type_v. Across 48 layers for one sequence that is 48 × (30,720 + 786,432) × 4 = 156,893,184 bytes, and it is constant regardless of context length.

The whole budget at 32,768 context, --parallel 1, q8_0 KV:

That 1.41 GiB is what the CUDA context, the compute buffers, the vision encoder graph and the display driver's own reservation have to live in. It is a reservation, not a measurement — llama.cpp allocates those from the same pool and their size depends on your batch and image sizes, and none of it was measured on this card. This is the number that decides whether the configuration actually runs, and it is the one number here that is not derived from a byte count; treat 32K as a starting point to verify rather than a guarantee. If your run overflows, the ladder under Troubleshooting walks you down.

One thing works in your favour that the table does not credit: llama.cpp skips the model's multi-token-prediction block unless you actually ask for speculation. qwen35.cpp creates those tensors with int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0;, so the extra blk.64 carried inside the unsloth, bartowski and lmstudio-community files is on disk but not in VRAM by default.

Picking a quant on 16 GB

There is no first-party GGUF. Qwen publishes Qwen3.8-27B and an FP8 sibling as safetensors only; filtering their HF org to "Qwen3.8" returns exactly four repositories and none of them is a -GGUF, while the same org's GGUF-filtered listing returns 54 repositories for other model families. Both listings were re-run for this page. Every GGUF below is a community conversion, so the publisher is part of what you are choosing.

Sizes below are size fields from https://huggingface.co/api/models/<repo>/tree/main?recursive=true, re-fetched for this page rather than copied from a sibling recipe, converted to GiB.

Scope of this survey, so you know what it does and does not cover. HuggingFace lists over 200 GGUF repositories matching "Qwen3.8-27B" — abliterated forks, distills, ROCm/NVFP4 repacks, and one publisher shipping 90+ single-quant split repos. Enumerating that tail is not feasible and most of it is not this model. What follows is every 4-bit build from the five publishers this recipe surveys — ggml-org, unsloth, bartowski, lmstudio-community and AtomicChat: 20 files, enumerated from the tree API rather than sampled. Claims below are bounded to that set.

The quants worth considering on this card. The 3-bit tier, plus the two 4-bit entries a reader is most likely to reach for:

The complete 4-bit set, so the claim below is reproducible. All 20 four-bit files from the five publishers, ascending. Ten are under 15.6 GiB:

Entries 14 and up exceed the card as weights alone — unsloth's Q4_1 at 16.336 GiB is already over, which is worth checking against the table rather than taking on trust.

Why the 4-bit tier is out, stated as arithmetic rather than as a size floor. Ten of the twenty 4-bit builds are under 15.6 GiB, so "4-bit is too big to load" would be false. The binding constraint is what has to sit beside the weights: projector 0.864 + recurrent state 0.146 + KV at 32K q8_0 1.062 = 2.073 GiB of fixed load (2,225,351,360 bytes). Add that to the smallest uniform 4-bit build, bartowski's IQ4_XS at 14.499, and you get 16.571 GiB — over the card before a single byte of runtime buffer. Shrinking the context does not rescue it: at 16K the same build totals 16.04 GiB, still over; at 8K it totals 15.774 GiB, leaving 0.226 GiB for every compute buffer, the CUDA context, the vision graph and the display driver combined. That is the real reason this recipe is 3-bit, and it holds for all twenty. (These totals are summed from unrounded byte counts and rounded once at the end, which is why two of them sit a thousandth below the figures on the sibling 5070 Ti page, where the rounded GiB values were added instead.)

Do not read "Q4_K_M" as one number. Across these publishers, files named Q4_K_M alone span 15.656 to 17.671 GiB — a 2.0 GiB spread on one label — and the spread is not noise. ggml-org's convert.log shows their build was quantised with --pure --tensor-type output.weight=q6_k --tensor-type shexp=q8_0 --tensor-type latent=q8_0 --tensor-type attn_=q8_0 --tensor-type ssm_=q8_0, i.e. large parts of it are actually q8_0. A quant name is a recipe the publisher chose, not a size. Always name the publisher and the file — and note that a range measured on one quant name says nothing about its tier.

On quality at this tier, the only comparative measurement found is vendor-published and self-interested: AtomicChat, whose founder posted it and who sell a competing quant set, ran 20 community files plus 16 of their own through one harness on 4× RTX 5090 against their own BF16 logits, and report unsloth's Q3_K_M at 0.0484 mean KLD against their same-size file at 0.0325, with their 13.8 GB build reaching "92.4% top-1". Treat the ranking as a vendor claim. The useful, publisher-independent part is their statement that "below 10 GB every quant of this model degrades fast" — which is why this recipe does not send you to a 2-bit build to buy context.

Installation

1. Build or fetch llama.cpp with CUDA

There are three different "which build" claims in circulation for this model, and only one of them is a minimum. Keeping them apart matters, because a reader on a build a few tags old should not conclude they are unsupported.

1. Architecture support floor — old, and any current build clears it. Both structural requirements are already present at b8001, the oldest tag checked: src/llama-arch.cpp carries qwen35 and tools/mtmd/clip-impl.h carries the qwen3vl_merger projector type. Both files returned HTTP 200 at that tag with non-empty bodies, re-checked for this page, so this is a real reading and not a 404 misread as absence. b8001 is the floor of the window searched, not the landing build — the true one is older and was not established here. Practically: if your llama.cpp is recent enough to be called recent, it supports this model.
2. A functional floor for a specific capability — none applies to this configuration. This recipe does not depend on any post-release fix that was sourced to a bug report. If you go looking for very long context you may find one; nothing in the path documented here does.
3. The build this page was authored against — b10442. That is provenance for the source line numbers and flag names quoted throughout, not a requirement. Use it if you want byte-identical footing with this text.

The toolkit floor for this card, from llama.cpp's own build file. ggml/src/ggml-cuda/CMakeLists.txt annotates the architecture list it compiles, and the entry for this card reads "Blackwell, needs CUDA v12.8, FP4 tensor cores"; the surrounding logic appends 120a-real only when CUDAToolkit_VERSION is at least 12.8. That is the whole requirement, and it is worth being precise because the comment block around it is easy to misread: the file does list CMake ≥ 3.31.8 (below 4.0.0) or ≥ 4.0.2 as needed versions, but that paragraph is about 120f-virtual, an architecture the project explicitly declines to ship — its very next line records that "120a-real and 121a-real should work with basically any CMake version", and 120a-real is what the build actually appends. So: CUDA 12.8+, no CMake floor. The 40-series siblings of this recipe need only CUDA 11.8, which is the one place an Ada card is less constrained than a 50-series one.

The one genuine build-time requirement in this area is a compile flag, not a version — and it does not apply here either. GGML_CUDA_FA_ALL_QUANTS defaults to OFF in ggml/CMakeLists.txt, and a default CUDA build compiles only four FlashAttention vector instances: f16-f16, q4_0-q4_0, q8_0-q8_0 and bf16-bf16. That list is produced by an else() branch in ggml/src/ggml-cuda/CMakeLists.txt that names the four files literally and is not conditioned on CMAKE_CUDA_ARCHITECTURES — so it is the same set on sm_120 as on any other target, and this is one piece of the sibling recipe's reasoning that transfers to your card without adjustment. This recipe's matched q8_0/q8_0 cache is in that default set, so no flag and no newer build is needed. Mixed or q4_1/q5_x cache types are not — which is the mechanism behind the prefill collapse in Troubleshooting.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10442   # optional — pins you to the exact tree this page quotes; plain `master` is fine
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Video input is compiled in by default — the mtmd CMakeLists.txt sets MTMD_VIDEO ON with the description "enable video support in mtmd (requires ffmpeg binary in PATH)". If you do not have ffmpeg, images still work; only video is affected.

A prebuilt container works too, and is what the ghcr.io/ggml-org/llama.cpp:server-cuda13 image is for.

2. Download the weights and the projector

pip install -U huggingface_hub
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q3_K_XL.gguf --local-dir ./qwen38
hf download unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf --local-dir ./qwen38

Both files must come down. Without mmproj-F16.gguf the server loads text-only and rejects images with image input is not supported - hint: if this is unexpected, you may need to provide the mmproj.

Running

./build/bin/llama-server \
  --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf \
  --mmproj ./qwen38/mmproj-F16.gguf \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --image-min-tokens 1024 \
  --jinja \
  --no-reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

Four of those flags are doing load-bearing work on a 16 GB card and are not defaults:

• --parallel 1. This one is easy to skip and it is worth 0.438 GiB. The common_params struct initialises n_parallel to 1, but that is not the value llama-server runs with: common/arg.cpp overrides it in its per-example defaults block — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default } — and tools/server/server.cpp then resolves any negative value with "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true". You do not have to type anything to get four slots; you have to type --parallel 1 to avoid them. The startup log in issue #27124, from a command line carrying no -np flag at all, prints n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true' — which is the behaviour, observed rather than inferred. Because the recurrent state is allocated per sequence — llama_memory_hybrid gets recurrent_kv_size = std::max((uint32_t) 1, cparams.n_seq_max) — the default costs 627,572,736 bytes instead of 156,893,184: 0.438 GiB, or about 13,500 more tokens of context. Note also that this state is hardcoded GGML_TYPE_F32 in llama-model.cpp, so --cache-type-k/v does not shrink it — those flags reach only the 16 attention layers.

  The KV cache itself is not multiplied by the slot count, and it is worth knowing why the two behave differently. The auto path also flips kv_unified to true, and llama-context.cpp then sets n_ctx_seq = n_ctx rather than n_ctx / n_seq_max — so -c 32768 allocates 32,768 cells either way. What changes is who owns them: at the default, four slots share one 32,768-token window, so a single long conversation can be evicted by a concurrent request. With --parallel 1 the whole window belongs to one conversation and the recurrent state drops to one copy. On a single-user desktop the flag costs you nothing you wanted.

• --cache-type-k q8_0 --cache-type-v q8_0. Halves KV against fp16. Do not go to 4-bit KV, and do not mix the two types — see Troubleshooting.

• --chat-template-kwargs '{"reasoning_effort":"medium"}' and --no-reasoning-preserve. Thinking is on by default and the model card's default effort is xhigh; the card documents "supported levels are xhigh, medium, and low". Preserved thinking is also on by default and "retains thinking blocks from all historical messages", which on this card means every past reasoning trace keeps occupying your 32K window. Both are context-budget decisions, not quality opinions — raise them if you have context to spare.

• --image-min-tokens 1024. llama.cpp itself warns at load time that "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks" — the string is in tools/mtmd/clip.cpp, emitted whenever image_min_pixels is below that threshold. Budget for it: at 1024 tokens and q8_0 KV, each image costs about 35 MB of cache.

The sampler values are the model card's own thinking-mode recommendation (temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0).

Point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Images go in as image_url content parts; GET /props reports whether the build accepted the projector and whether video is available.

Text-only, if you want the context back

Dropping the projector frees 927,607,488 bytes — about 26,600 more tokens of q8_0 KV. It is a real option on this card and costs you the model's defining capability:

./build/bin/llama-server --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf --no-mmproj \
  --n-gpu-layers 99 --ctx-size 65536 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja

Why not Ollama on this card

Ollama's library page for this model lists twelve tags and the smallest is 18 GB — re-checked for this page — so there is no tag below the card's capacity and any of them will partially offload to system RAM. A user in discussion #80 reports the 27b-q4_K_M weights blob is 16.81 GB, which is the same figure lmstudio-community's Q4_K_M rounds to (16,810,714,336 B = 16.81 GB) — a match to the precision quoted, not a verified identity of the blobs; adding a 0.93 GB projector accounts for the displayed 18 GB, though that last step is inference on my part rather than something Ollama documents. Either way the tag does not fit. Pull the GGUF directly instead.

Results

• Speed: omitted, and the absence is bounded rather than assumed. Three spaces were searched. (1) /check/qwen3-8-27b/rtx-5080 returns verdict: unknown with zero benchmarks. (2) All 98 discussions on the model card were fetched individually through the JSON API — enumerated, not sampled, and not keyword-searched from a listing page — and 83,110 characters of body text scanned: the RTX 5080 is named in none of them. The same scan run as a control does find 5070, 4090 and 3090 in four separate threads, so the zero is a real absence and not a broken search. (3) The llama.cpp issue tracker returns nothing for this card, and the one trap there is worth naming: searching the repo for this card's model number also matches the repository's own closed pull request #5080, "Server enhancements - grammar segfault and helper titles.", because the token matches an issue number. That is the whole space this sentence covers; a measurement posted anywhere else would not appear in it. If you run this configuration, please post your numbers via /contribute so /check/qwen3-8-27b/rtx-5080 stops being empty.
• VRAM usage: derived, not measured — 14.59 GiB of 16 GiB at 32K context with the projector loaded, itemised in the budget table above. Live data, when it exists, will be at /check/qwen3-8-27b/rtx-5080.
• Quality notes: this card runs the model at 3 bits — see Picking a quant for why 4-bit cannot coexist with the projector — and vision is the part that suffers first. In discussion #69 a community user (a reply in the thread, not its opener) reports that "the multi modal ability is inferior to 3.6 27b when comes to 4 bit quant" — that is a comparison to the previous generation at 4 bits, one tier above what fits here, and no equivalent report exists at 3 bits either way. Expect the text side to hold up better than the vision side, and treat OCR and fine grounding work as things to verify before relying on.

For the full benchmark data, see /check/qwen3-8-27b/rtx-5080.

Troubleshooting

Prompt processing collapses to tens of tokens per second

You quantised the KV cache to 4 bits, or you mixed two cache types. llama.cpp issue #27109, open as of writing, reports that switching K from q8_0 to q4_1 on a qwen35 hybrid model drops prefill from 991–1276 t/s to 34–106 t/s while generation is unaffected — measured by the reporter on an RTX 3090, and attributed in the report to "the CUDA flash-attention / quantized-KV dequant kernel selection for 4-bit KV on this hybrid architecture" rather than to anything specific to that card. Community advice to run "128K context at q4 KV" on 16 GB predates this report; take the shorter context and keep q8_0.

Read the thread, not the title. The issue is titled as a q4_1/q4_0 problem, and the reporter's own later root-cause comment narrows it: on a default build, K=q8_0, V=q8_0 and K=q4_0, V=q4_0 are both fine and stay on the GPU. What fails is an unsupported type or a mismatched pair. Both guards are visible in ggml/src/ggml-cuda/fattn.cu: ggml_cuda_fattn_kv_type_supported() returns false for Q4_1, Q5_0 and Q5_1 unless GGML_CUDA_FA_ALL_QUANTS is defined, and a few lines above it a second #ifndef GGML_CUDA_FA_ALL_QUANTS block rejects any configuration where K->type != V->type. Either rejection returns BEST_FATTN_KERNEL_NONE, the scheduler moves the whole attention op to the CPU backend, and nothing is logged — you just get a thirty-fold prefill regression. Neither guard is conditioned on compute capability, so this behaves the same on sm_120 as on the RTX 3090 the report was filed from. Rebuilding with -DGGML_CUDA_FA_ALL_QUANTS=ON compiles the rest; using matched q8_0/q8_0, as this recipe does, avoids the question entirely and costs nothing.

It OOMs at startup

The ladder, in the order that costs you least:

1. Drop --ctx-size to 16384. At q8_0 that halves the KV line to 0.531 GiB and buys back 0.531 GiB.
2. Confirm --parallel 1 is actually on the command line — the auto default is 4 and costs 0.438 GiB of recurrent state.
3. Drop the projector with --no-mmproj and run text-only: 0.864 GiB.
4. Drop a quant tier to Qwen3.8-27B-UD-Q3_K_XL's smaller siblings — unsloth's Q3_K_S is 11.711 GiB and UD-IQ3_XXS is 11.095 GiB, but note the quality warning above about going below ~10 GB.

Free VRAM is reported as system RAM, and llama.cpp sizes itself for a card you don't have

Blackwell-specific, mostly Linux-specific, and worth knowing about precisely because this recipe leaves only 1.410 GiB of headroom. llama.cpp issue #26901 — open, community-reported, zero comments and no maintainer response as of writing — names the RTX 5080 in its title and reports that discrete Blackwell GPUs come back from cudaGetDeviceProperties with prop.integrated > 0, which llama.cpp reads as "this is an iGPU".

Its headline symptom cannot reach you here. The reported breakage is that only the first iGPU-classified device is added to the device list, so --tensor-split silently does nothing across two cards. This recipe is a single card and passes no split flag.

A second consequence is not multi-GPU-gated, and that one can. In ggml/src/ggml-cuda/ggml-cuda.cu the free/total memory query computes is_uma = prop.integrated > 0 || uma_env and, when true, replaces the free-VRAM figure with available system memory read out of /proc/meminfo — a branch introduced for genuine unified-memory machines. There is no device-count test on it. It is, however, wrapped in #if defined(__linux__), verified by reading the file rather than taken on trust: on Windows and macOS that branch is compiled out entirely and this particular misreport cannot occur. So: single card, Linux, Blackwell — in scope; single card, Windows — not.

The classification itself is not platform-gated, and it has one cross-platform side effect worth naming. ggml_backend_cuda_device_get_type() returns prop.integrated ? GGML_BACKEND_DEVICE_TYPE_IGPU : GGML_BACKEND_DEVICE_TYPE_GPU with no __linux__ guard, and the device's advertised capabilities are derived from the result — .mmap_support is set to props->type != GGML_BACKEND_DEVICE_TYPE_IGPU. A card misclassified as integrated therefore advertises no mmap support on any platform, which changes how the 12.518 GiB of weights are brought in at load time. It costs nothing in VRAM and is not why anyone files a bug, but it is the part of this a Windows reader is still exposed to.

Why this recipe is insulated, stated as a mechanism rather than a hope. llama.cpp's auto-fitter acts only on arguments you did not set, and it says so by aborting: common/fit.cpp throws n_gpu_layers already set by user to <n>, abort as soon as mparams->n_gpu_layers differs from the default, and common/fit.h documents that the context size is modified if and only if it is 0. The command above pins --n-gpu-layers 99 — the default is -1 — and --ctx-size 32768, so the fitter bails out before it can act on any memory figure at all, correct or not. Drop those two flags on a Blackwell card under Linux and that protection is gone, which makes the reader most likely to be bitten the one who typed the least.

Two honest caveats. The claim that discrete Blackwell reports prop.integrated > 0 is the reporter's observation on his own hardware; it was not verified here, no maintainer has confirmed it, and everything above is a statement about what the code does if it is true. It is structurally plausible rather than merely asserted, though — llama.cpp already distrusts that field elsewhere, force-setting its cached integrated flag to false on every non-HIP build with the comment // Temporarily disabled due to issues with corrupted output, while the two call sites above read cudaGetDeviceProperties directly and bypass that override. If you can check, comparing nvidia-smi --query-gpu=memory.total,memory.used --format=csv against llama.cpp's own startup log is the one-command test, and a mismatch is worth adding to the issue.

Turning on MTP makes it slower, not faster

The model ships a multi-token-prediction head, and llama.cpp does wire it into speculative decoding — --spec-type draft-mtp works against the inline blk.64 block with no separate draft file. On a 16 GB card, leave it off. It costs VRAM: enabling it sets n_rs_seq to your draft length, and llama_memory_recurrent allocates mem_size * (1 + n_rs_seq) rows of recurrent state, so --spec-draft-n-max 4 takes the recurrent line from 156,893,184 bytes to 784,465,920 — 0.584 GiB gone. And the payoff is doubtful on the GGUF path: in discussion #80 a user measuring the same weights blob with speculation off and on found "under Ollama, speculation repays its own overhead in the best case and never more", with unpredictable content running 2.3× slower than with no speculation at all. It is also implicated in two open CUDA stability reports on this exact model — #27122 and #27105 — both of which involve MTP, and neither of which is on this card.

The model thinks for minutes on a trivial question

Widely reported — one thread is titled "After waiting 49 minutes and 16 seconds while the model was still thinking, I'm simply giving up...". The model card's own control is the lever: reasoning_effort accepts xhigh (the default), medium and low, and thinking can be turned off entirely per request. On a card whose entire KV cache is the 1.062 GiB budgeted above this is not a comfort setting — a runaway reasoning trace is what will exhaust your context.

Tool calls fail in agent clients

Multiple reports tie this to the packaged chat template rather than to the quant: llama.cpp issue #27139 is titled "Qwen3.8 Codex error resolved by using the Qwen3.6 chat template file", and discussion #68 reports the same class of failure with a community-maintained replacement template. If your client's tool calling misbehaves, try --chat-template-file with a corrected template before blaming the 3-bit weights.

Your tool says the file is Q4_K_S

A metadata defect in unsloth's Dynamic builds, verified by reading the GGUF header directly: Qwen3.8-27B-UD-Q3_K_XL.gguf carries general.file_type = 14, which is Q4_K_S, while the same repo's plain Qwen3.8-27B-Q3_K_M.gguf correctly carries 12. It is a label, not the tensors — the file is 12.518 GiB, which no real Q4_K_S of this model could be. It affects display only.

Vision crashes

One open vision-crash report exists, llama.cpp issue #27124, and it is on a Vulkan build on an AMD Ryzen AI MAX+ 395 under Windows — a different backend and a different vendor from this recipe's CUDA path. No CUDA equivalent surfaced in the llama.cpp issues opened since the model's release. If you hit one on CUDA, that is new information worth filing via /contribute as well as upstream.