Qwen3.8-27B on RTX 5070 Ti: a vision-capable 27B inside 16 GB with llama.cpp

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — a 27B dense vision-language model — on a single 16 GB RTX 5070 Ti, with the vision projector loaded and a 32,768-token context. The install leads with llama-server from llama.cpp, a 3-bit dynamic GGUF from unsloth, and an 8-bit KV cache.

Hardware data: RTX 5070 Ti (16 GB VRAM) · derived working set 14.590 GiB of 16 GiB · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5070 Ti. /check/qwen3-8-27b/rtx-5070-ti returns verdict: unknown with zero benchmarks, and no first-party or community measurement for this card surfaced during research. Every number below is either a byte count read from the artifact you download or arithmetic derived from llama.cpp's own source, shown in full so you can check it. If you run this, please contribute your numbers — that is how the /check page gets real data.

ℹ️ 16 GB is the tight tier for this model, and the fit is not comfortable. Plenty of 4-bit builds are small enough to load on this card — ten of the twenty surveyed below are under 15.6 GiB. None of them leaves room for the vision projector and a usable KV cache at the same time, which is the constraint that actually decides the quant. The arithmetic is under Picking a quant; the 3-bit tier it forces is a genuine quality cost and is discussed there rather than glossed over.

Requirements

Why a 27B fits at all: the architecture

Qwen3.8-27B is not a conventional 64-layer dense transformer, and that is the whole reason a 16 GB card is even in the conversation. Its config.json declares 64 layers of which 48 are linear_attention and 16 are full_attention, at indices 3, 7, 11 … 63 — the model card describes the pattern as "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))".

llama.cpp implements this as architecture qwen35 and derives the same split independently. In src/models/qwen35.cpp it marks recurrent layers as is_recr_impl[i] = (i < n_layer()) && ((i + 1) % full_attn_interval != 0), with full_attn_interval read from the GGUF key qwen35.full_attention_interval (which is 4 in every published build). Layers 3, 7, 11 … 63 are therefore the only ones that get a KV cache; src/llama-model.cpp confirms this with a layer filter of il < hparams.n_layer() && !hparams.is_recr(il) when it builds the hybrid memory.

The consequence: only 16 layers hold a KV cache instead of 64, so KV costs a quarter of what a conventional 27B would charge. The other 48 layers hold a fixed-size recurrent state that does not grow with context at all.

Count the layers from 64, not from the GGUF's block_count. Publishers disagree here: ggml-org's files declare block_count = 64 and ship the multi-token-prediction head as a separate mtp-*.gguf, while unsloth, bartowski and lmstudio-community declare 65 with nextn_predict_layers = 1 and the head inline. Both describe the same model. llama.cpp reconciles them in llama-hparams.cpp with n_layer() { return n_layer_all - n_layer_nextn; } — commented in the header as "number of effective layers (excludes nextn layers)" — and the MTP block is excluded from the attention cache by the same il < hparams.n_layer() test. Sizing KV off a raw block_count of 65 would give 16.25 layers and overstate the cache by about 6%.

VRAM budget

Every figure here is either a byte count from the HuggingFace tree API or arithmetic over values read out of the GGUF header and llama.cpp's source.

On units, because this is where fit calculations go wrong. Everything below is GiB (2^30 bytes) on both sides of the comparison. A "16 GB" graphics card is 16 GiB = 16,384 MiB — GPU memory is quoted in binary units, unlike disk capacity, and the whole stack agrees: ggml-cuda.cu prints device memory as prop.totalGlobalMem / (1024 * 1024) labelled MiB, and nvidia-smi reports the same. Do not convert the card's "16 GB" as if it were a decimal figure — that would understate it by 7% and turn a fit into a miss. Two real deductions do apply and both come out of the headroom line rather than the components: cudaDeviceProp::totalGlobalMem sits slightly under the nominal 16,384 MiB, and the display driver reserves more on top — noticeably more on Windows with a monitor attached than on a headless Linux box. Check yours with nvidia-smi --query-gpu=memory.total,memory.used --format=csv before you size the context.

KV cache, per token. 16 full-attention layers × 4 KV heads × 256 head dim × 2 (K and V) = 32,768 elements per token. The GGUF header confirms the shape: qwen35.attention.head_count_kv = 4, qwen35.attention.key_length = 256, qwen35.attention.value_length = 256. At fp16 that is 65,536 bytes per token; at q8_0 (34 bytes per 32-element block) it is 34,816 bytes per token.

Recurrent state, per sequence. llama_hparams::n_embd_r() returns (ssm_d_conv - 1) * (ssm_d_inner + 2 * ssm_n_group * ssm_d_state) = 3 × (6144 + 2×16×128) = 30,720 elements of convolution state, and n_embd_s() returns ssm_d_state * ssm_d_inner = 128 × 6144 = 786,432 elements of recurrent state, per layer. Both are allocated as GGML_TYPE_F32 — the constructor call in llama-model.cpp passes GGML_TYPE_F32 for both recurrent_type_k and recurrent_type_v. Across 48 layers for one sequence that is 48 × (30,720 + 786,432) × 4 = 156,893,184 bytes, and it is constant regardless of context length.

The whole budget at 32,768 context, --parallel 1, q8_0 KV:

That 1.410 GiB is what the CUDA context, the compute buffers, the vision encoder graph and the display driver's own reservation have to live in. It is a reservation, not a measurement — llama.cpp allocates those from the same pool and their size depends on your batch and image sizes, and none of it was measured on this card. This is the number that decides whether the configuration actually runs, and it is the one number here that is not derived from a byte count; treat 32K as a starting point to verify rather than a guarantee. If your run overflows, the ladder under Troubleshooting walks you down.

One thing works in your favour that the table does not credit: llama.cpp skips the model's multi-token-prediction block unless you actually ask for speculation. qwen35.cpp creates those tensors with int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0;, so the extra blk.64 carried inside the unsloth, bartowski and lmstudio-community files is on disk but not in VRAM by default.

Picking a quant on 16 GB

There is no first-party GGUF. Qwen publishes Qwen3.8-27B and an FP8 sibling as safetensors only; their HF org carries 54 -GGUF repositories and none of them is for Qwen3.8. Every GGUF below is a community conversion, so the publisher is part of what you are choosing.

Sizes below are size fields from https://huggingface.co/api/models/<repo>/tree/main?recursive=true, converted to GiB.

Scope of this survey, so you know what it does and does not cover. HuggingFace lists over 200 GGUF repositories matching "Qwen3.8-27B" — abliterated forks, distills, ROCm/NVFP4 repacks, and one publisher shipping 90+ single-quant split repos. Enumerating that tail is not feasible and most of it is not this model. What follows is every 4-bit build from the five publishers this recipe surveys — ggml-org, unsloth, bartowski, lmstudio-community and AtomicChat: 20 files, enumerated from the tree API rather than sampled. Claims below are bounded to that set.

The quants worth considering on this card. The 3-bit tier, plus the two 4-bit entries a reader is most likely to reach for:

The complete 4-bit set, so the claim below is reproducible. All 20 four-bit files from the five publishers, ascending. Ten are under 15.6 GiB:

Entries 15 and up exceed the card as weights alone.

Why the 4-bit tier is out, stated as arithmetic rather than as a size floor. Ten of the twenty 4-bit builds are under 15.6 GiB, so "4-bit is too big to load" would be false. The binding constraint is what has to sit beside the weights: projector 0.864 + recurrent state 0.146 + KV at 32K q8_0 1.063 = 2.073 GiB of fixed load. Add that to the smallest uniform 4-bit build, bartowski's IQ4_XS at 14.499, and you get 16.572 GiB — over the card before a single byte of runtime buffer. Shrinking the context does not rescue it: at 16K the same build totals 16.040 GiB, still over; at 8K it totals 15.775 GiB, leaving 0.225 GiB for every compute buffer, the CUDA context, the vision graph and the display driver combined. That is the real reason this recipe is 3-bit, and it holds for all twenty.

Do not read "Q4_K_M" as one number. Across these publishers, files named Q4_K_M alone span 15.656 to 17.671 GiB — a 2.0 GiB spread on one label — and the spread is not noise. ggml-org's convert.log shows their build was quantised with --pure --tensor-type output.weight=q6_k --tensor-type shexp=q8_0 --tensor-type latent=q8_0 --tensor-type attn_=q8_0 --tensor-type ssm_=q8_0, i.e. large parts of it are actually q8_0. A quant name is a recipe the publisher chose, not a size. Always name the publisher and the file — and note that a range measured on one quant name says nothing about its tier, which is exactly the error this section was rewritten to remove.

On quality at this tier, the only comparative measurement found is vendor-published and self-interested: AtomicChat, whose founder posted it and who sell a competing quant set, ran 20 community files plus 16 of their own through one harness on 4× RTX 5090 against their own BF16 logits, and report unsloth's Q3_K_M at 0.0484 mean KLD against their same-size file at 0.0325, with their 13.8 GB build reaching "92.4% top-1". Treat the ranking as a vendor claim. The useful, publisher-independent part is their statement that "below 10 GB every quant of this model degrades fast" — which is why this recipe does not send you to a 2-bit build to buy context.

Installation

1. Build or fetch llama.cpp with CUDA

There are three different "which build" claims in circulation for this model, and only one of them is a minimum. Keeping them apart matters, because a reader on a build a few tags old should not conclude they are unsupported.

1. Architecture support floor — old, and any current build clears it. Both structural requirements are already present at b8001, the oldest tag checked: src/llama-arch.cpp carries qwen35 and tools/mtmd/clip-impl.h carries the qwen3vl_merger projector type. Both files returned HTTP 200 at that tag, so this is a real reading and not a 404 misread as absence. b8001 is the floor of the window searched, not the landing build — the true one is older and was not established here. Practically: if your llama.cpp is recent enough to be called recent, it supports this model.
2. A functional floor for a specific capability — none applies to this configuration. This recipe does not depend on any post-release fix that was sourced to a bug report. If you go looking for very long context you may find one; nothing in the path documented here does.
3. The build this page was authored against — b10442. That is provenance for the source line numbers and flag names quoted throughout, not a requirement. Use it if you want byte-identical footing with this text.

The one genuine build-time requirement in this area is a compile flag, not a version — and it does not apply here either. GGML_CUDA_FA_ALL_QUANTS defaults to OFF in ggml/CMakeLists.txt, and a default CUDA build compiles only four FlashAttention vector instances: f16-f16, q4_0-q4_0, q8_0-q8_0 and bf16-bf16. This recipe's matched q8_0/q8_0 cache is in that default set, so no flag and no newer build is needed. Mixed or q4_1/q5_x cache types are not — which is the mechanism behind the prefill collapse in Troubleshooting, and no version bump substitutes for the flag.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10442   # optional — pins you to the exact tree this page quotes; plain `master` is fine
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Video input is compiled in by default — the mtmd CMakeLists.txt sets MTMD_VIDEO ON with the description "enable video support in mtmd (requires ffmpeg binary in PATH)". If you do not have ffmpeg, images still work; only video is affected.

A prebuilt container works too, and is what the ghcr.io/ggml-org/llama.cpp:server-cuda13 image is for.

2. Download the weights and the projector

pip install -U huggingface_hub
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q3_K_XL.gguf --local-dir ./qwen38
hf download unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf --local-dir ./qwen38

Both files must come down. Without mmproj-F16.gguf the server loads text-only and rejects images with image input is not supported - hint: if this is unexpected, you may need to provide the mmproj.

Running

./build/bin/llama-server \
  --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf \
  --mmproj ./qwen38/mmproj-F16.gguf \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --image-min-tokens 1024 \
  --jinja \
  --no-reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

Four of those flags are doing load-bearing work on a 16 GB card and are not defaults:

• --parallel 1. This one is easy to skip and it is worth 0.438 GiB. The common_params struct initialises n_parallel to 1, but that is not the value llama-server runs with: common/arg.cpp overrides it in its per-example defaults block — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default } — and tools/server/server.cpp then resolves any negative value with "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true". You do not have to type anything to get four slots; you have to type --parallel 1 to avoid them. The startup log in issue #27124, from a command line carrying no -np flag at all, prints n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true' — which is the behaviour, observed rather than inferred. Because the recurrent state is allocated per sequence — llama_memory_hybrid gets recurrent_kv_size = std::max((uint32_t) 1, cparams.n_seq_max) — the default costs 627,572,736 bytes instead of 156,893,184: 0.438 GiB, or about 13,500 more tokens of context. Note also that this state is hardcoded GGML_TYPE_F32 in llama-model.cpp, so --cache-type-k/v does not shrink it — those flags reach only the 16 attention layers.

  The KV cache itself is not multiplied by the slot count, and it is worth knowing why the two behave differently. The auto path also flips kv_unified to true, and llama-context.cpp then sets n_ctx_seq = n_ctx rather than n_ctx / n_seq_max — so -c 32768 allocates 32,768 cells either way. What changes is who owns them: at the default, four slots share one 32,768-token window, so a single long conversation can be evicted by a concurrent request. With --parallel 1 the whole window belongs to one conversation and the recurrent state drops to one copy. On a single-user desktop the flag costs you nothing you wanted.

• --cache-type-k q8_0 --cache-type-v q8_0. Halves KV against fp16. Do not go to 4-bit KV — see Troubleshooting.

• --chat-template-kwargs '{"reasoning_effort":"medium"}' and --no-reasoning-preserve. Thinking is on by default and the model card's default effort is xhigh; the card documents "supported levels are xhigh, medium, and low". Preserved thinking is also on by default and "retains thinking blocks from all historical messages", which on this card means every past reasoning trace keeps occupying your 32K window. Both are context-budget decisions, not quality opinions — raise them if you have context to spare.

• --image-min-tokens 1024. llama.cpp itself warns at load time that "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks" — the string is in tools/mtmd/clip.cpp, emitted whenever image_min_pixels is below that threshold. Budget for it: at 1024 tokens and q8_0 KV, each image costs about 35 MB of cache.

The sampler values are the model card's own thinking-mode recommendation (temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0).

Point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Images go in as image_url content parts; GET /props reports whether the build accepted the projector and whether video is available.

Text-only, if you want the context back

Dropping the projector frees 927,607,488 bytes — about 26,600 more tokens of q8_0 KV. It is a real option on this card and costs you the model's defining capability:

./build/bin/llama-server --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf --no-mmproj \
  --n-gpu-layers 99 --ctx-size 65536 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja

Why not Ollama on this card

Ollama's library page for this model lists twelve tags and the smallest is 18 GB — there is no tag below the card's capacity, so any of them will partially offload to system RAM. A user in discussion #80 reports the 27b-q4_K_M weights blob is 16.81 GB, which is the same figure lmstudio-community's Q4_K_M rounds to (16,810,714,336 B = 16.81 GB) — a match to the precision quoted, not a verified identity of the blobs; adding a 0.93 GB projector accounts for the displayed 18 GB, though that last step is inference on my part rather than something Ollama documents. Either way the tag does not fit. Pull the GGUF directly instead.

Results

• Speed: omitted. There is no measurement of this model on an RTX 5070 Ti anywhere this research reached — not on /check, not in the model's 97 HuggingFace discussions, not in llama.cpp's issue tracker. The one thread that names the card, discussion #36, carries advice but no measurement on it. It does contain figures — the thread opener reports about 20 tk/s for the previous model at 3-bit, which is a different model — but the only number posted there for this model on anything called a 5070 Ti comes from a commenter running an RTX 5070 Ti Mobile with 12 GB on an eco power profile, with most of the model on CPU. That is a different card with a different memory size, and it is not transferable. If you run this configuration, please post your numbers via /contribute so /check/qwen3-8-27b/rtx-5070-ti stops being empty.
• VRAM usage: derived, not measured — 14.590 GiB of 16 GiB at 32K context with the projector loaded, itemised in the budget table above. Live data, when it exists, will be at /check/qwen3-8-27b/rtx-5070-ti.
• Quality notes: this card runs the model at 3 bits — see Picking a quant for why 4-bit cannot coexist with the projector — and vision is the part that suffers first. In discussion #69 a community user (a reply in the thread, not its opener) reports that "the multi modal ability is inferior to 3.6 27b when comes to 4 bit quant" — that is a comparison to the previous generation at 4 bits, one tier above what fits here, and no equivalent report exists at 3 bits either way. Expect the text side to hold up better than the vision side, and treat OCR and fine grounding work as things to verify before relying on.

For the full benchmark data, see /check/qwen3-8-27b/rtx-5070-ti.

Troubleshooting

Prompt processing collapses to tens of tokens per second

You quantised the KV cache to 4 bits. llama.cpp issue #27109, open as of writing, reports that switching K from q8_0 to q4_1 on a qwen35 hybrid model drops prefill from 991–1276 t/s to 34–106 t/s while generation is unaffected — measured by the reporter on an RTX 3090, and attributed in the report to "the CUDA flash-attention / quantized-KV dequant kernel selection for 4-bit KV on this hybrid architecture" rather than to anything specific to that card. Community advice to run "128K context at q4 KV" on 16 GB predates this report; take the shorter context and keep q8_0.

There is a mechanism for this in llama.cpp's build system, which also tells you the fix is not a newer build. GGML_CUDA_FA_ALL_QUANTS is OFF by default, and a default CUDA build compiles exactly four FlashAttention vector instances — f16-f16, q4_0-q4_0, q8_0-q8_0, bf16-bf16. Those are all matched K/V pairs. The reported configuration, K=q4_1 with V=q8_0, is neither matched nor among them, so there is no compiled instance for it. Rebuilding with -DGGML_CUDA_FA_ALL_QUANTS=ON compiles the rest; using matched q8_0/q8_0, as this recipe does, avoids the question entirely and costs nothing.

It OOMs at startup

The ladder, in the order that costs you least:

1. Drop --ctx-size to 16384. At q8_0 that halves the KV line to 0.531 GiB and buys back 0.531 GiB.
2. Confirm --parallel 1 is actually on the command line — the auto default is 4 and costs 0.438 GiB of recurrent state.
3. Drop the projector with --no-mmproj and run text-only: 0.864 GiB.
4. Drop a quant tier to Qwen3.8-27B-UD-Q3_K_XL's smaller siblings — unsloth's Q3_K_S is 11.711 GiB and UD-IQ3_XXS is 11.095 GiB, but note the quality warning above about going below ~10 GB.

Turning on MTP makes it slower, not faster

The model ships a multi-token-prediction head, and llama.cpp does wire it into speculative decoding — --spec-type draft-mtp works against the inline blk.64 block with no separate draft file. On a 16 GB card, leave it off. It costs VRAM: enabling it sets n_rs_seq to your draft length, and llama_memory_recurrent allocates mem_size * (1 + n_rs_seq) rows of recurrent state, so --spec-draft-n-max 4 takes the recurrent line from 156,893,184 bytes to 784,465,920 — 0.584 GiB gone. And the payoff is doubtful on the GGUF path: in discussion #80 a user measuring the same weights blob with speculation off and on found "under Ollama, speculation repays its own overhead in the best case and never more", with unpredictable content running 2.3× slower than with no speculation at all.

The model thinks for minutes on a trivial question

Widely reported — one thread is titled "After waiting 49 minutes and 16 seconds while the model was still thinking, I'm simply giving up...". The model card's own control is the lever: reasoning_effort accepts xhigh (the default), medium and low, and thinking can be turned off entirely per request. On a card whose entire KV cache is the 1.063 GiB budgeted above this is not a comfort setting — a runaway reasoning trace is what will exhaust your context.

Tool calls fail in agent clients

Multiple reports tie this to the packaged chat template rather than to the quant: llama.cpp issue #27139 is titled "Qwen3.8 Codex error resolved by using the Qwen3.6 chat template file", and discussion #68 reports the same class of failure with a community-maintained replacement template. If your client's tool calling misbehaves, try --chat-template-file with a corrected template before blaming the 3-bit weights.

Your tool says the file is Q4_K_S

A metadata defect in unsloth's Dynamic builds, verified here by reading the GGUF header directly: Qwen3.8-27B-UD-Q3_K_XL.gguf carries general.file_type = 14, which is Q4_K_S, while the same repo's plain Qwen3.8-27B-Q3_K_M.gguf correctly carries 12. It is a label, not the tensors — the file is 12.518 GiB, which no real Q4_K_S of this model could be. It affects display only.

Vision crashes

One open vision-crash report exists, llama.cpp issue #27124, and it is on a Vulkan build on an AMD Ryzen AI MAX+ 395 under Windows — a different backend and a different vendor from this recipe's CUDA path. No CUDA equivalent surfaced in the llama.cpp issues opened since the model's release. If you hit one on CUDA, that is new information worth filing.