Qwen3.8-27B on RTX 4070 Ti Super: a vision-capable 27B inside 16 GB with llama.cpp

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — a 27B dense vision-language model — on a single 16 GB RTX 4070 Ti Super, with the vision projector loaded and a 32,768-token context. The install leads with llama-server from llama.cpp, a 3-bit dynamic GGUF from unsloth, and an 8-bit KV cache.

Hardware data: RTX 4070 Ti Super (16 GB VRAM) · derived working set 14.59 GiB of 16 GiB · See benchmark data

⚠️ Nothing on this page was measured on an RTX 4070 Ti Super. /check/qwen3-8-27b/rtx-4070-ti-super returns verdict: unknown with zero benchmarks, and no measurement of this model on this card surfaced anywhere this research reached — see Results for exactly which spaces were searched and how. Every number below is either a byte count read from the artifact you download or arithmetic derived from llama.cpp's own source, shown in full so you can check it. If you run this, please contribute your numbers — that is how the /check page gets real data.

ℹ️ 16 GB is the tight tier for this model, and the fit is not comfortable. Plenty of 4-bit builds are small enough to load on this card — ten of the twenty surveyed below are under 15.6 GiB. None of them leaves room for the vision projector and a usable KV cache at the same time, which is the constraint that actually decides the quant. The arithmetic is under Picking a quant; the 3-bit tier it forces is a genuine quality cost and is discussed there rather than glossed over.

What this card changes

The RTX 4070 Ti Super is the card the SUPER refresh dragged into this tier, and the slowest reader of weights among its 256-bit members. ASUS's TUF Gaming RTX 4070 Ti SUPER tech-spec sheet gives 8,448 CUDA cores, and NVIDIA's spec page lists it as 16 GB GDDR6X on a 256-bit interface — the same bus width and the same capacity as the RTX 5070 Ti this recipe was cloned from, which is why the entire VRAM budget below transfers unchanged.

The refresh is the reason this card is on this page at all. The same NVIDIA page lists the plain RTX 4070 Ti as 12 GB GDDR6X on a 192-bit interface, and the RTX Blackwell architecture whitepaper puts that card at 504 GB/sec in Table 5 (GeForce RTX 5070 Ti vs GeForce RTX 4070 Ti vs GeForce RTX 3070 Ti). The SUPER moved to 16 GB on 256 bits. A 12 GB card cannot run this recipe — the lead quant alone is 12.518 GiB, before the projector — so if your card does not carry the SUPER suffix, the whole budget below is out of reach and the honest answer is a smaller model.

What differs from the 5070 Ti is the memory generation. NVIDIA publishes no peak-bandwidth figure for the SUPER refresh, but the ASUS sheet gives the memory speed as 21 Gbps, which on 256 bits is 672 GB/s; the whitepaper gives the RTX 5070 Ti 896 GB/sec on 28 Gbps GDDR7. That is 25.0% lower — equivalently, the RTX 5070 Ti carries 33.3% more.

That gap is the widest among the 256-bit 16 GB cards, and it is nowhere near the widest in the tier. Superlatives here need the set named inside them, so: among the catalogue's seven NVIDIA 16 GB cards this one is third-fastest, and it is the slowest only of the five 256-bit parts. The two below it are narrow-bus — NVIDIA lists the RTX 4060 Ti and the RTX 5060 Ti on 128-bit buses, half this card's, and ASUS's spec sheets give their memory speeds as 18 Gbps GDDR6 and 28 Gbps GDDR7, which at 16 bytes per clock is 288 GB/s and 448 GB/s. So the tier-wide floor is 288, not 672, and the real spread is 288 against the RTX 5080's 960. This card is the tight end of the wide-bus family, not of the 16 GB tier — a page claiming otherwise would contradict our own RTX 4060 Ti and RTX 5060 Ti pages.

That 672 GB/s is arithmetic rather than a published figure, and the arithmetic is anchored on NVIDIA's own numbers rather than assumed: 256 bits is 32 bytes per memory clock, and the whitepaper's own rows satisfy 32 × data-rate exactly — 22.4 Gbps × 32 = 716.8 GB/sec for the RTX 4080, 28 Gbps × 32 = 896 GB/sec for the RTX 5070 Ti, 30 Gbps × 32 = 960 GB/sec for the RTX 5080, all on the same 256-bit interface. Applying it to 21 Gbps gives 672 GB/s. It is still a derivation; treat it as a size, not a measurement.

Why generation is the phase that cares. Producing one token means reading the whole weight set — 13,441,059,904 bytes of it — out of VRAM, so token generation is bounded by memory bandwidth, not by shader count. Prompt processing and the vision encoder are the other half of the workload and they are compute-bound. Neither half has been measured on this card, or on the RTX 5070 Ti sibling, so this is a statement about which spec governs which phase and not a prediction with a number attached.

And this is why 16 GB cards need separate pages at all. Across the catalogue's seven NVIDIA 16 GB cards — RTX 4060 Ti 16GB, RTX 5060 Ti, RTX 4070 Ti Super, RTX 4080, RTX 4080 Super, RTX 5070 Ti, RTX 5080 — peak memory bandwidth spans 288 GB/s to 960 GB/s, a factor of 3.33, at identical capacity. The floor is the RTX 4060 Ti 16GB: NVIDIA lists it on a 128-bit bus and ASUS gives its memory speed as 18 Gbps, which at 16 bytes per clock is 288 GB/s; the ceiling is the RTX 5080 at 960 GB/sec in the whitepaper's Table 4 (GeForce RTX 5080 vs GeForce RTX 4080 vs GeForce RTX 3080 Specs). Every one of them fits this model at exactly the same quant and context, because capacity is what decides the fit; none of them generates at the same speed, because bandwidth is what decides that. (The tier has two further 16 GB members, the Apple M2 Pro and the Radeon RX 7800 XT; neither is characterised here, so every bandwidth superlative on this page ranges over the seven NVIDIA cards only.)

Compute has no first-party figure here either. The whitepaper's tables cover the RTX 4080, RTX 4070 Ti, RTX 5070 Ti and RTX 5080 but not the SUPER refresh, so no peak-tensor-TFLOPS number for this SKU is quoted and none is invented. The shader count above is what is published. Expect this card to generate more slowly than any other 256-bit 16 GB card — and faster than the two 128-bit ones — while fitting exactly the same model, at exactly the same quant, with exactly the same context.

Ignore the NVFP4 repositories. HuggingFace carries dozens of NVFP4 conversions of this model and none of them is for this card: llama.cpp's CUDA build file attributes FP4 tensor cores to compute capability 12.0 (Blackwell) while annotating this card's 8.9 as "RTX 4000, needs CUDA v11.8". Nothing in the Ada feature set accelerates FP4, so an NVFP4 build buys you nothing here even where one is small enough to load.

Requirements

Why a 27B fits at all: the architecture

Qwen3.8-27B is not a conventional 64-layer dense transformer, and that is the whole reason a 16 GB card is even in the conversation. Its config.json declares 64 layers of which 48 are linear_attention and 16 are full_attention, at indices 3, 7, 11 … 63 — the model card describes the pattern as "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))".

llama.cpp implements this as architecture qwen35 and derives the same split independently. In src/models/qwen35.cpp it marks recurrent layers as is_recr_impl[i] = (i < n_layer()) && ((i + 1) % full_attn_interval != 0), with full_attn_interval read from the GGUF key qwen35.full_attention_interval (which is 4 in every published build). Layers 3, 7, 11 … 63 are therefore the only ones that get a KV cache; src/llama-model.cpp confirms this with a layer filter of il < hparams.n_layer() && !hparams.is_recr(il) when it builds the hybrid memory.

The consequence: only 16 layers hold a KV cache instead of 64, so KV costs a quarter of what a conventional 27B would charge. The other 48 layers hold a fixed-size recurrent state that does not grow with context at all.

Count the layers from 64, not from the GGUF's block_count. Publishers disagree here: ggml-org's files declare block_count = 64 and ship the multi-token-prediction head as a separate mtp-*.gguf, while unsloth, bartowski and lmstudio-community declare 65 with nextn_predict_layers = 1 and the head inline. Both describe the same model. llama.cpp reconciles them in llama-hparams.cpp with n_layer() { return n_layer_all - n_layer_nextn; } — commented in the header as "number of effective layers (excludes nextn layers)" — and the MTP block is excluded from the attention cache by the same il < hparams.n_layer() test. Sizing KV off a raw block_count of 65 would give 16.25 layers and overstate the cache by about 6%.

VRAM budget

Every figure here is either a byte count from the HuggingFace tree API, re-fetched for this page rather than inherited, or arithmetic over values read out of the GGUF header and llama.cpp's source.

On units, because this is where fit calculations go wrong. Everything below is GiB (2^30 bytes) on both sides of the comparison. A "16 GB" graphics card is 16 GiB = 16,384 MiB — GPU memory is quoted in binary units, unlike disk capacity, and the whole stack agrees: ggml-cuda.cu prints device memory as prop.totalGlobalMem / (1024 * 1024) labelled MiB, and nvidia-smi reports the same. Do not convert the card's "16 GB" as if it were a decimal figure — that would understate it by 7% and turn a fit into a miss. Two real deductions do apply and both come out of the headroom line rather than the components: cudaDeviceProp::totalGlobalMem sits slightly under the nominal 16,384 MiB, and the display driver reserves more on top — noticeably more on Windows with a monitor attached than on a headless Linux box. Check yours with nvidia-smi --query-gpu=memory.total,memory.used --format=csv before you size the context.

KV cache, per token. 16 full-attention layers × 4 KV heads × 256 head dim × 2 (K and V) = 32,768 elements per token. The GGUF header confirms the shape: qwen35.attention.head_count_kv = 4, qwen35.attention.key_length = 256, qwen35.attention.value_length = 256. At fp16 that is 65,536 bytes per token; at q8_0 (34 bytes per 32-element block) it is 34,816 bytes per token.

Recurrent state, per sequence. llama_hparams::n_embd_r() returns (ssm_d_conv - 1) * (ssm_d_inner + 2 * ssm_n_group * ssm_d_state) = 3 × (6144 + 2×16×128) = 30,720 elements of convolution state, and n_embd_s() returns ssm_d_state * ssm_d_inner = 128 × 6144 = 786,432 elements of recurrent state, per layer. Both are allocated as GGML_TYPE_F32 — the constructor call in llama-model.cpp passes GGML_TYPE_F32 for both recurrent_type_k and recurrent_type_v. Across 48 layers for one sequence that is 48 × (30,720 + 786,432) × 4 = 156,893,184 bytes, and it is constant regardless of context length.

The whole budget at 32,768 context, --parallel 1, q8_0 KV:

That 1.41 GiB is what the CUDA context, the compute buffers, the vision encoder graph and the display driver's own reservation have to live in. It is a reservation, not a measurement — llama.cpp allocates those from the same pool and their size depends on your batch and image sizes, and none of it was measured on this card. This is the number that decides whether the configuration actually runs, and it is the one number here that is not derived from a byte count; treat 32K as a starting point to verify rather than a guarantee. If your run overflows, the ladder under Troubleshooting walks you down.

One thing works in your favour that the table does not credit: llama.cpp skips the model's multi-token-prediction block unless you actually ask for speculation. qwen35.cpp creates those tensors with int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0;, so the extra blk.64 carried inside the unsloth, bartowski and lmstudio-community files is on disk but not in VRAM by default.

Picking a quant on 16 GB

There is no first-party GGUF. Qwen publishes Qwen3.8-27B and an FP8 sibling as safetensors only; filtering their HF org to "Qwen3.8" returns exactly four repositories and none of them is a -GGUF, while the same org's GGUF-filtered listing returns 54 repositories for other model families. Both listings were re-run for this page. Every GGUF below is a community conversion, so the publisher is part of what you are choosing.

Sizes below are size fields from https://huggingface.co/api/models/<repo>/tree/main?recursive=true, re-fetched for this page rather than copied from a sibling recipe, converted to GiB.

Scope of this survey, so you know what it does and does not cover. HuggingFace lists over 200 GGUF repositories matching "Qwen3.8-27B" — abliterated forks, distills, ROCm/NVFP4 repacks, and one publisher shipping 90+ single-quant split repos. Enumerating that tail is not feasible and most of it is not this model. What follows is every 4-bit build from the five publishers this recipe surveys — ggml-org, unsloth, bartowski, lmstudio-community and AtomicChat: 20 files, enumerated from the tree API rather than sampled. Claims below are bounded to that set.

The quants worth considering on this card. The 3-bit tier, plus the two 4-bit entries a reader is most likely to reach for:

The complete 4-bit set, so the claim below is reproducible. All 20 four-bit files from the five publishers, ascending. Ten are under 15.6 GiB:

Entries 14 and up exceed the card as weights alone — unsloth's Q4_1 at 16.336 GiB is already over, which is worth checking against the table rather than taking on trust.

Why the 4-bit tier is out, stated as arithmetic rather than as a size floor. Ten of the twenty 4-bit builds are under 15.6 GiB, so "4-bit is too big to load" would be false. The binding constraint is what has to sit beside the weights: projector 0.864 + recurrent state 0.146 + KV at 32K q8_0 1.062 = 2.073 GiB of fixed load (2,225,351,360 bytes). Add that to the smallest uniform 4-bit build, bartowski's IQ4_XS at 14.499, and you get 16.571 GiB — over the card before a single byte of runtime buffer. Shrinking the context does not rescue it: at 16K the same build totals 16.04 GiB, still over; at 8K it totals 15.774 GiB, leaving 0.226 GiB for every compute buffer, the CUDA context, the vision graph and the display driver combined. That is the real reason this recipe is 3-bit, and it holds for all twenty. (These totals are summed from unrounded byte counts and rounded once at the end, which is why two of them sit a thousandth below the figures on the sibling 5070 Ti page, where the rounded GiB values were added instead.)

Do not read "Q4_K_M" as one number. Across these publishers, files named Q4_K_M alone span 15.656 to 17.671 GiB — a 2.0 GiB spread on one label — and the spread is not noise. ggml-org's convert.log shows their build was quantised with --pure --tensor-type output.weight=q6_k --tensor-type shexp=q8_0 --tensor-type latent=q8_0 --tensor-type attn_=q8_0 --tensor-type ssm_=q8_0, i.e. large parts of it are actually q8_0. A quant name is a recipe the publisher chose, not a size. Always name the publisher and the file — and note that a range measured on one quant name says nothing about its tier.

On quality at this tier, the only comparative measurement found is vendor-published and self-interested: AtomicChat, whose founder posted it and who sell a competing quant set, ran 20 community files plus 16 of their own through one harness on 4× RTX 5090 against their own BF16 logits, and report unsloth's Q3_K_M at 0.0484 mean KLD against their same-size file at 0.0325, with their 13.8 GB build reaching "92.4% top-1". Treat the ranking as a vendor claim. The useful, publisher-independent part is their statement that "below 10 GB every quant of this model degrades fast" — which is why this recipe does not send you to a 2-bit build to buy context.

Installation

1. Build or fetch llama.cpp with CUDA

There are three different "which build" claims in circulation for this model, and only one of them is a minimum. Keeping them apart matters, because a reader on a build a few tags old should not conclude they are unsupported.

1. Architecture support floor — old, and any current build clears it. Both structural requirements are already present at b8001, the oldest tag checked: src/llama-arch.cpp carries qwen35 and tools/mtmd/clip-impl.h carries the qwen3vl_merger projector type. Both files returned HTTP 200 at that tag with non-empty bodies, re-checked for this page, so this is a real reading and not a 404 misread as absence. b8001 is the floor of the window searched, not the landing build — the true one is older and was not established here. Practically: if your llama.cpp is recent enough to be called recent, it supports this model.
2. A functional floor for a specific capability — none applies to this configuration. This recipe does not depend on any post-release fix that was sourced to a bug report. If you go looking for very long context you may find one; nothing in the path documented here does.
3. The build this page was authored against — b10442. That is provenance for the source line numbers and flag names quoted throughout, not a requirement. Use it if you want byte-identical footing with this text.

The toolkit floor for this card, from llama.cpp's own build file. ggml/src/ggml-cuda/CMakeLists.txt annotates the architecture list it compiles, and the entry for this card reads "RTX 4000, needs CUDA v11.8"; the surrounding logic appends 89-real only when CUDAToolkit_VERSION is at least 11.8. That is a low bar and any toolkit you are likely to have clears it. It is also lower than the Blackwell cards in this family need, which is the one place where an Ada card is less constrained than a 50-series one: 120a-real is appended only from CUDA 12.8. There is no additional CMake-version floor on that path — the CMake versions the same comment block names apply to 120f-virtual, which the project declines to ship.

The one genuine build-time requirement in this area is a compile flag, not a version — and it does not apply here either. GGML_CUDA_FA_ALL_QUANTS defaults to OFF in ggml/CMakeLists.txt, and a default CUDA build compiles only four FlashAttention vector instances: f16-f16, q4_0-q4_0, q8_0-q8_0 and bf16-bf16. That list is produced by an else() branch in ggml/src/ggml-cuda/CMakeLists.txt that names the four files literally and is not conditioned on CMAKE_CUDA_ARCHITECTURES — so it is the same set on sm_89 as on any other target, and this is one piece of the sibling recipe's reasoning that transfers to your card without adjustment. This recipe's matched q8_0/q8_0 cache is in that default set, so no flag and no newer build is needed. Mixed or q4_1/q5_x cache types are not — which is the mechanism behind the prefill collapse in Troubleshooting.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10442   # optional — pins you to the exact tree this page quotes; plain `master` is fine
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Video input is compiled in by default — the mtmd CMakeLists.txt sets MTMD_VIDEO ON with the description "enable video support in mtmd (requires ffmpeg binary in PATH)". If you do not have ffmpeg, images still work; only video is affected.

A prebuilt container works too, and is what the ghcr.io/ggml-org/llama.cpp:server-cuda13 image is for.

2. Download the weights and the projector

pip install -U huggingface_hub
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q3_K_XL.gguf --local-dir ./qwen38
hf download unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf --local-dir ./qwen38

Both files must come down. Without mmproj-F16.gguf the server loads text-only and rejects images with image input is not supported - hint: if this is unexpected, you may need to provide the mmproj.

Running

./build/bin/llama-server \
  --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf \
  --mmproj ./qwen38/mmproj-F16.gguf \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --image-min-tokens 1024 \
  --jinja \
  --no-reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

Four of those flags are doing load-bearing work on a 16 GB card and are not defaults:

• --parallel 1. This one is easy to skip and it is worth 0.438 GiB. The common_params struct initialises n_parallel to 1, but that is not the value llama-server runs with: common/arg.cpp overrides it in its per-example defaults block — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default } — and tools/server/server.cpp then resolves any negative value with "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true". You do not have to type anything to get four slots; you have to type --parallel 1 to avoid them. The startup log in issue #27124, from a command line carrying no -np flag at all, prints n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true' — which is the behaviour, observed rather than inferred. Because the recurrent state is allocated per sequence — llama_memory_hybrid gets recurrent_kv_size = std::max((uint32_t) 1, cparams.n_seq_max) — the default costs 627,572,736 bytes instead of 156,893,184: 0.438 GiB, or about 13,500 more tokens of context. Note also that this state is hardcoded GGML_TYPE_F32 in llama-model.cpp, so --cache-type-k/v does not shrink it — those flags reach only the 16 attention layers.

  The KV cache itself is not multiplied by the slot count, and it is worth knowing why the two behave differently. The auto path also flips kv_unified to true, and llama-context.cpp then sets n_ctx_seq = n_ctx rather than n_ctx / n_seq_max — so -c 32768 allocates 32,768 cells either way. What changes is who owns them: at the default, four slots share one 32,768-token window, so a single long conversation can be evicted by a concurrent request. With --parallel 1 the whole window belongs to one conversation and the recurrent state drops to one copy. On a single-user desktop the flag costs you nothing you wanted.

• --cache-type-k q8_0 --cache-type-v q8_0. Halves KV against fp16. Do not go to 4-bit KV, and do not mix the two types — see Troubleshooting.

• --chat-template-kwargs '{"reasoning_effort":"medium"}' and --no-reasoning-preserve. Thinking is on by default and the model card's default effort is xhigh; the card documents "supported levels are xhigh, medium, and low". Preserved thinking is also on by default and "retains thinking blocks from all historical messages", which on this card means every past reasoning trace keeps occupying your 32K window. Both are context-budget decisions, not quality opinions — raise them if you have context to spare.

• --image-min-tokens 1024. llama.cpp itself warns at load time that "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks" — the string is in tools/mtmd/clip.cpp, emitted whenever image_min_pixels is below that threshold. Budget for it: at 1024 tokens and q8_0 KV, each image costs about 35 MB of cache.

The sampler values are the model card's own thinking-mode recommendation (temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0).

Point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Images go in as image_url content parts; GET /props reports whether the build accepted the projector and whether video is available.

Text-only, if you want the context back

Dropping the projector frees 927,607,488 bytes — about 26,600 more tokens of q8_0 KV. It is a real option on this card and costs you the model's defining capability:

./build/bin/llama-server --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf --no-mmproj \
  --n-gpu-layers 99 --ctx-size 65536 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja

Why not Ollama on this card

Ollama's library page for this model lists twelve tags and the smallest is 18 GB — re-checked for this page — so there is no tag below the card's capacity and any of them will partially offload to system RAM. A user in discussion #80 reports the 27b-q4_K_M weights blob is 16.81 GB, which is the same figure lmstudio-community's Q4_K_M rounds to (16,810,714,336 B = 16.81 GB) — a match to the precision quoted, not a verified identity of the blobs; adding a 0.93 GB projector accounts for the displayed 18 GB, though that last step is inference on my part rather than something Ollama documents. Either way the tag does not fit. Pull the GGUF directly instead.

Results

• Speed: omitted, and the absence is bounded rather than assumed. Three spaces were searched. (1) /check/qwen3-8-27b/rtx-4070-ti-super returns verdict: unknown with zero benchmarks. (2) All 98 discussions on the model card were fetched individually through the JSON API — enumerated, not sampled, and not keyword-searched from a listing page — and 83,110 characters of body text scanned: the RTX 4070 Ti Super is named in none of them. The same scan run as a control does find 5070, 4090 and 3090 in four separate threads, so the zero is a real absence and not a broken search. (3) The llama.cpp issue tracker returns nothing for this card, and the one trap there is worth naming: searching the repo for this card's model number also matches the repository's own open pull request #4070, "Update gpt2 preprocess and add deepseek coder preprocess", because the token matches an issue number. That is the whole space this sentence covers; a measurement posted anywhere else would not appear in it. If you run this configuration, please post your numbers via /contribute so /check/qwen3-8-27b/rtx-4070-ti-super stops being empty.
• VRAM usage: derived, not measured — 14.59 GiB of 16 GiB at 32K context with the projector loaded, itemised in the budget table above. Live data, when it exists, will be at /check/qwen3-8-27b/rtx-4070-ti-super.
• Quality notes: this card runs the model at 3 bits — see Picking a quant for why 4-bit cannot coexist with the projector — and vision is the part that suffers first. In discussion #69 a community user (a reply in the thread, not its opener) reports that "the multi modal ability is inferior to 3.6 27b when comes to 4 bit quant" — that is a comparison to the previous generation at 4 bits, one tier above what fits here, and no equivalent report exists at 3 bits either way. Expect the text side to hold up better than the vision side, and treat OCR and fine grounding work as things to verify before relying on.

For the full benchmark data, see /check/qwen3-8-27b/rtx-4070-ti-super.

Troubleshooting

Prompt processing collapses to tens of tokens per second

You quantised the KV cache to 4 bits, or you mixed two cache types. llama.cpp issue #27109, open as of writing, reports that switching K from q8_0 to q4_1 on a qwen35 hybrid model drops prefill from 991–1276 t/s to 34–106 t/s while generation is unaffected — measured by the reporter on an RTX 3090, and attributed in the report to "the CUDA flash-attention / quantized-KV dequant kernel selection for 4-bit KV on this hybrid architecture" rather than to anything specific to that card. Community advice to run "128K context at q4 KV" on 16 GB predates this report; take the shorter context and keep q8_0.

Read the thread, not the title. The issue is titled as a q4_1/q4_0 problem, and the reporter's own later root-cause comment narrows it: on a default build, K=q8_0, V=q8_0 and K=q4_0, V=q4_0 are both fine and stay on the GPU. What fails is an unsupported type or a mismatched pair. Both guards are visible in ggml/src/ggml-cuda/fattn.cu: ggml_cuda_fattn_kv_type_supported() returns false for Q4_1, Q5_0 and Q5_1 unless GGML_CUDA_FA_ALL_QUANTS is defined, and a few lines above it a second #ifndef GGML_CUDA_FA_ALL_QUANTS block rejects any configuration where K->type != V->type. Either rejection returns BEST_FATTN_KERNEL_NONE, the scheduler moves the whole attention op to the CPU backend, and nothing is logged — you just get a thirty-fold prefill regression. Neither guard is conditioned on compute capability, so this behaves the same on sm_89 as on the RTX 3090 the report was filed from. Rebuilding with -DGGML_CUDA_FA_ALL_QUANTS=ON compiles the rest; using matched q8_0/q8_0, as this recipe does, avoids the question entirely and costs nothing.

It OOMs at startup

The ladder, in the order that costs you least:

1. Drop --ctx-size to 16384. At q8_0 that halves the KV line to 0.531 GiB and buys back 0.531 GiB.
2. Confirm --parallel 1 is actually on the command line — the auto default is 4 and costs 0.438 GiB of recurrent state.
3. Drop the projector with --no-mmproj and run text-only: 0.864 GiB.
4. Drop a quant tier to Qwen3.8-27B-UD-Q3_K_XL's smaller siblings — unsloth's Q3_K_S is 11.711 GiB and UD-IQ3_XXS is 11.095 GiB, but note the quality warning above about going below ~10 GB.

Turning on MTP makes it slower, not faster

The model ships a multi-token-prediction head, and llama.cpp does wire it into speculative decoding — --spec-type draft-mtp works against the inline blk.64 block with no separate draft file. On a 16 GB card, leave it off. It costs VRAM: enabling it sets n_rs_seq to your draft length, and llama_memory_recurrent allocates mem_size * (1 + n_rs_seq) rows of recurrent state, so --spec-draft-n-max 4 takes the recurrent line from 156,893,184 bytes to 784,465,920 — 0.584 GiB gone. And the payoff is doubtful on the GGUF path: in discussion #80 a user measuring the same weights blob with speculation off and on found "under Ollama, speculation repays its own overhead in the best case and never more", with unpredictable content running 2.3× slower than with no speculation at all. It is also implicated in two open CUDA stability reports on this exact model — #27122 and #27105 — both of which involve MTP, and neither of which is on this card.

The model thinks for minutes on a trivial question

Widely reported — one thread is titled "After waiting 49 minutes and 16 seconds while the model was still thinking, I'm simply giving up...". The model card's own control is the lever: reasoning_effort accepts xhigh (the default), medium and low, and thinking can be turned off entirely per request. On a card whose entire KV cache is the 1.062 GiB budgeted above this is not a comfort setting — a runaway reasoning trace is what will exhaust your context.

Tool calls fail in agent clients

Multiple reports tie this to the packaged chat template rather than to the quant: llama.cpp issue #27139 is titled "Qwen3.8 Codex error resolved by using the Qwen3.6 chat template file", and discussion #68 reports the same class of failure with a community-maintained replacement template. If your client's tool calling misbehaves, try --chat-template-file with a corrected template before blaming the 3-bit weights.

Your tool says the file is Q4_K_S

A metadata defect in unsloth's Dynamic builds, verified by reading the GGUF header directly: Qwen3.8-27B-UD-Q3_K_XL.gguf carries general.file_type = 14, which is Q4_K_S, while the same repo's plain Qwen3.8-27B-Q3_K_M.gguf correctly carries 12. It is a label, not the tensors — the file is 12.518 GiB, which no real Q4_K_S of this model could be. It affects display only.

Vision crashes

One open vision-crash report exists, llama.cpp issue #27124, and it is on a Vulkan build on an AMD Ryzen AI MAX+ 395 under Windows — a different backend and a different vendor from this recipe's CUDA path. No CUDA equivalent surfaced in the llama.cpp issues opened since the model's release. If you hit one on CUDA, that is new information worth filing via /contribute as well as upstream.