Qwen3.8-27B on RTX 5060 Ti: a vision-capable 27B in 16 GB on a 128-bit bus

What You'll Build

A local OpenAI-compatible server running Qwen3.8-27B — a 27B dense vision-language model — on a single 16 GB RTX 5060 Ti, with the vision projector loaded and a 32,768-token context. The install leads with llama-server from llama.cpp, a 3-bit dynamic GGUF from unsloth, and an 8-bit KV cache.

Hardware data: RTX 5060 Ti (16 GB VRAM, 128-bit GDDR7, 448 GB/s) · derived working set 14.590 GiB of 16 GiB · See benchmark data

⚠️ Nothing on this page was measured on an RTX 5060 Ti. /check/qwen3-8-27b/rtx-5060-ti returns verdict: unknown with zero benchmarks, and no measurement of this model on this card surfaced anywhere this research reached. Every number below is either a byte count read from the artifact you download, arithmetic derived from llama.cpp's own source, or a clearly-labelled figure for a different model on this same card. If you run this, please contribute your numbers — that is how the /check page gets real data.

ℹ️ The capacity is the same as every other 16 GB card in this tier; the bus is not. The model fits — the arithmetic under VRAM budget is identical on any 16 GB card. What differs is how fast those bytes can be read back, and the RTX 5060 Ti carries its 16 GB of GDDR7 on a 128-bit interface at 448 GB/s. Among the NVIDIA cards this catalogue holds at 16 GB that is under half of the fastest, the RTX 5080 at 960 GB/s. A 27B at 3 bits is bandwidth-bound the moment it starts generating, so that gap is the number that decides what this configuration feels like. What the 128-bit bus costs you sets the expectation with arithmetic rather than adjectives.

Requirements

Why a 27B fits at all: the architecture

Qwen3.8-27B is not a conventional 64-layer dense transformer, and that is the whole reason a 16 GB card is even in the conversation. Its config.json declares 64 layers of which 48 are linear_attention and 16 are full_attention, at indices 3, 7, 11 … 63 — the model card describes the pattern as "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))".

llama.cpp implements this as architecture qwen35 and derives the same split independently. In src/models/qwen35.cpp it marks recurrent layers as is_recr_impl[i] = (i < hparams.n_layer()) && ((i + 1) % full_attn_interval != 0), with full_attn_interval defaulting to 4 and read from the GGUF key qwen35.full_attention_interval. Layers 3, 7, 11 … 63 are therefore the only ones that get a KV cache; src/llama-model.cpp confirms this with a layer filter of il < hparams.n_layer() && !hparams.is_recr(il) when it builds the hybrid memory.

The consequence: only 16 layers hold a KV cache instead of 64, so KV costs a quarter of what a conventional 27B would charge. The other 48 layers hold a fixed-size recurrent state that does not grow with context at all.

Count the layers from 64, not from the GGUF's block_count. Publishers disagree here: ggml-org's files declare block_count = 64 and ship the multi-token-prediction head as a separate mtp-*.gguf, while unsloth, bartowski and lmstudio-community declare 65 with nextn_predict_layers = 1 and the head inline. Both describe the same model. llama.cpp reconciles them in src/llama-hparams.cpp with n_layer() { return n_layer_all - n_layer_nextn; } — commented in the header as "number of effective layers (excludes nextn layers)" — and the MTP block is excluded from the attention cache by the same il < hparams.n_layer() test. Sizing KV off a raw block_count of 65 would give 16.25 layers and overstate the cache by about 6%.

VRAM budget

Every figure here is either a byte count from the HuggingFace tree API, re-read for this recipe rather than inherited, or arithmetic over values in config.json and llama.cpp's source.

On units, because this is where fit calculations go wrong. Everything below is GiB (2^30 bytes) on both sides of the comparison. A "16 GB" graphics card is 16 GiB = 16,384 MiB — GPU memory is quoted in binary units, unlike disk capacity, and the whole stack agrees: ggml-cuda.cu prints device memory as prop.totalGlobalMem / (1024 * 1024) labelled MiB, and nvidia-smi reports the same. Do not convert the card's "16 GB" as if it were a decimal figure — that would understate it by 7% and turn a fit into a miss. Two real deductions do apply and both come out of the headroom line rather than the components: cudaDeviceProp::totalGlobalMem sits slightly under the nominal 16,384 MiB, and the display driver reserves more on top — noticeably more on Windows with a monitor attached than on a headless Linux box. Check yours with nvidia-smi --query-gpu=memory.total,memory.used --format=csv before you size the context.

KV cache, per token. 16 full-attention layers × 4 KV heads × 256 head dim × 2 (K and V) = 32,768 elements per token. config.json gives num_key_value_heads = 4 and head_dim = 256, and the GGUF header carries the same shape as qwen35.attention.head_count_kv, key_length and value_length. At fp16 that is 65,536 bytes per token; at q8_0 (34 bytes per 32-element block) it is 34,816 bytes per token.

Recurrent state, per sequence. llama_hparams::n_embd_r() returns (ssm_d_conv - 1) * (ssm_d_inner + 2 * ssm_n_group * ssm_d_state) = 3 × (6144 + 2×16×128) = 30,720 elements of convolution state, and n_embd_s() returns ssm_d_state * ssm_d_inner = 128 × 6144 = 786,432 elements of recurrent state, per layer. Those inputs come straight out of config.json (linear_num_value_heads 48 × linear_value_head_dim 128 = 6144; linear_num_key_heads 16; linear_key_head_dim 128; linear_conv_kernel_dim 4). Both are allocated as GGML_TYPE_F32. Across 48 layers for one sequence that is 48 × (30,720 + 786,432) × 4 = 156,893,184 bytes, and it is constant regardless of context length.

The whole budget at 32,768 context, --parallel 1, q8_0 KV:

That 1.410 GiB is what the CUDA context, the compute buffers, the vision encoder graph and the display driver's own reservation have to live in. It is a reservation, not a measurement — llama.cpp allocates those from the same pool and their size depends on your batch and image sizes, and none of it was measured on this card. Treat 32K as a starting point to verify rather than a guarantee. If your run overflows, the ladder under Troubleshooting walks you down.

One thing works in your favour that the table does not credit: llama.cpp skips the model's multi-token-prediction block unless you actually ask for speculation. qwen35.cpp creates those tensors with int mtp_flags = !ml.load_mtp ? TENSOR_SKIP : 0;, so the extra blk.64 carried inside the unsloth, bartowski and lmstudio-community files is on disk but not in VRAM by default.

What the 128-bit bus costs you

This is the section that makes an RTX 5060 Ti page different from a page about any other 16 GB card, so it gets numbers rather than adjectives.

The specs. NVIDIA's own product page lists the RTX 5060 Ti as 16 GB GDDR7 on a 128-bit interface; it does not publish a bandwidth figure, and Hardware Corner's RTX 5060 Ti 16GB page states 448 GB/s against that same 128-bit bus. The comparison that matters is inside the tier, because capacity is held constant. Among the NVIDIA cards this catalogue holds at 16 GB — RTX 4060 Ti 16GB, RTX 4070 Ti Super, RTX 4080, RTX 4080 Super, RTX 5060 Ti, RTX 5070 Ti and RTX 5080 — the range runs from 288 GB/s on the RTX 4060 Ti 16GB to 960 GB/s on the RTX 5080, a 3.33× spread at identical capacity. (The tier also contains an Apple M2 Pro and a Radeon RX 7800 XT, which this page does not characterise.) This card sits low in that range at 448 GB/s: the RTX 5070 Ti runs the same GDDR7 on a 256-bit bus for exactly twice this card's figure, and the RTX 5080 goes further still. Bus width, not memory generation, is what separates them — this card has the newer memory and half the lanes.

Why that matters more here than for most models. Generating a token requires reading the whole weight set — 13,441,059,904 bytes at this quant — plus whatever KV has accumulated. Prompt processing does not: it reads the weights once and then does arithmetic over many tokens at a time, so it is limited by compute rather than by memory. A narrow bus therefore taxes one half of the workload and mostly spares the other.

That split is measurable across the tier. The figures below are Hardware Corner's published tables for Qwen3 14B at Q4_K at a 32K context — a different, smaller model, quoted as a hardware reference point and not as a figure for Qwen3.8-27B:

At that 32K column this card decodes at 0.50× the RTX 5080 on 0.47× its bandwidth, and at 0.57× the RTX 5070 Ti on exactly half of its — slightly ahead of the raw bandwidth ratio in both cases, which is the usual sign that the larger card is not fully fed either. Looking down instead of up, the same rows put this card 1.45× ahead of the RTX 4060 Ti on decode but only 1.15× ahead on prefill — those two share a bus width and differ almost only in memory generation, so the gap that opens is the bandwidth-bound half.

One row in that table is worth not glossing over: at 32K the RTX 5080's prefill (1,326.1 t/s) comes in below the RTX 5070 Ti's (1,658.2), even though it has more bandwidth — and the two swap back at 4K and 16K in Hardware Corner's fuller tables. Decode, by contrast, rises monotonically with bandwidth down the whole column. That is the same point from the other direction: generation tracks memory bandwidth closely enough to rank cards by it, and prompt processing does not, so a prefill number is not something you should extrapolate from a bandwidth figure.

Our own catalogue carries the 4K figures for this card rather than the 32K ones above — 41.1 tok/s decode and 1,743.0 t/s prefill, at /check/qwen3-14b/rtx-5060-ti. The 32K column is Hardware Corner's alone.

The ceiling for this model, as arithmetic. Divide the bandwidth by the bytes that must be read per generated token. This is reasoning, not measurement — it is an upper bound that no real runtime reaches, because it credits the card with perfect memory efficiency and zero time spent on anything else:

• with an empty context, 448 GB/s ÷ 13.441 GB of weights ≈ 33 tokens/s
• with a full 32K window, 448 GB/s ÷ 14.739 GB (weights + KV + recurrent state) ≈ 30 tokens/s

Expect to land materially under both. The honest summary is that this configuration reads and looks at images comfortably and generates at conversational-but-not-brisk speed, and that a long chain of thinking tokens is the thing that will test your patience. A community user on an unnamed 16 GB card put a harsher version of the same conclusion in discussion #54 — Qwen3.8-27B is "not really usable with that speed on a 16gb vram". Their post names no card, no quant and no runtime, and its numbers are a comparison against a different model, so it is sentiment rather than a measurement; it is quoted here because it is the closest thing to first-hand 16 GB experience that this research found, and because its direction agrees with the arithmetic.

If you run this configuration, the one thing that would improve this page is your llama-bench output via /contribute.

Picking a quant on 16 GB

There is no first-party GGUF. Qwen publishes Qwen3.8-27B and an FP8 sibling as safetensors only; filtering their org's model list to "Qwen3.8" returns exactly four repositories and none of them is a -GGUF repo, while the same org's GGUF-filtered listing returns 54 repos for other model families. Every GGUF below is a community conversion, so the publisher is part of what you are choosing.

Sizes below are size fields from https://huggingface.co/api/models/<repo>/tree/main?recursive=true, converted to GiB, re-read for this recipe.

Scope of this survey, so you know what it does and does not cover. HuggingFace lists over 200 GGUF repositories matching "Qwen3.8-27B" — abliterated forks, distills, ROCm/NVFP4 repacks, and one publisher shipping 90+ single-quant split repos. Enumerating that tail is not feasible and most of it is not this model. What follows is every 4-bit build from the five publishers this recipe surveys — ggml-org, unsloth, bartowski, lmstudio-community and AtomicChat: 20 files, enumerated from the tree API rather than sampled. Claims below are bounded to that set.

The quants worth considering on this card:

The complete 4-bit set, so the claim below is reproducible. All 20 four-bit files from the five publishers, ascending. Ten are under 15.6 GiB:

Entries 14 and up exceed the card as weights alone.

Why the 4-bit tier is out, stated as arithmetic rather than as a size floor. Ten of the twenty 4-bit builds are under 15.6 GiB, so "4-bit is too big to load" would be false. The binding constraint is what has to sit beside the weights: projector 0.864 + recurrent state 0.146 + KV at 32K q8_0 1.063 = 2.073 GiB of fixed load. Add that to the smallest uniform 4-bit build, bartowski's IQ4_XS at 14.499, and you get 16.571 GiB — over the card before a single byte of runtime buffer. Shrinking the context does not rescue it: at 16K the same build totals 16.040 GiB, still over; at 8K it totals 15.775 GiB, leaving 0.225 GiB for every compute buffer, the CUDA context, the vision graph and the display driver combined. That is the real reason this recipe is 3-bit, and it holds for all twenty.

Your card has FP4 tensor cores. It does not help here.

This is the one quant question an RTX 5060 Ti owner has that an Ada owner does not, so it is worth answering rather than leaving implied. Blackwell is the first consumer generation with FP4 tensor cores — llama.cpp's own CUDA build file annotates compute capability 120 as "Blackwell, needs CUDA v12.8, FP4 tensor cores" — and this card has them.

Start with what the five publishers surveyed above ship, because that is where a GGUF reader looks first: none of them publishes an NVFP4 GGUF. Across the four that carry the bulk of the files — ggml-org 12 files, unsloth 29, bartowski 35, lmstudio-community 6, 82 in all — not one .gguf is NVFP4. So the recommendation above is not passing over a Blackwell-native option from a publisher you already trust.

Two NVFP4 things do exist, and neither becomes your path:

• unsloth's Qwen3.8-27B-NVFP4 is a real build from an established publisher, but it is not a GGUF — its tree is two .safetensors and a handful of JSON, tagged compressed-tensors, i.e. a vLLM-class artifact that llama-server does not load. And it would not fit anyway: model.safetensors alone is 22,568,192,096 bytes. The quantisation is selective rather than uniform, which is why a "4-bit" 27B lands at 22.6 GB.
• llama.cpp does carry a GGML_TYPE_NVFP4 type, and community NVFP4 GGUFs of this model exist outside those five publishers. Thirteen were enumerated for this recipe across four HuggingFace queries, and the smallest is QQZ2026/Qwen3.8-27B-NVFP4-Q5K-no-MTP-GGUF at 15,466,938,208 B — 14.405 GiB, the same neighbourhood as the uniform 4-bit builds above and the same verdict: 14.405 + 2.073 GiB of fixed load = 16.477 GiB, over the card before any runtime buffer. That is a bounded check, not a recommendation.

The short version: the FP4 hardware on this card is real, the format is supported by the runtime, and the thing that stops you using it here is capacity — plus the fact that nobody you would want to download from has published one in the format you need.

Do not read "Q4_K_M" as one number. Across these publishers, files named Q4_K_M alone span 15.656 to 17.671 GiB — a 2.0 GiB spread on one label — and the spread is not noise. ggml-org's convert.log shows their build was quantised with --pure --tensor-type output.weight=q6_k --tensor-type shexp=q8_0 --tensor-type latent=q8_0 --tensor-type attn_=q8_0 --tensor-type ssm_=q8_0, i.e. large parts of it are actually q8_0. A quant name is a recipe the publisher chose, not a size. Always name the publisher and the file.

On quality at this tier, the only comparative measurement found is vendor-published and self-interested: AtomicChat, whose founder posted it and who sell a competing quant set, ran 20 community files plus 16 of their own through one harness on 4× RTX 5090 against their own BF16 logits, and report unsloth's Q3_K_M at 0.0484 mean KLD against their same-size file at 0.0325. Treat the ranking as a vendor claim. The useful, publisher-independent part is their statement that "below 10 GB every quant of this model degrades fast" — which is why this recipe does not send you to a 2-bit build to buy speed. Dropping a tier would buy you a few percent of decode rate for a quality cost the same source calls steep.

Installation

1. Build or fetch llama.cpp with CUDA

There are three different "which build" claims in circulation for this model, and only one of them is a minimum. Keeping them apart matters, because a reader on a build a few tags old should not conclude they are unsupported.

1. Architecture support floor — old, and any current build clears it. Both structural requirements are already present at b8001, the oldest tag checked: src/llama-arch.cpp carries qwen35 and tools/mtmd/clip-impl.h carries the qwen3vl_merger projector type. Both files returned HTTP 200 at that tag, so this is a real reading and not a 404 misread as absence. b8001 is the floor of the window searched, not the landing build — the true one is older and was not established here.
2. A functional floor for a specific capability — none applies to this configuration. This recipe does not depend on any post-release fix that was sourced to a bug report.
3. The build this page was authored against — b10442. That is provenance for the source line numbers and flag names quoted throughout, not a requirement.

What is card-specific here is the CUDA toolkit, and Blackwell asks for a newer one than Ada does. llama.cpp's ggml/src/ggml-cuda/CMakeLists.txt annotates its architecture list directly: compute capability 120 is "Blackwell, needs CUDA v12.8, FP4 tensor cores", and the file only appends 120a-real to CMAKE_CUDA_ARCHITECTURES when the detected toolkit is 12.8 or newer. If you build against an older toolkit you will not get native sm_120 code, and that is a property of the toolkit rather than of the build mode: a GGML_NATIVE build shortcuts to the detected architecture once the toolkit is ≥ 11.6 and CMake ≥ 3.24, but it still cannot emit sm_120 from a toolkit that does not know sm_120. Either way the real floor for this card is CUDA 12.8.

The one genuine build-time requirement in this area is a compile flag, not a version — and it does not apply here either. GGML_CUDA_FA_ALL_QUANTS defaults to OFF in ggml/CMakeLists.txt, and a default CUDA build compiles only four FlashAttention vector instances: f16-f16, q4_0-q4_0, q8_0-q8_0 and bf16-bf16. This recipe's matched q8_0/q8_0 cache is in that default set, so no flag and no newer build is needed. Mixed or q4_1/q5_x cache types are not — which is the mechanism behind the prefill collapse in Troubleshooting.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b10442   # optional — pins you to the exact tree this page quotes; plain `master` is fine
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Video input is compiled in by default — the mtmd CMakeLists.txt sets MTMD_VIDEO ON with the description "enable video support in mtmd (requires ffmpeg binary in PATH)". If you do not have ffmpeg, images still work; only video is affected.

A prebuilt container works too, and is what the ghcr.io/ggml-org/llama.cpp:server-cuda13 image is for. Note it is built without GGML_CUDA_FA_ALL_QUANTS, like any default build.

2. Download the weights and the projector

pip install -U huggingface_hub
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-Q3_K_XL.gguf --local-dir ./qwen38
hf download unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf --local-dir ./qwen38

Both files must come down. Without mmproj-F16.gguf the server loads text-only and rejects images with image input is not supported - hint: if this is unexpected, you may need to provide the mmproj.

Running

./build/bin/llama-server \
  --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf \
  --mmproj ./qwen38/mmproj-F16.gguf \
  --alias qwen3.8-27b \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --flash-attn on \
  --batch-size 2048 --ubatch-size 512 \
  --image-min-tokens 1024 \
  --jinja \
  --no-reasoning-preserve \
  --chat-template-kwargs '{"reasoning_effort":"medium"}' \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
  --presence-penalty 0.0 --repeat-penalty 1.0

Four of those flags are doing load-bearing work on a 16 GB card and are not defaults:

• --parallel 1. This one is easy to skip and it is worth 0.438 GiB. The common_params struct initialises n_parallel to 1, but that is not the value llama-server runs with: common/arg.cpp overrides it in its per-example defaults block — else if (ex == LLAMA_EXAMPLE_SERVER) { params.n_parallel = -1; // auto by default } — and tools/server/server.cpp then resolves any negative value, logging "n_parallel is set to auto, using n_parallel = 4 and kv_unified = true". You do not have to type anything to get four slots; you have to type --parallel 1 to avoid them. The startup log in issue #27124, from a command line carrying no -np flag at all, prints n_slots = 4, n_ctx_slot = 65536, kv_unified = 'true' — which is the behaviour, observed rather than inferred. Because the recurrent state is allocated per sequence — llama_memory_hybrid gets recurrent_kv_size = std::max((uint32_t) 1, cparams.n_seq_max) — the default costs 627,572,736 bytes instead of 156,893,184: 0.438 GiB, or about 13,500 more tokens of context. Note also that this state is hardcoded GGML_TYPE_F32, so --cache-type-k/v does not shrink it — those flags reach only the 16 attention layers.

  The KV cache itself is not multiplied by the slot count, and it is worth knowing why the two behave differently. The auto path also flips kv_unified to true, and src/llama-context.cpp then sets n_ctx_seq = n_ctx rather than n_ctx / n_seq_max — so -c 32768 allocates 32,768 cells either way. What changes is who owns them: at the default, four slots share one 32,768-token window, so a single long conversation can be evicted by a concurrent request. With --parallel 1 the whole window belongs to one conversation and the recurrent state drops to one copy. On a single-user desktop the flag costs you nothing you wanted.

• --cache-type-k q8_0 --cache-type-v q8_0. Halves KV against fp16. Do not go to 4-bit KV, and do not mix the two types — see Troubleshooting.

• --chat-template-kwargs '{"reasoning_effort":"medium"}' and --no-reasoning-preserve. Thinking is on by default and the model card's default effort is xhigh; the card documents "supported levels are xhigh, medium, and low". Preserved thinking is also on by default and "retains thinking blocks from all historical messages", which on this card means every past reasoning trace keeps occupying your 32K window. On a card in this bandwidth range, reasoning effort is also the biggest single lever on how long you wait — every thinking token is generated at the same rate as a visible one.

• --image-min-tokens 1024. llama.cpp itself warns at load time that "Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks" — the string is in tools/mtmd/clip.cpp, emitted whenever image_min_pixels is below that threshold. Budget for it: at 1024 tokens and q8_0 KV, each image costs about 35 MB of cache. Image ingestion is prompt processing, which is the half of the workload a narrow bus penalises least.

The sampler values are the model card's own thinking-mode recommendation (temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0).

Point any OpenAI-compatible client at http://127.0.0.1:8080/v1. Images go in as image_url content parts; GET /props reports whether the build accepted the projector and whether video is available.

Text-only, if you want the context back

Dropping the projector frees 927,607,488 bytes — about 26,600 more tokens of q8_0 KV. It is a real option on this card and costs you the model's defining capability:

./build/bin/llama-server --model ./qwen38/Qwen3.8-27B-UD-Q3_K_XL.gguf --no-mmproj \
  --n-gpu-layers 99 --ctx-size 65536 --parallel 1 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --jinja

Note this buys context, not speed — the weights are unchanged, so the per-token read that sets the decode rate is unchanged too.

Why not Ollama on this card

Ollama's library page for this model lists twelve tags and the smallest is 18 GB — there is no tag below the card's capacity, so any of them will partially offload to system RAM. On a card whose on-GPU bandwidth is already the constraint, pushing layers across PCIe to DDR memory is the worst available trade. Its 27b-nvfp4 tag does not rescue this either: it is listed at 18 GB, tagged MLX, and is a byte-identical manifest to 27b-mlx. Pull the GGUF directly instead.

Results

• Speed: omitted as a measurement, because none exists. There is no benchmark for this model on an RTX 5060 Ti in /check, none in the model card's 100 HuggingFace discussions (all of which were enumerated, and none of which mentions a 5060 Ti at all), and none in llama.cpp's issue tracker. What this page offers instead is set out in What the 128-bit bus costs you: a bandwidth ceiling of roughly 30–33 tokens/s that is arithmetic rather than a prediction, and this card's measured position on a different model — 25.9 tok/s on Qwen3 14B at Q4_K and 32K context, against 45.5 for an RTX 5070 Ti on the same row. Post real numbers via /contribute and /check/qwen3-8-27b/rtx-5060-ti stops being empty.
• VRAM usage: derived, not measured — 14.590 GiB of 16 GiB at 32K context with the projector loaded, itemised in the budget table above. Live data, when it exists, will be at /check/qwen3-8-27b/rtx-5060-ti.
• Quality notes: this card runs the model at 3 bits — see Picking a quant for why 4-bit cannot coexist with the projector — and vision is the part that suffers first. In discussion #69 a community user (a reply in the thread, not its opener) reports that "the multi modal ability is inferior to 3.6 27b when comes to 4 bit quant" — that is a comparison to the previous generation at 4 bits, one tier above what fits here, and no equivalent report exists at 3 bits either way. Expect the text side to hold up better than the vision side, and treat OCR and fine grounding work as things to verify before relying on.

For the full benchmark data, see /check/qwen3-8-27b/rtx-5060-ti.

Troubleshooting

Prompt processing collapses to tens of tokens per second

You quantised the KV cache to 4 bits, or mixed two cache types. llama.cpp issue #27109, open as of writing, reports prefill dropping from 991–1276 t/s to 34–106 t/s on a qwen35 hybrid model — measured by the reporter on an RTX 3090, and attributed by them to "the CUDA flash-attention / quantized-KV dequant kernel selection for 4-bit KV on this hybrid architecture".

The issue's own title blames q4_1/q4_0, and the reporter's later root-cause comment narrows that considerably — read it rather than the title. GGML_CUDA_FA_ALL_QUANTS is OFF by default, so a default CUDA build compiles only matched K/V FlashAttention instances, and when no instance exists the scheduler silently moves the whole attention op to the CPU backend. By that analysis K=q8_0, V=q8_0 and K=q4_0, V=q4_0 are both "supported, stays on GPU", while K=q4_1, V=q8_0 and K=q4_0, V=q8_0 are not — the mismatch is as fatal as the type. This recipe's matched q8_0/q8_0 is in the supported set. Also from that comment: "Generation speed unchanged" — the failure is prefill-only, so if your generation rate is what disappoints you, this is not your bug and the bandwidth section above is the honest explanation. Rebuilding with -DGGML_CUDA_FA_ALL_QUANTS=ON compiles the rest if you want the mixed types.

It OOMs at startup

The ladder, in the order that costs you least:

1. Drop --ctx-size to 16384. At q8_0 that halves the KV line to 0.531 GiB and buys back 0.531 GiB.
2. Confirm --parallel 1 is actually on the command line — the auto default is 4 and costs 0.438 GiB of recurrent state.
3. Drop the projector with --no-mmproj and run text-only: 0.864 GiB.
4. Drop a quant tier — unsloth's Q3_K_S is 11.711 GiB and UD-IQ3_XXS is 11.095 GiB, but note the quality warning above about going below ~10 GB.

A Blackwell card can be misreported as an iGPU — and what that does and does not affect

llama.cpp issue #26901 reports that Blackwell discrete GPUs set cudaDeviceProp::integrated, so ggml_backend_cuda_device_get_type() classifies them as GGML_BACKEND_DEVICE_TYPE_IGPU. It is worth knowing about on this card and it is worth knowing precisely, because most of it cannot reach a single-card setup:

• The headline symptom needs two GPUs. The report is that a second Blackwell card is silently dropped and --tensor-split stops working. That is not a failure mode a one-card configuration has.
• The second symptom can reach you, on Linux only. The same misclassification sends ggml_backend_dev_memory() down a UMA path that reads free system RAM instead of free VRAM, and that path is inside #if defined(__linux__). llama.cpp's automatic parameter fitting reads exactly that figure — but only to adjust arguments you did not supply: -fit is documented as "whether to adjust unset arguments to fit in device memory", and common/fit.cpp bails out with "n_gpu_layers already set by user to" the moment you pin it yourself. The command line above pins both --n-gpu-layers and --ctx-size, so nothing in this recipe depends on the reported free-VRAM number. If you prefer to let the fitter decide, know that it may be reading your system RAM.
• Bounds, so you can weigh it. The report names an RTX 5080 and an RTX 5070 Ti, not this card; it is community-filed with no maintainer response as of writing; and the arithmetic in this recipe never consults reported free memory. Treat it as a reason to keep your flags explicit rather than as a defect you should expect to hit.

Turning on MTP looks like the right fix for a bandwidth-bound card, and probably still isn't

The reasoning is genuinely tempting on a 128-bit card: speculative decoding amortises one weight read across several accepted tokens, which is exactly the bottleneck here. The model ships a multi-token-prediction head and llama.cpp wires it into speculation — --spec-type draft-mtp works against the inline blk.64 block with no separate draft file. Two things argue against it anyway. It costs VRAM you do not have: enabling it sets n_rs_seq to your draft length, and llama_memory_recurrent allocates mem_size * (1 + n_rs_seq) rows of recurrent state, so --spec-draft-n-max 4 takes the recurrent line from 156,893,184 bytes to 784,465,920 — 0.584 GiB out of 1.410 GiB of headroom. And the only reported measurement is unfavourable: in discussion #80 a user measuring the same weights blob with speculation off and on found "under Ollama, speculation repays its own overhead in the best case and never more", with unpredictable content running 2.3× slower than with no speculation at all. That measurement is on a different runtime and says nothing directly about llama-server, which is precisely why this is worth trying if you have the patience — and worth reporting either way, because on this card it is the most valuable open question on the page.

The model thinks for minutes on a trivial question

Widely reported — one thread is titled "After waiting 49 minutes and 16 seconds while the model was still thinking, I'm simply giving up...". The model card's own control is the lever: reasoning_effort accepts xhigh (the default), medium and low, and thinking can be turned off entirely per request. On a card generating in this range it is not a comfort setting — a runaway reasoning trace is both your context budget and your afternoon.

Tool calls fail in agent clients

Multiple reports tie this to the packaged chat template rather than to the quant: llama.cpp issue #27139 is titled "Qwen3.8 Codex error resolved by using the Qwen3.6 chat template file.", and discussion #68 reports the same class of failure with a community-maintained replacement template. If your client's tool calling misbehaves, try --chat-template-file with a corrected template before blaming the 3-bit weights.

Your tool says the file is Q4_K_S

A metadata defect in unsloth's Dynamic builds: Qwen3.8-27B-UD-Q3_K_XL.gguf carries general.file_type = 14, which is Q4_K_S, while the same repo's plain Qwen3.8-27B-Q3_K_M.gguf correctly carries 12. It is a label, not the tensors — the file is 12.518 GiB, which no real Q4_K_S of this model could be. It affects display only.

Vision crashes

One open vision-crash report exists, llama.cpp issue #27124, and it is on a Vulkan build on an AMD Ryzen AI MAX+ 395 under Windows — a different backend and a different vendor from this recipe's CUDA path. No CUDA equivalent surfaced in the llama.cpp issues opened since the model's release. If you hit one on CUDA, that is new information worth filing.