How much VRAM does Agents-A1 4B need?

About 8 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Agents-A1 4B on RTX 3060 Ti (8GB): Vision-Capable Local Agent via llama.cpp

What You'll Build

A local, vision-capable agent server on an 8GB RTX 3060 Ti: Agents-A1 4B — InternScience's Apache-2.0 dense agentic model — loaded into llama.cpp from InternScience's own Q4_K_M GGUF plus its own multimodal projector, exposed as an OpenAI-compatible endpoint that accepts both text and images. The interesting part of this recipe is not whether the weights fit — they fit with room to spare — but how much context you can afford on top of them. That question has an exact answer for this architecture, and it is the spine of everything below.

Hardware data: RTX 3060 Ti (8GB VRAM, Ampere GA104, sm_86) · Q4_K_M language weights 2.52GiB + vision projector 0.63GiB = 3.15GiB resident, leaving the rest of the card for KV cache and compute · See benchmark data

⚠️ The vendor does not document this runtime. The Agents-A1-4B model card documents vLLM and SGLang only — its note says these artifacts are "compatible with Hugging Face Transformers, vLLM, SGLang, etc." and llama.cpp, Ollama and LM Studio appear nowhere in it. What InternScience does ship is a set of first-party GGUF repositories including a mmproj projector file, which is a llama.cpp-specific artifact. The install steps below therefore come from llama.cpp's own multimodal documentation, and the architecture-support claims are verified against llama.cpp's source rather than against vendor prose.

ℹ️ Vision is a runnable affordance here, not a measured strength. 297 of the model's 723 weight tensors are model.visual.* (per its safetensors index), and llama.cpp recognises the projector type this mmproj declares, so image input is wired end to end. But every benchmark InternScience publishes for this model is text-agentic — BrowseComp, XBench-DS, GAIA, FrontierScience-Research, IFEval. The card never demonstrates or scores the vision path. Treat images as a capability you can use, not as a capability anyone has graded.

⚠️ Do not confuse the repo names. InternScience/Agents-A1-Q4_K_M-GGUF (no -4B) is the 35B sibling and will not fit this card. The repository this recipe uses is InternScience/Agents-A1-4B-Q4_K_M-GGUF. The -4B infix is the only thing distinguishing them.

Requirements

Component	Minimum	Tested
GPU	8GB VRAM, CUDA compute capability 8.6	RTX 3060 Ti (8GB)
RAM	16GB	—
Storage	3.38GB (Q4_K_M 2.71GB + projector 0.67GB)	—
Software	llama.cpp with CUDA, carrying `qwen35` + `qwen3vl_merger`	—

Sizing the Context to the Card

This is the part worth reading before you type a command.

What is fixed

Two files sit in VRAM for the whole session. Byte counts are from the HuggingFace tree API for the Q4_K_M repository:

File	Bytes	GiB
`Agents-A1-4B-Q4_K_M.gguf`	2,708,805,312	2.52
`Agents-A1-4B-mmproj.gguf`	672,423,200	0.63
Resident total	3,381,228,512	3.15

The projector is a separate file because llama.cpp keeps image encoding out of the language model — its docs state that running a multimodal model "typically requires two GGUF files". It is not optional if you want image input.

The heavier builds do not leave room for a session on this card. Q8_0 (4.17GiB) plus the projector is 4.80GiB resident; the F16 GGUF (7.85GiB) plus the projector is 8.47GiB, which exceeds the card before any context exists. The BF16 safetensors weights total 9,078,620,456 bytes (8.46GiB) and the first-party FP8 build is 7,793,059,280 bytes (7.26GiB) — neither leaves usable room on an 8GB card, and both are safetensors artifacts for the vLLM/SGLang path rather than GGUFs llama.cpp can load. Q4_K_M is the build this card wants.

What scales: the KV cache

The GGUF header carries the numbers that decide this. These are read straight out of Agents-A1-4B-Q4_K_M.gguf — no download required, an HTTP Range request for the first few megabytes of the file returns the whole metadata block. HuggingFace also parses it server-side, so ?expand[]=gguf on the repository will confirm architecture and context_length in one call:

qwen35.block_count             = 32
qwen35.full_attention_interval = 4
qwen35.attention.head_count_kv = 4
qwen35.attention.key_length    = 256
qwen35.attention.value_length  = 256
qwen35.context_length          = 262144

full_attention_interval = 4 over 32 blocks means only 8 layers are full-attention; the other 24 are linear-attention layers carrying a fixed-size recurrent state that does not grow with context. This is not an inference from the config — llama.cpp implements it structurally: llm_arch_is_hybrid(LLM_ARCH_QWEN35) returns true in src/llama-arch.cpp, and in src/llama-model.cpp the hybrid memory is built with filter_attn = il < hparams.n_layer() && !hparams.is_recr(il), so the context-scaling KV cache is allocated for the non-recurrent layers only.

So, per token, at the default f16 cache:

8 full-attn layers × (key 256 + value 256) × 4 KV heads × 2 bytes = 32,768 bytes = 32 KiB/token

With -ctk q8_0 -ctv q8_0 each cache element costs 34 bytes per block of 32 instead of 2 bytes each (block_q8_0 in ggml/src/ggml-common.h), which is 17 KiB/token. Both q8_0 and f16 are in the allowed -ctk/-ctv type list in common/arg.cpp.

The budget

Against a card that reports 8192 MiB, with 3.15GiB already resident:

Context	KV f16	Accounted (f16)	KV q8_0	Accounted (q8_0)
16,384	0.50 GiB	3.65 GiB	0.27 GiB	3.41 GiB
32,768	1.00 GiB	4.15 GiB	0.53 GiB	3.68 GiB
65,536	2.00 GiB	5.15 GiB	1.06 GiB	4.21 GiB
131,072	4.00 GiB	7.15 GiB	2.12 GiB	5.27 GiB
262,144	8.00 GiB	11.15 GiB — impossible	4.25 GiB	7.40 GiB

Read the last row carefully. The 262,144-token context the model advertises is not reachable on this card. At the default cache type its KV alone wants 8.00GiB — the entire card, before a single weight. Even with an 8-bit cache it needs 4.25GiB of KV on top of 3.15GiB of resident weights, leaving under a gigabyte for CUDA compute buffers, the vision encoder's working set and whatever your desktop has already taken. qwen35.context_length = 262144 describes what the model was trained for, not what this GPU can hold.

Recommendations for this card:

-c 32768 (default cache). 4.15GiB accounted, and the remainder of the card absorbs compute buffers, image encoding and a desktop session. This is the setting to start from.
-c 131072 -ctk q8_0 -ctv q8_0 (stretch). 5.27GiB accounted. This is the honest "long context on an 8GB card" answer, and it is the reason the 8-bit cache is worth the accuracy trade.
Do not ask for 262,144. See above.

The accounted column is arithmetic over file sizes and cache geometry, not a measured peak — llama.cpp's CUDA compute buffers and the vision encoder's activations are real and are not in it. No measured peak for this pair exists yet; see /check/agents-a1-4b/rtx-3060-ti and please send one via /contribute.

What an image costs

Images are charged to the same KV budget, as context tokens. For the qwen3vl_merger projector, tools/mtmd/clip.cpp computes n_patches = (nx / (patch_size * 2)) * (ny / (patch_size * 2)), and the projector's clip.vision.patch_size is 16 — so one output token per 32×32 pixels:

Image	Tokens	KV at f16	KV at q8_0
512×512	256	8 MiB	4 MiB
768×768	576	18 MiB	10 MiB
1024×1024	1024	32 MiB	17 MiB

At 32,768 tokens of context a 768×768 image spends under two percent of your window. Images are cheap here; long transcripts are what fill the cache.

Installation

1. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

2. Confirm your build carries both halves of this model

The language model and the image projector are separate support surfaces. Check both in the checkout you just built:

grep -n 'LLM_ARCH_QWEN35,'   src/llama-arch.cpp     # text architecture "qwen35"
grep -n 'qwen3vl_merger'     tools/mtmd/clip-impl.h # vision projector

Both must print a line. On master as of 2026-07-29 they resolve to { LLM_ARCH_QWEN35, "qwen35" } and { PROJECTOR_TYPE_QWEN3VL, "qwen3vl_merger" } respectively; both were also present in tagged builds b9960 and b10091. If either grep is silent, your checkout predates support and the model will refuse to load.

3. Get the weights

Download both files into a directory you control. The commands in ## Running below assume this layout:

pip install -U huggingface_hub
hf download InternScience/Agents-A1-4B-Q4_K_M-GGUF \
  Agents-A1-4B-Q4_K_M.gguf Agents-A1-4B-mmproj.gguf \
  --local-dir ./models/agents-a1-4b

Alternatively you can skip this step entirely and let llama.cpp fetch both files into its own HuggingFace cache at first launch — -hf selects the sibling GGUF whose filename contains mmproj, and Agents-A1-4B-mmproj.gguf matches, so the projector is downloaded and loaded without being named. If you take that route, use the -hf form of the run command in ## Running rather than the -m/--mmproj form, because the files will not be in ./models/agents-a1-4b/.

Running

Start the server with the context sized deliberately rather than left to auto-fit:

./build/bin/llama-server \
  -m ./models/agents-a1-4b/Agents-A1-4B-Q4_K_M.gguf \
  --mmproj ./models/agents-a1-4b/Agents-A1-4B-mmproj.gguf \
  -ngl 99 \
  -c 32768 \
  --host 127.0.0.1 --port 8000 \
  --temp 0.85 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.1

The sampling values are the ones InternScience recommends on the model card for stable multi-turn behaviour (temperature 0.85, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.1, repetition_penalty 1.0 — llama.cpp's repeat penalty already defaults to 1.0, so it is left off the line). -ngl 99 puts every layer on the GPU; -ngl all is the equivalent modern spelling.

For the long-context configuration, swap the last two lines of budget:

./build/bin/llama-server \
  -m ./models/agents-a1-4b/Agents-A1-4B-Q4_K_M.gguf \
  --mmproj ./models/agents-a1-4b/Agents-A1-4B-mmproj.gguf \
  -ngl 99 -c 131072 -ctk q8_0 -ctv q8_0 \
  --host 127.0.0.1 --port 8000

If you skipped the download step, the same server straight from HuggingFace — -hf resolves both the quant and its mmproj sibling into llama.cpp's own cache. Carry the same flags: a bare llama-server -hf … runs at -c 0 with --fit on, which is the misconfiguration described under Troubleshooting below.

./build/bin/llama-server \
  -hf InternScience/Agents-A1-4B-Q4_K_M-GGUF \
  -ngl 99 \
  -c 32768 \
  --host 127.0.0.1 --port 8000 \
  --temp 0.85 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.1

Whichever form you used, the server answers on http://127.0.0.1:8000/v1. Send an image the OpenAI way:

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Describe what this screenshot shows and list any UI actions available."},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,'"$(base64 -w0 screenshot.png)"'"}}
    ]}],
    "max_tokens": 512
  }'

To sanity-check the vision path without a client, llama-mtmd-cli takes a file directly:

./build/bin/llama-mtmd-cli \
  -m ./models/agents-a1-4b/Agents-A1-4B-Q4_K_M.gguf \
  --mmproj ./models/agents-a1-4b/Agents-A1-4B-mmproj.gguf \
  -ngl 99 -c 8192 --image screenshot.png -p "What is in this image?"

The model is trained to run as a research/tool-calling agent and the card publishes a system prompt for that role, including a tavily_search tool and a "Current date" line the card notes "should be updated dynamically when possible". Send that system prompt yourself, with a live date. If you send none, llama.cpp applies the template baked into the GGUF, whose default system prompt hard-codes Current date: 2026-07-14 — so an agent left on the default believes it is permanently mid-July 2026, which is exactly the failure the card's note is warning about. (The card's own printed copy of the prompt says 2026-07-13; the shipped artifact says 2026-07-14. Neither is today.)

Results

Speed: omitted. No measurement of Agents-A1 4B on an RTX 3060 Ti — or on any comparable consumer GPU — exists in a citable source as of 2026-07-29. The model is two weeks old — the card's news line announces the 4B on 2026.7.14, while the HuggingFace API reports createdAt: 2026-07-13T02:15:21Z for the repository (the three first-party GGUF repositories were created the same day) — and InternScience publishes no throughput figures at all. Rather than extrapolate, this recipe leaves the number empty: if you run it, please post tok/s via /contribute and it will land on /check/agents-a1-4b/rtx-3060-ti.
VRAM usage: 3.15GiB resident for weights plus projector; 1.00GiB of KV at the recommended 32,768-token context, 2.12GiB at 131,072 with an 8-bit cache. Derived from published file sizes and the GGUF's own cache geometry, as shown above — not a measured peak. Live data at /check/agents-a1-4b/rtx-3060-ti.
Quality notes: the published scores — BrowseComp 66.8, XBench-DS-2510 90.0, GAIA 95.1, FrontierScience-Research 33.3, IFEval 94.8 — are all text-agentic, from the model card's own evaluation table. Nothing in the card grades the vision path. Q4_K_M is also a real quantization loss on a model whose selling point is long-horizon multi-step reasoning, where small errors compound; if you have the headroom on a larger card, Q8_0 exists for that reason.

Troubleshooting

The server started with a much smaller context than I asked for

llama.cpp fits unset parameters to available device memory by default: --fit defaults to on, and -c defaults to 0, which the source comments as "context the model was trained with" — 262,144 here. Left alone, the server shrinks the context to whatever fits without warning you — the resolved n_ctx does appear in the startup log, but nothing flags that it is not what the model declares, and --fit-print is off by default. Always pass -c explicitly, so the number in your command is the number you are running. Both defaults are in common/common.h (fit_params = true, n_ctx = 0).

Out of memory at a long context

In order of impact:

Lower -c. Every 32,768 tokens costs 1.00GiB at the default cache type. This is the biggest lever by far.
Quantize the cache with -ctk q8_0 -ctv q8_0, which takes the per-token cost from 32 KiB to 17 KiB.
Stop offloading the projector. llama.cpp's docs state "By default, multimodal projector will be offloaded to GPU." — adding --no-mmproj-offload returns 0.63GiB to the card and encodes images on the CPU instead, which is slower per image but does not touch text throughput.

If you do not need images at all, load the model without --mmproj entirely and reclaim the same 0.63GiB. This is the GGUF analogue of the vendor's own vLLM text-only mode, which the card describes as "(skips vision encoder to free KV cache memory)".

`ollama run` — there is no Ollama path here, verified

ollama.com/library/agents-a1, /agents-a1-4b and /internscience/agents-a1 all return 404 (checked 2026-07-29). Ollama's library search surfaces only user-namespaced re-uploads of unverified provenance, several of them visibly re-labelled derivatives. Ollama does pin a llama.cpp build (LLAMA_CPP_VERSION = b10091) that carries the qwen35 text architecture — so the language half would very likely load — but this recipe has not verified that Ollama's model importer picks up the separate mmproj sidecar or routes image input through the qwen3vl_merger projector. Do not assume image input works there. Use llama.cpp directly, where both halves are verified above.

Images are accepted but nothing visual comes back

Check the server log for a projector load line at startup. If you launched with -hf and passed --no-mmproj, or supplied -m without --mmproj, the model is running text-only and will answer image prompts from the text alone. When loading local files you need both flags — llama.cpp's documentation says to use -m together with --mmproj "to specify text and multimodal projector respectively".

Expect churn on the multimodal path

llama.cpp's own multimodal README says the subsystem is under very heavy development and that "breaking changes are expected". Flag names and projector handling in this area move faster than the rest of the project. Pin the build you validated rather than tracking master on a machine you rely on.

A throughput regression has been reported for this architecture on older NVIDIA silicon

llama.cpp Issue #25162 ("Performance regression on Turing GPUs after 9e58d4d69 — Qwen35 SSM kernels affected", open as of 2026-07-29) reports a 24–42% token-generation regression on Qwen35-architecture models, traced to a change in how __restrict__ is applied in the SSM/linear-attention CUDA kernels this model uses. The report was measured by a community user on an RTX 2080 Ti (Turing, SM75); nobody in the thread has posted an Ampere measurement, and no maintainer has responded — every participant is a non-member. It is listed here because the affected kernels are the ones this architecture leans on, so it is worth knowing if you compare two builds and see an unexplained gap. It is not a claim about the RTX 3060 Ti.