How much VRAM does Agents-A1 35B-A3B need?

About 24 GB — the minimum this recipe targets.

How hard is this setup?

Advanced — follow the steps above.

Agents-A1 35B-A3B on RTX 3090: 128K Agentic Serving via llama.cpp, and What Vision Costs on 24 GB

What You'll Build

A private, local agent endpoint: Agents-A1 35B-A3B — InternScience's Apache-2.0 Mixture-of-Experts agentic model (35B total parameters, 256 experts, 8 active per token) — served as an OpenAI-compatible API by llama.cpp on a single 24 GiB RTX 3090, ready for any OpenAI-compatible agent client. The recipe leads with the vendor's own Q4_K_M GGUF at a 131,072-token context window, then shows exactly what the vision projector costs if you want image input on top of that — because on a 24 GiB card that 0.84 GiB is not free, and the honest answer is that you buy it with context.

Hardware data: RTX 3090 (24 GiB VRAM) · first-party Q4_K_M GGUF, 19.71 GiB of weights + a 0.84 GiB projector · 128K context with 8-bit KV cache · See benchmark data

ℹ️ An MoE keeps every expert resident — the file size is the VRAM cost. Agents-A1 routes 8 of 256 experts per token, but all of them stay loaded, so the memory footprint is the whole quant file (19.71 GiB at Q4_K_M) and not some smaller "active" fraction. A low active-parameter count buys throughput, not VRAM.

⚠️ Vision works and the vendor says so — but no vision-specific benchmark is published, and it is off by default in this recipe. The weights genuinely carry a vision tower, InternScience ships a first-party mmproj alongside every GGUF quant, and mainline llama.cpp can take image input (see Running). Asked directly whether multimodal capability survived post-training, InternScience's sY713 answered on discussion #18: "Agents-A1 does support multimodal reasoning. In fact, Agents-A1 has achieved strong results on challenging multimodal scientific reasoning benchmarks such as HLE and HiPhO." — both of which are rows in the card's own results table. What the vendor does not publish is a vision-specific eval: there is no VQA, MMMU, MMBench or DocVQA number anywhere, and in that same reply the team writes "We would be very interested to see further evaluations on broader multimodal tasks such as VQA, MMMU, MMBench, etc." — i.e. they are asking the community to produce them. The card's pipeline_tag is still text-generation, and a request to change it is open and unanswered. So: image input is a supported capability with a measurement gap, not an unknown. Evaluate it on your own data before depending on it.

ℹ️ This recipe is the 35B. There is also a 4B. InternScience also ships Agents-A1-4B with its own GGUF ladder — per its Q4_K_M repo file tree the text weights are 2.52 GiB (2,708,805,312 bytes) plus a 0.63 GiB projector, which fits an 8 GB card comfortably. If your card is smaller than 24 GiB, that is the model to use; nothing below applies to it.

⚠️ No Ollama library entry, and the vendor documents only vLLM and SGLang. https://ollama.com/library/agents-a1 returns HTTP 404 (checked 2026-07-29), and the model card never mentions llama.cpp, Ollama or LM Studio — its note block says only that "These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, etc." The llama.cpp path below is justified by two independent facts, not by that sentence: InternScience's own team published the GGUFs (see Requirements), and mainline llama.cpp has a first-class implementation of this architecture.

Requirements

Component	Minimum	This recipe
GPU	24 GiB VRAM (Q4_K_M is 19.71 GiB before any KV cache)	RTX 3090 (24 GiB, Ampere GA102, sm_86)
RAM	16 GB system RAM	32 GB comfortable for an agent + its workspace
Storage	~23 GB free	21.17 GB (`Agents-A1-Q4_K_M.gguf`) + 0.90 GB (`Agents-A1-mmproj.gguf`)
Software	llama.cpp CUDA build; the `hf` CLI to download	llama.cpp `llama-server` / `llama-mtmd-cli`

The GGUFs are first-party — the card just doesn't say so

The model card documents only Transformers, vLLM and SGLang, but the quantized builds are published by InternScience itself under the same org, and team members said so directly. On discussion #3, Lonepic (a verified org member) wrote: "Multiple quantized versions of the GGUF model are available in this collection, feel free to try them out~", linking the Agents-A1 collection. This matters because it means the GGUF you download was produced by the people who trained the model, not reconstructed by a third party.

Picking a quant for 24 GiB

Byte counts below come from the file trees of the three first-party GGUF repos. The GiB column is what nvidia-smi and llama.cpp report; the GB column is the decimal figure Hugging Face shows on its Files tab.

Build	Size (GB, decimal)	Size (GiB)	Fits 24 GiB?
F16	69.38	64.61	No
Q8_0	36.90	34.37	No
Q4_K_M	21.17	19.71	Yes — this recipe's pick
`Agents-A1-mmproj.gguf` (vision projector)	0.90	0.84	Optional add-on, shipped in all three repos

There is no official Q3 or Q2 build, so Q4_K_M is the floor and 24 GiB is the entry tier. The projector is byte-identical across the three GGUF repos (899,282,976 bytes in each) and is independent of the text quant, so you fetch it once.

Why the GGUF path, and not vLLM or SGLang

The card's own launch examples are --tp-size 1 / --tensor-parallel-size 1 — a single GPU — but they serve the unquantized bf16 checkpoint, whose safetensors come to 65.40 GiB in the canonical file tree. That cannot sit on a 24 GiB card in any arrangement. InternScience also publishes an FP8 build at 35.09 GiB, still well past 24 GiB — and Ampere has no FP8 tensor cores anyway. On a single RTX 3090 the 4-bit GGUF is the only build that fits, and the missing FP8 hardware is irrelevant because K-quants are integer formats.

Installation

1. Install llama.cpp with CUDA support

The RTX 3090 is Ampere (GA102, sm_86) and needs nothing beyond a standard CUDA build — no FP8 path, and no separate flash-attn package (llama.cpp ships its own attention kernels).

# conda-forge — the pre-built channel llama.cpp documents as CUDA-enabled
conda install -c conda-forge llama.cpp

# …or build from source with CUDA (always explicit, always correct)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Confirm CUDA is actually active before continuing — llama-server prints its backend and the offloaded-layer count at startup. A CPU or Vulkan build will "work" and then run at a fraction of the speed.

Use a build from master, and recent. Two things must be present, and both are verifiable in the source rather than in anyone's prose:

The text architecture. src/llama-arch.cpp registers LLM_ARCH_QWEN35MOE under the GGUF architecture string qwen35moe — which is exactly what this model's GGUF header declares — and classifies it as hybrid via llm_arch_is_hybrid(). That classification is what makes the 128K context below affordable (see Running).
The vision projector. tools/mtmd/clip-impl.h maps PROJECTOR_TYPE_QWEN3VL to the projector string qwen3vl_merger, which is what this model's mmproj header declares, and tools/mtmd/clip.cpp builds a dedicated graph for it. Multimodal support in llama.cpp lives in tools/mtmd and is, in the maintainers' own words on that page, "under very heavy development, and breaking changes are expected" — another reason to pin a recent build and re-test after upgrades.

2. Download the GGUF — and decide about the projector now

The shortest route is to let llama-server fetch everything. -hf takes <user>/<model> and resolves the files itself:

# pulls Agents-A1-Q4_K_M.gguf (19.71 GiB) AND Agents-A1-mmproj.gguf (0.84 GiB)
llama-server -hf InternScience/Agents-A1-Q4_K_M-GGUF

# text-only: same download, projector skipped entirely
llama-server -hf InternScience/Agents-A1-Q4_K_M-GGUF --no-mmproj

⚠️ -hf picks up the projector on its own — and puts it in VRAM. common/download.cpp selects "the best sibling GGUF whose filename contains keyword" with the keyword mmproj, and Agents-A1-mmproj.gguf matches. The defaults in common/common.h then compound it: no_mmproj = false (auto-fetch on) and mmproj_use_gpu = true (load it onto the GPU). On a card this tight that is 0.84 GiB you did not budget for, and nothing in the startup output shouts about it. If you are not sending images, pass --no-mmproj.

If you would rather have the files where you can see them, fetch them explicitly instead:

pip install -U huggingface_hub

# text weights only (19.71 GiB)
hf download InternScience/Agents-A1-Q4_K_M-GGUF \
  --include "Agents-A1-Q4_K_M.gguf" --local-dir ./

# the vision projector (0.84 GiB) — only if you want image input
hf download InternScience/Agents-A1-Q4_K_M-GGUF \
  --include "Agents-A1-mmproj.gguf" --local-dir ./

Every llama-server command below uses -m <file> against these local paths, which is also the reason they need no --no-mmproj: the auto-sibling lookup only runs for -hf. If you swap in the -hf form, carry --no-mmproj with it on the text-only commands.

3. There is no chat-template step — and that is worth knowing

Third-party quants of fast-moving models routinely freeze a stale chat template, so the usual advice is to fetch the canonical chat_template.jinja and pass it with --chat-template-file. That is unnecessary here. The template Hugging Face parses out of the published GGUF is byte-identical to the canonical chat_template.jinja in the safetensors repo — same 7,756 bytes, same SHA-1 f54c154d3de0a2c6c71a39ea77d8d819b74e54e3, verified 2026-07-29. That is the benefit of a first-party quant: there is no window between the vendor fixing a config file and a redistributor re-uploading.

The template does carry guards that agent harnesses trip over, though. See Troubleshooting before you wire an agent to it.

Running

1. Serve text-only at a 128K context

llama-server \
    -m Agents-A1-Q4_K_M.gguf \
    -a agents-a1 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    -ngl 99 \
    -c 131072 \
    -fa on \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.85 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 1.1

Every flag is defined in the canonical llama.cpp flag definitions (-ngl/--gpu-layers, -c/--ctx-size, -fa/--flash-attn which takes [on|off|auto], -ctk/--cache-type-k, -ctv/--cache-type-v, --jinja, and -a/--alias, defined there as "set model name aliases, comma-separated (to be used by API)" — without it the server reports the GGUF's path as its model id, and the agent config in step 5 has to match whatever it reports). The sampling values are the ones the model card lists under "Recommended Sampling Parameters": temperature 0.85, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.1. The card lists a sixth, repetition_penalty 1.0, which is omitted above only because --repeat-penalty already defaults to 1.0 — pass it explicitly if you prefer the command to be self-documenting.

--jinja is what makes llama-server apply the model's own Jinja template, and it is also what turns on tool calling. There is no llama.cpp equivalent of the --tool-call-parser qwen3_coder flag the card passes to vLLM and SGLang: llama.cpp derives a parser from the template itself at load time — the machinery lives in common/chat-auto-parser*.cpp and common/chat-diff-analyzer.cpp, which render the template with and without tool calls and diff the results. This model's template emits an XML-shaped <tool_call><function=…><parameter=…> block rather than JSON, and the auto-parser is what turns that back into structured tool calls for your client.

This exposes an OpenAI-compatible API at http://127.0.0.1:8080/v1, plus llama.cpp's built-in web UI at http://127.0.0.1:8080 for a quick smoke test.

2. Why 128K fits where a normal 35B would not

A 30B-class model on a 24 GiB card is usually forced down to an 8–16K context, because a KV cache on every layer eats the few GB the weights leave behind. This architecture is different, and the reason is structural.

In the model's config.json, text_config declares num_hidden_layers: 40 with full_attention_interval: 4, and its layer_types array spells the pattern out: 10 of the 40 layers are full_attention; the other 30 are linear_attention. The same fact is baked into the GGUF header as qwen35moe.full_attention_interval = 4 over qwen35moe.block_count = 40. Only the full-attention layers keep a cache that grows with context; the 30 linear layers carry a fixed-size recurrent state that does not grow as the conversation gets longer.

llama.cpp models this directly. llm_arch_is_hybrid() in src/llama-arch.cpp returns true for LLM_ARCH_QWEN35MOE, and create_memory() in src/llama-model.cpp builds a llama_memory_hybrid whose attention filter for this architecture is il < hparams.n_layer() && !hparams.is_recr(il) — i.e. KV cache is allocated for the non-recurrent layers only.

So the KV budget is a derived envelope you can compute yourself. The GGUF header gives attention.head_count_kv = 2 and attention.key_length = attention.value_length = 256, so each token costs 2 × (256 + 256) = 1024 cache values per full-attention layer:

Context	fp16 KV (default)	`q8_0` KV (this recipe)
32,768	0.63 GiB	~0.33 GiB
65,536	1.25 GiB	~0.66 GiB
131,072	2.50 GiB	~1.3 GiB
262,144 (native max)	5.00 GiB	~2.7 GiB

At 20 KiB per token in fp16 across those 10 layers, -ctk q8_0 -ctv q8_0 roughly halves it — which is the difference between 128K being comfortable and 128K being the thing that OOMs you.

3. The 24 GiB budget, and where the projector fits

Adding up the pieces above against the card's 24 GiB:

Configuration	Weights	KV cache	Projector	Subtotal	Slack before compute buffers
Text-only, `-c 131072`	19.71	~1.3	—	~21.0 GiB	~3.0 GiB
Vision on, `-c 65536`	19.71	~0.66	0.84	~21.2 GiB	~2.8 GiB
Vision on, `-c 131072`	19.71	~1.3	0.84	~21.9 GiB	~2.1 GiB
Text-only, `-c 262144`	19.71	~2.7	—	~22.4 GiB	~1.6 GiB
Vision on, `-c 262144`	19.71	~2.7	0.84	~23.3 GiB	~0.7 GiB — don't

The 30 linear-attention layers are not in the table because their recurrent state is roughly 60 MiB per sequence in total — below the table's rounding, and genuinely context-independent, which is the whole point of the split.

These are derived figures, not a measurement: /check/agents-a1-35b-a3b/rtx-3090 currently has no benchmark data. Compute buffers, the CUDA context and anything driving your display all come out of the remaining slack, which is why the last row is not a recipe. If you measure real peaks on this card, please submit them via /contribute so this table can be replaced by numbers somebody actually observed.

The practical reading: on 24 GiB you pick two of {vision, 128K+ context, comfortable headroom}. That is not a llama.cpp quirk — InternScience makes the same trade explicitly on the vLLM side, where the card's third launch example is labelled "Text-Only" and described as "skips vision encoder to free KV cache memory".

4. Turning vision on

If you want image input, pass the projector. llama-mtmd-cli is the dedicated multimodal CLI; llama-server takes the same flag and exposes images through the OpenAI-compatible endpoint.

# one-shot, from the CLI
llama-mtmd-cli \
    -m Agents-A1-Q4_K_M.gguf \
    --mmproj Agents-A1-mmproj.gguf \
    -ngl 99 -c 65536 -fa on -ctk q8_0 -ctv q8_0 \
    --image ./screenshot.png \
    -p "Describe what this interface is showing."

# …or serve it
llama-server \
    -m Agents-A1-Q4_K_M.gguf \
    --mmproj Agents-A1-mmproj.gguf \
    --host 127.0.0.1 --port 8080 --jinja \
    -ngl 99 -c 65536 -fa on -ctk q8_0 -ctv q8_0

Note the context dropped from 131,072 to 65,536 — that is the projector being paid for. If you would rather keep the long context, there is a middle path: --no-mmproj-offload keeps the projector in system RAM instead of VRAM. GPU offload of the projector is the default (mmproj_use_gpu = true in common/common.h), so this is something you turn off deliberately. Image encoding gets slower; the 0.84 GiB goes back to the KV cache. A third lever, if you need both vision and the full context, is -ncmoe/--n-cpu-moe N, documented in common/arg.cpp as "keep the Mixture of Experts (MoE) weights of the first N layers in the CPU" — that trades throughput for VRAM and needs system RAM to hold whatever you push off the card.

One structural detail worth knowing, because it is unusual for a Qwen3-VL-style projector: this one declares 27 vision blocks with the deepstack path switched off. Its GGUF header carries clip.vision.is_deepstack_layers as 27 entries that are all false, it contains no v.deepstack.* tensors, and that matches the empty deepstack_visual_indexes list in the canonical config.json. The practical consequence is that it emits a single 2,048-wide embedding stream — exactly matching the text model's embedding_length of 2,048, which is the only compatibility check tools/mtmd performs at load time — instead of the wider concatenated stream a deepstack projector would produce. It is also why the projector is a flat 0.84 GiB in all three GGUF repos rather than something that scales with the text quant.

5. Point an agent client at the server

Adapted from the llama.cpp provider block in the opencode provider docs, with the model id and context limit changed to match the server above. Per the opencode config docs, save it as opencode.json in your project root or as ~/.config/opencode/opencode.json for a user-wide setting:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "agents-a1": {
          "name": "Agents-A1 35B-A3B (local)",
          "limit": {
            "context": 131072,
            "output": 65536
          }
        }
      }
    }
  }
}

The model reasons inside <think> blocks by default — its template opens one on every generation prompt unless enable_thinking is false. Server-side you can switch that off with -rea/--reasoning off, which takes [on|off|auto] per the canonical flag list. Leave it on for agentic work; the reasoning trace is where this model does its planning.

Results

Speed: Omitted. No throughput measurement for this model on an RTX 3090 exists — /check/agents-a1-35b-a3b/rtx-3090 has no benchmark data, and the community throughput figures on the model card discussions are all from other vendors' cards and other runtimes, so quoting them here would be a hardware claim nobody made. If you run it, please submit your numbers via /contribute. Note also that this is a reasoning model: much of each turn is <think> content you discard, so effective throughput is lower than a raw tok/s figure suggests.
VRAM usage: Q4_K_M weights are 19.71 GiB (21,166,757,632 bytes, from the first-party GGUF file tree) and must stay resident. With q8_0 KV at 131,072 tokens the derived total is ~21.0 GiB of the card's 24 GiB text-only, or ~21.9 GiB with the projector loaded — see the budget table in Running. Derived, not measured; /check/ is where a measured figure will land.
Quality notes: InternScience reports Seal-0 56.36, HiPhO 46.4, FrontierScience-Olympiad 79.0, FrontierScience-Research 40.0, IFBench 80.61, IFEval 94.82, BrowseComp 75.51 and GAIA 96.04 on the model card, with the evaluation harness open-sourced in the GitHub repo and the method described in the technical report. These are the vendor's own runs, not independent measurements and not measured on this GPU — and they were produced from the bf16 checkpoint, whereas this recipe serves a 4-bit quant. Two of them — HiPhO and HLE w/ tools — are ones InternScience describes on discussion #18 as "challenging multimodal scientific reasoning benchmarks", so the table is not purely text-agentic. What is absent is any vision-specific eval — no VQA, MMMU, MMBench or DocVQA figure is published, and the team asks the community to run those.

For the full benchmark data, see /check/agents-a1-35b-a3b/rtx-3090.

Troubleshooting

Out of memory at launch

The 19.71 GiB of weights leave 4.29 GiB on a 24 GiB card, so an OOM at startup almost always means the KV cache, the projector, or both are too large for what's left. Work down this ladder:

Keep -fa on and -ctk q8_0 -ctv q8_0 — dropping either roughly doubles the cache.
Drop the projector: --no-mmproj on the -hf form, or simply don't pass --mmproj. That is 0.84 GiB back immediately, and if you weren't sending images you were paying it for nothing.
Move the projector off the GPU instead of dropping it: --no-mmproj-offload.
Lower -c (65536, then 32768). Each halving halves the KV cache.
Offload some experts to system RAM with -ncmoe/--n-cpu-moe N (flag definition) — this trades throughput for VRAM.

Watch nvidia-smi during a real task rather than at idle: a hard problem produces a long <think> block that grows the cache mid-generation, so size for the peak.

An agent harness fails with "System message must be at the beginning"

The chat template raises a Jinja exception when a message with role system appears anywhere except first — the guard is literally raise_exception('System message must be at the beginning.'). Many agent harnesses inject a system message mid-conversation, and with --jinja that is a hard failure rather than a warning. Fix it in the client: merge all system content into the single leading system message. Do not hand-edit the template — it is the vendor's current file, not a stale copy (see installation step 3).

The same template carries a second guard agents trip over: raise_exception('No user query found in messages.') fires when every user-role message in the history is a bare <tool_response> block. If your harness replays a tool loop without ever including a real user turn, that is the error you will see. And raise_exception('System message cannot contain images.') fires if you attach an image to the system message rather than to a user message.

The model behaves differently inside an agent harness than in a plain chat

This is acknowledged upstream. On discussion #7, a user reported a reasoning question answered correctly in a plain chat UI but not inside an agent harness; InternScience's sY713 replied that agent harnesses "usually inject additional system prompts / instructions by default, which can dilute or override some of the key steering signals and make the model behave differently compared with a clean LM Studio chat setting." If a task fails in your agent but passes in the web UI, audit what your harness is prepending before you blame the quant.

Tool calls come out malformed

A community user reports on discussion #2 that in agent work the card's recommended temperature — "0.85 is sometimes fail in tool calling" — was worse for them than a lower value. This is a single community report with no vendor confirmation, so the recipe keeps the card's documented sampling values as the default — but lowering temperature is the cheapest thing to try before changing anything structural. Also confirm --jinja is actually set: without it, llama-server never applies the template that emits <tool_call> blocks at all, and every tool call comes back as plain prose.

Images are ignored, or the projector won't load

Check three things in order. First, that your build has multimodal support and you passed --mmproj (or used -hf without --no-mmproj). Second, that the two files match: the projector's clip.vision.projection_dim is 2,048 and the text model's embedding_length is 2,048, and tools/mtmd refuses to start with an error naming the two mismatched n_embd values if they don't — so that error means you paired the wrong files, most likely a 4B projector with 35B weights. Third, remember that multimodal support in llama.cpp is explicitly documented as fast-moving; if image input regressed after an upgrade, re-test against the previous build before assuming the weights are at fault.

llama.cpp reports no GPU, or layers stay on CPU

Confirm the build has CUDA enabled (-DGGML_CUDA=ON from source) and that -ngl 99 is offloading every layer — llama.cpp prints the offloaded-layer count at startup. The RTX 3090 needs no special flags: you do not need FP8 support (K-quants are integer) and you do not need to pip install flash-attn (llama.cpp's -fa on uses its own kernels).