How much VRAM does Agents-A1 4B need?

About 14 GB — the minimum this recipe targets.

How hard is this setup?

Intermediate — follow the steps above.

Agents-A1 4B on Apple M2 Max: First-Party GGUF + Vision on Metal at the Full 262K Context

What You'll Build

A local, private agentic assistant: Agents-A1-4B — InternScience's Apache-2.0, 4.54B-parameter dense vision-language model — served as an OpenAI-compatible endpoint on an Apple M2 Max, with image input working, at the full 262,144-token context the model declares, and with no NVIDIA GPU, no CUDA toolkit and no FlashAttention wheel anywhere in the stack. The lead path is llama.cpp with the Metal backend running InternScience's own Q8_0 GGUF (4.17 GiB) plus its own mmproj projector (0.63 GiB); a fully-documented MLX alternative is in Installation → Option B.

Hardware data: Apple M2 Max (64 GB unified memory) · Q8_0 GGUF 4.17 GiB + mmproj 0.63 GiB · ~12.80 GiB working set at the full 262,144-token context · See benchmark data

ℹ️ Unified memory is not VRAM — but on this machine, for this model, fit is the boring axis. The M2 Max has 64 GB of unified memory shared by the OS, CPU and GPU; it is not 64 GB of dedicated VRAM. macOS lets the GPU address roughly 75% of it by default (~48 GB via Metal's recommendedMaxWorkingSetSize on a ≥ 64 GB Mac), and every statement below is against that ~48 GB addressable pool, not the raw 64 GB. The point of stating it here is that this model does not come close to it: weights plus the projector plus a full-length KV cache land near 12.8 GiB — about a quarter of the pool. Even the F16 build at the same context is ~16.5 GiB. No wired-limit raise is required for anything in this recipe, and the margin is wide enough that the GiB-versus-GB distinction changes no conclusion here. Troubleshooting still documents the iogpu.wired_limit_mb lever, because it is the right tool if you run this model alongside something large.

The axes that do bite on this machine are the ones below: how much context you actually want to pay for, and how many bytes per token you are willing to stream through a bandwidth-bound GPU.

ℹ️ Vision here is a runnable affordance, not a measured strength. 297 of the model's 723 tensors are model.visual.*, InternScience ships a first-party projector, and llama.cpp loads it — so image input genuinely works, and this recipe shows you how. But the vendor never demonstrates or benchmarks it: the model card publishes only text-agentic results (BrowseComp, XBench-DS-2510, GAIA, FrontierScience-Research, IFEval), shows no image example, and mentions vision in a usage context exactly once — as a vLLM flag that disables it, annotated "skips vision encoder to free KV cache memory". Treat the visual path as something you should evaluate on your own data before depending on it.

ℹ️ This recipe leads the first-party GGUF, not MLX — because of the artifacts, not the framework. MLX can run this model: mlx-vlm implements the qwen3_5 architecture with a full vision tower (see mlx_vlm/models/qwen3_5/). What is missing is a trustworthy build. The mlx-community Agents-A1 collection — a 3/4/5/6/8-bit, bf16 and OptiQ-4bit ladder — is entirely for the 35B sibling: every one of those repos declares base_model: InternScience/Agents-A1, not the 4B. For the 4B, every MLX build on Hugging Face is a single-author conversion with 0–1 likes, and the most-downloaded of them has no vision at all (see Option B). The GGUF path, by contrast, is published by InternScience itself under the same org and the same Apache-2.0 licence. That is the whole reason for the ordering; if a mlx-community 4B build appears, the calculus changes.

⚠️ There is no Ollama library entry. ollama.com/library/agents-a1, ollama.com/library/agents-a1-4b and ollama.com/library/internscience/agents-a1 all return HTTP 404. This recipe makes no Ollama claim. More surprising: InternScience documents no GGUF runtime at all. Its GGUF repositories carry a copy of the main model card, whose Usage section covers only SGLang and vLLM — the vendor ships the GGUF and the projector, then leaves you to find a runtime. The llama.cpp support this recipe relies on is verified below against llama.cpp's own source, not against a vendor claim.

Requirements

Component	Minimum	Tested
GPU / memory	~14 GB of GPU-addressable unified memory for the lead configuration; ~6.3 GB if you cap context at 32,768	Apple M2 Max (38-core GPU, 64 GB unified memory)
RAM	Same pool — unified memory	64 GB unified
Storage	~5.2 GB (Q8_0 4.17 GiB + mmproj 0.63 GiB = 4.80 GiB)	4.80 GiB downloaded
Software	llama.cpp b10176+ (Metal, the macOS default); optionally `mlx-vlm` ≥ 0.6.6 and Python 3.10+	`llama-server`, `llama-mtmd-cli`

Per-file sizes, byte counts taken from the Hugging Face tree API for each first-party repository. Every one of the three GGUF repositories ships the same Agents-A1-4B-mmproj.gguf (672,423,200 bytes), so the projector cost is identical whichever quant you pick:

Build	Language weights	+ projector	Total	Notes
Q4_K_M	2.52 GiB	0.63 GiB	3.15 GiB	Smallest; pick it to leave the machine free, not because you must
Q8_0	4.17 GiB	0.63 GiB	4.80 GiB	Recommended lead — near-lossless, and half the bytes-per-token of F16
F16	7.85 GiB	0.63 GiB	8.47 GiB	Reference fidelity; fits comfortably, but see the note below

Why Q8_0 rather than F16, when memory is not the constraint. This is the question a 64 GB Mac forces, and the answer is not fit — both fit with enormous margin. It is bandwidth. Token generation on Apple Silicon is memory-bandwidth-bound, and the M2 Max has "400GB/s of unified memory bandwidth" (Apple Newsroom, January 2023). The F16 file is 8,424,394,432 bytes against Q8_0's 4,482,404,032 — 1.88× the bytes to stream for every token generated, for a quantization step (16-bit → 8-bit integer) that is conventionally treated as near-lossless. There is no published quantization-quality study for this model, so that last clause is a general expectation rather than a measurement here; if fidelity is what you are testing, F16 is available and it fits. But when memory is free and bandwidth is not, 8-bit is the better default.

Where min_vram_gb: 14 comes from — the context arithmetic. The model declares a context_length of 262,144 (confirmed in the GGUF header itself, via Hugging Face's server-side parse of InternScience/Agents-A1-4B-Q4_K_M-GGUF, which also reports architecture: qwen35). Whether that is reachable depends entirely on the KV cache, and this architecture is hybrid, which makes the cache far cheaper than a 262K context normally implies.

From the published config.json — the fields below all live under text_config — the model has 32 layers whose layer_types are 24 linear_attention and only 8 full_attention (full_attention_interval: 4), with num_key_value_heads: 4 and head_dim: 256. Only those 8 full-attention layers hold a cache that grows with context. At fp16 that is 8 × 2 (K and V) × 4 × 256 × 2 bytes = 32 KiB per token:

Context	KV at fp16	KV at `q8_0`	Working set (Q8_0 + projector + fp16 KV)
32,768	1.00 GiB	0.53 GiB	5.80 GiB
131,072	4.00 GiB	~2.1 GiB	8.80 GiB
262,144 (full)	8.00 GiB	4.25 GiB	12.80 GiB

The 24 linear-attention layers carry a fixed-size recurrent state instead — it does not grow with context and is small enough (tens of MB, from the linear_* dimensions in the same config) to disappear into rounding at this scale. So min_vram_gb: 14 records the 12.80 GiB working set of the lead command below, which runs at the full declared context with an unquantized cache. Mind the units: the table above is in GiB, but min_vram_gb is compared against the catalog's vram_gb — decimal GB, which is why this machine is listed as 64. the working set is 13,744,761,824 bytes — 12.80 GiB but 13.74 GB — so the field reads 14, not 13. Cap -c at 32,768 and the same setup needs 5.80 GiB (6.23 GB). This is a derived envelope from published byte counts and the published config, not a measured peak — see /check/agents-a1-4b/m2-max for measured data as it lands.

Installation

1. Install llama.cpp with Metal

There is nothing CUDA-shaped to install on Apple Silicon — no CUDA toolkit, no cu12x wheel index, no flash-attn build, no FP8 or NVFP4 path, no bitsandbytes, no GPTQ/AWQ/Marlin/ExLlamaV2, no ROCm and no HSA_OVERRIDE. llama.cpp's Metal backend runs the model on the Apple GPU and is on by default: "On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU." (llama.cpp build docs).

# Homebrew ships a Metal-enabled build on macOS
brew install llama.cpp

# …or build from source (Metal is on by default on macOS)
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j

Use a current build. Both halves of this model are recent additions, and both are verifiable in llama.cpp's own source rather than in anyone's prose:

The text architecture string in the GGUF header is qwen35, registered in src/llama-arch.cpp as { LLM_ARCH_QWEN35, "qwen35" }. The same file's llm_arch_is_hybrid() returns true for it, which is what routes it to the hybrid attention/recurrent memory path the KV arithmetic above depends on.
The projector type in the mmproj header is qwen3vl_merger, mapped in tools/mtmd/clip-impl.h as { PROJECTOR_TYPE_QWEN3VL, "qwen3vl_merger" } and built by clip_graph_qwen3vl in tools/mtmd/clip.cpp.

Release b10176 (2026-07-29) or newer carries both. Anything materially older is untested against these files — see Troubleshooting if you get gibberish or a load failure.

2. Get the weights and the projector

llama-server can pull both straight from Hugging Face. Its downloader picks the main GGUF and then looks for a sibling whose filename contains mmproj (common/download.cpp, find_best_mmproj); Agents-A1-4B-mmproj.gguf matches, so the projector comes down automatically:

# Downloads Agents-A1-4B-Q8_0.gguf (4.17 GiB) + Agents-A1-4B-mmproj.gguf (0.63 GiB)
llama-server -hf InternScience/Agents-A1-4B-Q8_0-GGUF

If you would rather have the files where you can see them, download explicitly and pass them by path — the form documented in llama.cpp's multimodal guide is -m model.gguf together with --mmproj file.gguf:

pip install -U huggingface_hub

hf download InternScience/Agents-A1-4B-Q8_0-GGUF Agents-A1-4B-Q8_0.gguf --local-dir .
hf download InternScience/Agents-A1-4B-Q8_0-GGUF Agents-A1-4B-mmproj.gguf --local-dir .

Swap Q8_0 for Q4_K_M or F16 in both the repository name and the filename to change tier; each repository carries its own copy of the identical projector.

The chat template is current — no patching needed. The canonical repository and all three GGUF repositories were created on 2026-07-13 and last modified on 2026-07-14, and the commit that last touched chat_template.jinja is that same 2026-07-14 revision — so the quants never had a window in which they could drift behind the template. Hugging Face's server-side parse of the Q4_K_M GGUF returns an embedded template that is byte-for-byte identical to the canonical file (SHA-1 0609d0e38f32, 8 raise_exception calls in both). Redistributed quants frequently freeze a pre-fix template; these do not, so there is no template step to perform. Pass --jinja and llama.cpp will use the one baked into the file.

Option B — the MLX path

MLX is Apple's own array framework and the other native Apple Silicon runtime. Stock mlx-vlm implements this architecture — mlx_vlm/models/qwen3_5/ contains the language, vision and config modules, with vision.py subclassing the Qwen3-VL vision model — so the framework is not the problem. The builds are.

Choose carefully. The most-downloaded MLX build of this model has no vision tower. wcamon/Agents-A1-4B-MLX-4bit is the top MLX result by downloads, and its own README is upfront that it is "an unofficial text-only MLX conversion": it was produced with mlx_lm.convert (the text-only converter) and its model.safetensors.index.json lists 924 tensors with zero vision entries, against the canonical model's 723 tensors of which 297 are model.visual.*. Its config.json has no vision_config block at all. It is a perfectly good text-only build and it says so — but if you pick it expecting a VLM, images will simply never work.

Builds that do carry the full vision tower — all 297 visual tensors, verified from their safetensors indexes — include the following. This is a list of notable builds rather than an exhaustive census of the Hub, and every one of them is a single-author repository with no more than one like:

Repository	Size on disk	Tensors (vision)	Notes
`ToPo-ToPo/Agents-A1-4B-mlx-4bit`	2.83 GiB	1221 (297)	`mlx-vlm 0.6.6`; card documents the conversion and a vision smoke test
`ToPo-ToPo/Agents-A1-4B-mlx-8bit`	4.78 GiB	1221 (297)	`mlx-vlm 0.6.6`; closest analogue to this recipe's Q8_0 lead
`ToPo-ToPo/Agents-A1-4B-mlx-bf16`	8.46 GiB	723 (297)	`mlx-vlm 0.6.6`; unquantized
`thoddnn/Agents-A1-4B-MLX-4bit`	2.83 GiB	1221 (297)	`mlx-vlm 0.6.6`; byte-identical weights to the ToPo-ToPo 4-bit
`Brooooooklyn/Agents-A1-4B-mxfp4-mlx`	4.45 GiB	923 (297)	MXFP4; zero downloads at the time of writing, untried here
`Brooooooklyn/Agents-A1-4B-nvfp4-mlx`	4.44 GiB	923 (297)	NVFP4; zero downloads at the time of writing, untried here

All six declare base_model: InternScience/Agents-A1-4B and license: apache-2.0 in their Hugging Face metadata. The 8-bit ToPo-ToPo build is the closest MLX analogue to this recipe's Q8_0 lead, and at 4.78 GiB it lands in essentially the same memory envelope, so the fit arithmetic in Requirements carries over unchanged. It is also the one with a documented conversion recipe and a stated vision check, which is why the command below uses it.

pip install -U "mlx-vlm>=0.6.6"

python -m mlx_vlm.generate \
    --model ToPo-ToPo/Agents-A1-4B-mlx-8bit \
    --image ./screenshot.png \
    --prompt "Describe what this interface is showing." \
    --max-tokens 512 \
    --temperature 0.85

The flag is --temperature here, confirmed against mlx_vlm/generate/dispatch.py — note that this differs from mlx-lm, whose CLI takes --temp, so do not carry flags between the two. That same argument parser exposes --repetition-penalty and --presence-penalty but not top_p/top_k/min_p, so the vendor's full recommended sampling set (below) cannot be expressed on the mlx-vlm command line; use the llama.cpp path if you want all of it.

mlx-vlm 0.6.6 or newer is required — the ToPo-ToPo card states that earlier versions do not implement the qwen3_5 architecture and can neither load nor convert this model. Note also that mlx-lm alone will only ever give you the text tower; the vision path lives in mlx-vlm.

Running

1. Serve it with Metal at the full context

Every flag below is defined in llama.cpp's canonical flag definitions (-ngl/--gpu-layers, -c/--ctx-size, -ctk/--cache-type-k, -ctv/--cache-type-v, --jinja, -fa/--flash-attn). Sampling values are the ones the model card recommends: temperature 0.85, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 1.1, repetition_penalty 1.0.

llama-server \
    -hf InternScience/Agents-A1-4B-Q8_0-GGUF \
    --port 8000 \
    -ngl 99 \
    -c 262144 \
    --jinja \
    --temp 0.85 --top-p 0.95 --top-k 20 --min-p 0.0 \
    --presence-penalty 1.1 --repeat-penalty 1.0

-ngl 99 offloads every layer to the Apple GPU through Metal. There are no CUDA semantics involved and no -DGGML_CUDA=ON to set.
-c 262144 asks for the model's full declared context. That is the 8.00 GiB fp16 KV cache from the table above, on top of 4.80 GiB of weights and projector — a ~12.80 GiB working set inside a ~48 GB addressable pool. This is the configuration min_vram_gb: 14 describes. If you would rather not pay 8 GiB up front, -c 131072 halves it, and -ctk q8_0 -ctv q8_0 roughly halves it again (to 4.25 GiB at the full length) at some cost in cache precision.
Leave -fa at its default. The flag takes on, off or auto and defaults to auto, which lets llama.cpp decide whether its Flash-Attention kernels apply. This is a hybrid attention architecture and support for it is recent; forcing -fa on is not something this recipe has grounds to recommend.
--jinja uses the chat template embedded in the GGUF, which is current (Installation step 2).

This exposes an OpenAI-compatible API at http://localhost:8000/v1.

2. Talk to it, with an image

The server's /chat/completions endpoint accepts image content parts once an mmproj is loaded:

pip install -U openai
export OPENAI_BASE_URL="http://localhost:8000/v1"
export OPENAI_API_KEY="EMPTY"

import base64
from openai import OpenAI

client = OpenAI()

with open("screenshot.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="InternScience/Agents-A1-4B",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is this interface showing, and what would you click next?"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
    max_tokens=1024,
    temperature=0.85,
    top_p=0.95,
    presence_penalty=1.1,
    extra_body={"top_k": 20, "min_p": 0.0},
)
print(resp.choices[0].message.content)

For a quick one-shot check without writing any client code, llama-mtmd-cli takes the same two files directly:

llama-mtmd-cli \
    -m Agents-A1-4B-Q8_0.gguf \
    --mmproj Agents-A1-4B-mmproj.gguf \
    --image screenshot.png \
    -p "Describe this image."

3. The system prompt matters

The model card publishes a specific system prompt that establishes the assistant's identity as Intern-A1 and, more importantly, tells it when not to reach for tools — direct answers for everyday questions, search only for things that need up-to-date or multi-source verification. The card also notes that the Current date: line in that prompt should be updated dynamically. If you skip it, expect the model to over-research simple questions; copy it verbatim from the model card into your client's system message.

Results

Speed: Omitted deliberately, and the reason is worth stating precisely. /check/agents-a1-4b/m2-max has no benchmark data, and neither of the two public throughput figures for this model qualifies: wcamon's card reports its measurement on an M4 Pro, a different chip with different memory bandwidth, and the ToPo-ToPo card reports one without naming a chip at all. Token generation here is memory-bandwidth-bound, so a figure from another Apple part does not transfer, and an unnamed part cannot be checked. If you measure this on an M2 Max, please submit the run via /contribute and it will replace this paragraph with a real number.
Memory usage: Q8_0 weights are 4.17 GiB and the projector 0.63 GiB, both resident. The KV cache is the variable: 1.00 GiB at 32K, 4.00 GiB at 128K, 8.00 GiB at the full 262,144 tokens at fp16, or roughly half those with -ctk q8_0 -ctv q8_0. The lead command's ~12.80 GiB working set uses about a quarter of the M2 Max's ~48 GB addressable pool; F16 at the same context would be ~16.5 GiB, still comfortable. All file sizes are byte counts from the Hugging Face tree API for the linked repositories; the KV figures are derived from the published config.json, not measured.
Quality notes: InternScience reports BrowseComp 66.8, XBench-DS-2510 90.0, GAIA 95.1, FrontierScience-Research 33.3 and IFEval 94.8 for the 4B on the model card, positioning it against similarly-sized dense models and against 35B-A3B MoE systems. Those are the vendor's own evaluations, run through its open-sourced evaluation framework under vLLM or SGLang at the released BF16 precision — not independent results, not measured on this hardware, and not measured at Q8_0. Every one of them is a text benchmark; there are no published vision results. The card states no base model for this release, and while its config.json gives model_type: qwen3_5 and the GGUF header gives qwen35, an architecture string names an implementation class, not a lineage — so this recipe asserts no base model either.

For the full benchmark data, see /check/agents-a1-4b/m2-max.

Troubleshooting

Images are ignored, or the model insists it cannot see anything

Two causes, in order of likelihood.

You are running without a projector. Vision in llama.cpp requires the mmproj file alongside the language weights. -hf InternScience/Agents-A1-4B-Q8_0-GGUF fetches it automatically, but --no-mmproj disables it and a bare -m <file.gguf> without --mmproj loads text only. Confirm the server log mentions loading the multimodal projector; if not, pass --mmproj Agents-A1-4B-mmproj.gguf explicitly. If the projector is loaded but too slow, --no-mmproj-offload keeps it on the CPU — you will not want that on a Mac, where the GPU shares the same memory anyway.

You picked a text-only MLX build. wcamon/Agents-A1-4B-MLX-4bit contains no vision tower and no projector weights — 924 tensors, zero visual. Switch to one of the mlx-vlm-converted builds listed in Installation → Option B, or use the GGUF path.

MLX fails to load the model, or `mlx_lm` does not recognise `qwen3_5`

Use mlx-vlm ≥ 0.6.6 (pip install -U "mlx-vlm>=0.6.6"), not mlx-lm. The ToPo-ToPo card states that earlier mlx-vlm versions do not implement the qwen3_5 architecture and cannot load or convert this model, and mlx-lm covers only the text tower even where it does load. Note also that mlx-vlm's qwen3_5 config raises deliberately if a checkpoint declares non-empty deepstack_visual_indexes; that is not a concern for these weights, whose canonical vision_config.deepstack_visual_indexes is [] — and the first-party mmproj agrees, its clip.vision.is_deepstack_layers being 24 entries all set to false. Neither runtime is losing a feature to the other here.

Load fails, or output is gibberish, on an older llama.cpp

Update before investigating anything else. qwen35 and the qwen3vl_merger projector are both recent additions to llama.cpp; a build that predates either will refuse the file or produce nonsense. Use b10176 (2026-07-29) or newer, and check the startup log for the architecture line reading qwen35.

`-c 262144` allocates more than you wanted, or the machine gets sluggish

llama.cpp reserves the KV cache for the full -c value at load time, so -c 262144 claims 8.00 GiB up front whether or not your prompts are that long. Lower -c, or quantize the cache with -ctk q8_0 -ctv q8_0. If you genuinely need the full context and something else large resident at the same time, macOS's limit on how much unified memory the GPU may wire is the knob (macOS Sonoma 14 / Sequoia 15+):

# Only if you are running other large workloads alongside this one
sudo sysctl iogpu.wired_limit_mb=57344   # 56 GB on a 64 GB Mac; leaves ~8 GB for macOS

Nothing in this recipe needs that — the lead configuration sits at roughly a quarter of the default pool. The general form is total unified memory minus 8–16 GB, expressed in MB, so do not paste the value above on a smaller Mac: 56 GB exceeds a 48 GB machine's entire memory. The setting is temporary and resets on reboot (persist it via /etc/sysctl.conf); sudo sysctl iogpu.wired_limit_mb=0 restores the default. On macOS Monterey 12 / Ventura 13 the knob is sudo sysctl debug.iogpu.wired_limit=<bytes> instead. Watch Activity Monitor's Memory-Pressure gauge as you climb.

A guide told me to install CUDA, flash-attn, bitsandbytes, vLLM or SGLang

None of those are the Apple path. There is no CUDA on macOS, no FP8 or NVFP4 tensor cores, no FlashAttention wheel, no bitsandbytes GPU kernels, no GPTQ/AWQ/Marlin/ExLlamaV2, no xformers and no ROCm — llama.cpp uses its own Metal kernels with GGUF K-quants, and MLX uses its own. The vLLM and SGLang commands on the model card are real and are the only runtimes the vendor documents, but they target Linux CUDA serving, not a Mac. If a tutorial tells you to pass -DGGML_CUDA=ON, pip install flash-attn, or --load-in-4bit, skip that step entirely.

Looking for `ollama run agents-a1`?

It does not exist. ollama.com/library/agents-a1, /agents-a1-4b and /internscience/agents-a1 all return HTTP 404, so there is no library tag to pull and this recipe does not claim one. Use the llama.cpp or MLX paths above. InternScience's own card nods at the Mac — thanking the mlx-community for quantized versions, and closing with "Try running Agents-A1 on your Mac!" — but that note is dated 2026-07-02, twelve days before the 4B shipped, and the collection it points at contains only 35B builds. If an Ollama entry or a mlx-community 4B build appears later, please flag it through /contribute so this recipe can be updated.