What You'll Build
A from-scratch, end-to-end omni model running locally on the RTX 5060 Ti: MiniMind-O accepts text, speech, and image inputs and produces text plus streaming 24 kHz speech output. The dense 0.1B variant ships as ~1.7 GB of bfloat16 weights and is intended as an educational / research-grade reference implementation rather than a production-grade omni model.
Hardware data: RTX 5060 Ti (16GB VRAM) · 113M trainable parameters, bfloat16 inference · See benchmark data
Note: As of this writing the backend has no measured benchmarks for this pair (
/check/returnsverdict: unknown). Themin_vram_gb: 4figure is a conservative envelope derived from the official weight sizes and architecture (see the Results section); revisit/check/minimind-o/rtx-5060-tionce a community benchmark lands.
Requirements
| Component | Minimum | Tested |
|---|---|---|
| GPU | 4GB VRAM CUDA GPU | RTX 5060 Ti (16GB) — pair not yet benchmarked, see /check/ |
| RAM | 8GB | — |
| Storage | ~3 GB (code + all sub-model weights) | — |
| Software | Python 3.10, PyTorch with CUDA, Git | — |
Installation
1. Clone the official repo
git clone --depth 1 https://github.com/jingyaogong/minimind-o
cd minimind-o
2. Install Python dependencies
The repo pins Python==3.10. Create a fresh environment, then install the project requirements (the README uses the Tsinghua mirror for faster downloads in China — drop the -i flag if you're outside that region):
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
3. Download the model weights
MiniMind-O is an assembly of four sub-models — a 113M dense LLM trained from scratch plus three pre-trained encoders/decoders. Pull them all with modelscope (the official downloader for this project):
modelscope download --model gongjy/SenseVoiceSmall --local_dir ./model/SenseVoiceSmall
modelscope download --model gongjy/siglip2-base-p32-256-ve --local_dir ./model/siglip2-base-p32-256-ve
modelscope download --model gongjy/mimi --local_dir ./model/mimi
modelscope download --model gongjy/campplus --local_dir ./model/campplus
modelscope download --model gongjy/minimind-3o-pytorch llm_768.pth --local_dir ./out
The five files together total ~1.7 GB. SenseVoiceSmall handles speech input, SigLIP2 handles image input, Mimi is the streaming 24 kHz audio codec for speech output, CamPlus provides speaker embeddings, and llm_768.pth is the 113M-parameter language backbone trained from scratch on a small dataset by the project author.
Running
Run the omni inference script bundled with the repo:
python eval_omni.py --load_from model --weight sft_omni
This loads the SFT-tuned omni checkpoint and starts an interactive session that accepts text, speech, or image inputs and replies in text + 24 kHz audio. The model card on Hugging Face also exposes the dense checkpoint through standard transformers / vLLM / SGLang serving paths for text-only inference:
from transformers import pipeline
pipe = pipeline("text-generation", model="jingyaogong/minimind-3o", trust_remote_code=True)
pipe([{"role": "user", "content": "Who are you?"}])
Results
- Speed: Not yet measured on a 5060 Ti by any cited source; the project's documented training run fits on a single RTX 3090 (24GB) in ~2 hours per the official GitHub README, which strongly suggests inference latency on a 5060 Ti is interactive but is not a substitute for an empirical inference benchmark. See /check/ for live data.
- VRAM usage: Approximate envelope ~4 GB peak, derived from the official artifact sizes —
llm_768.pthis 138 MB andsft_omni_768.pthis 236 MB per the HF Files listing forjingyaogong/minimind-3o-pytorch. The architecture is 8 hidden layers × 768 hidden size × 32K max context in bfloat16 per theconfig.json, which keeps KV cache and decode buffers well under 1 GB for typical short conversations. Encoder/decoder bundles (SenseVoice, SigLIP2, Mimi, CamPlus) load alongside but at modest sizes — 4 GB is a conservative ceiling; expect real peak well below the 16 GB the 5060 Ti offers. - Quality notes: This is a from-scratch 113M-parameter omni model, deliberately small. Reported character error rate on English text-to-audio is
0.0897 avg CERper the GitHub README benchmark table — useful as a reference for what a tiny, hobbyist-trained omni stack achieves, not for comparison against frontier omni models like Gemini or Qwen3-Omni. The repo also ships a 0.3B MoE variant (minimind-3o-moe, 315M trainable) if you want to test a slightly larger configuration.
For the full benchmark data, see /check/minimind-o/rtx-5060-ti.
Troubleshooting
pip install is slow or fails on the Tsinghua mirror
The README defaults to a China-region PyPI mirror. Outside China, drop the -i https://pypi.tuna.tsinghua.edu.cn/simple flag and let pip use its default index:
pip install -r requirements.txt
ModelScope is unfamiliar — can I use Hugging Face instead?
Yes. The same checkpoints are mirrored on Hugging Face under jingyaogong/minimind-3o-pytorch for the raw .pth files and jingyaogong/minimind-3o for the from_pretrained-style packaging. Substitute huggingface-cli download jingyaogong/minimind-3o-pytorch llm_768.pth --local-dir ./out (and analogous calls for the encoders, which live under the gongjy/* mirrors) for the modelscope commands.
"What does 'omni' actually mean here?"
Three input modalities (text, speech, image) and two output modalities (text, streaming 24 kHz speech). Image output is not supported despite the "omni" branding — the project author describes the model as "能听能说能看" (capable of listening, speaking, and seeing) in the official README. If you need image generation as an output modality, this is not the right model.
No widely-reported issues yet
The HF discussions tab for jingyaogong/minimind-3o is empty as of this writing (model published ~12 days ago). Report problems via the submission form.