Mistral Small 3
A 23.6B dense model built explicitly for low latency — Mistral aimed this one at "fast enough to feel instant" rather than maximum benchmark score, and it shows in how little VRAM it needs.
Last verified 2026-07 · Page created; Q4_K_M/Q5_K_M/Q8_0/FP16 sizes verified against bartowski/Mistral-Small-24B-Instruct-2501-GGUF.
What it is good at
Mistral Small 3 (internally the 24B-class "2501" checkpoint) was released in January 2025 explicitly optimized for latency — Mistral's stated goal was a model that responds fast enough for real-time use without a round trip to a datacenter, not a benchmark-maximizing flagship. Apache 2.0 licensed, no usage restrictions.
It is a solid fit for local agents and function-calling workflows where response speed compounds across many small calls, more than for long, deliberate reasoning sessions.
The hardware it wants
Q4_K_M is 14.33 GB, fitting a 16 GB GPU with a little headroom, and running very comfortably on any 24 GB card. Q5_K_M (16.76 GB) is the practical ceiling for a 16 GB card; Q8_0 (25.05 GB) needs 32 GB or two pooled 24 GB GPUs. This is one of the more forgiving models in this registry for mid-range hardware — a single 16 GB card (RTX 4060 Ti 16GB, RTX 5060 Ti 16GB) covers it at Q4.
VRAM by quantization
| Quant | File size | VRAM @ 8K ctx | VRAM @ 32K ctx | 8 GB | 16 GB | 24 GB | 32 GB | 48 GB |
|---|---|---|---|---|---|---|---|---|
| FP16 | 47.1 GB | 57.6 GB | 67.8 GB | Doesn't fit | Doesn't fit | Doesn't fit | Doesn't fit | Doesn't fit |
| Q8_0 | 25.1 GB | 32.2 GB | 42.4 GB | Doesn't fit | Doesn't fit | Doesn't fit | Tight — reduced context | Fits comfortably |
| Q5_K_M | 16.8 GB | 22.7 GB | 32.8 GB | Doesn't fit | Doesn't fit | Fits comfortably | Fits comfortably | Fits comfortably |
| Q4_K_M | 14.3 GB | 19.9 GB | 30.1 GB | Doesn't fit | Doesn't fit | Fits comfortably | Fits comfortably | Fits comfortably |
Fit verdicts use the 8K context preset. Estimated from published weight sizes plus standard KV-cache math — see methodology.
Runs on
Unified-memory machines — Apple Silicon Mac Studio/MacBook Pro (Max/Ultra chips) and AMD Strix Halo mini-PCs — share system RAM and GPU memory in one pool, so a 64–128 GB configuration can hold models that would otherwise need multiple discrete GPUs. Throughput is bandwidth-bound like any GPU; it isn’t modeled in the table above yet.
Recommended builds
Three ways to size a machine for Mistral Small 3, from the smallest GPU that fits to the no-compromise option with headroom for the next model up.
Tokens/sec expectations
| GPU | Bandwidth | Est. tok/s (Q4_K_M) |
|---|---|---|
| RTX 5090 | 1792 GB/s | ~87.5 |
| RTX 4090 | 1008 GB/s | ~49.2 |
| RTX 6000 Ada | 960 GB/s | ~46.9 |
| RX 7900 XTX | 960 GB/s | ~46.9 |
| RTX 5080 | 960 GB/s | ~46.9 |
| RTX 5070 Ti | 896 GB/s | ~43.8 |
Estimated from memory bandwidth ÷ active weight size — a ceiling, not a measured figure. Real throughput depends on the inference engine, batch size, and context length.
Build it, or rent the compute?
Owning the hardware is a one-time cost that keeps paying off the more you actually use it — every run after the first is free, the data never leaves your machine, and the GPU still does everything else a PC does (gaming, editing, the rest of your workload) between AI sessions. Renting cloud GPU time trades that upfront cost for pay-as-you-go pricing with no idle hardware to maintain, which wins for occasional, bursty, or much-larger-than-consumer-VRAM jobs. For a model that fits comfortably in the VRAM tiers above and gets used regularly, owning the GPU tends to be the cheaper path over time — for anything larger than a single card can hold, renting is often the more practical starting point.
Frequently asked questions
Can I run Mistral Small 3 on a 16 GB GPU?
Yes — Q4_K_M is 14.33 GB, fitting a 16 GB card with room for context, and Q5_K_M at 16.76 GB is close to the ceiling on the same card.
How is this different from Mistral Small 3.1?
This page covers the original January 2025 "2501" release. 3.1 added vision input and an expanded context window on top of the same size class — if you need image understanding, look at the 3.1 checkpoint specifically; VRAM needs are similar.
Is this a good model for local agent/tool-calling workflows?
Yes — it was tuned for low latency and supports function calling, which is exactly what benefits from fast responses when a workflow makes many small model calls in sequence.