Llama 3.1 8B
The default first local model: small enough for an 8 GB card at Q4, good enough to be genuinely useful, and the standard against which starter AI builds are sized.
Last verified 2026-07 · Page created; quant sizes verified against the official GGUF repos.
Why this is the starter model
Llama 3.1 8B is the model most people run first: at Q4_K_M the weights are under 5 GB, so it fits comfortably on an 8 GB GPU with room for context, and it streams faster than reading speed on anything mid-range or better.
It is also the honest baseline for sizing a first AI build — if a machine cannot run an 8B at Q4 briskly, it is not an AI machine. Step up in VRAM, not in hype: the jump that matters is to 16 GB (13–14B class and long contexts), then 24 GB (32B class).
VRAM by quantization
| Quant | File size | VRAM @ 8K ctx | VRAM @ 32K ctx | 8 GB | 16 GB | 24 GB | 32 GB | 48 GB |
|---|---|---|---|---|---|---|---|---|
| FP16 | 16.1 GB | 19.6 GB | 23.1 GB | Doesn't fit | Doesn't fit | Fits comfortably | Fits comfortably | Fits comfortably |
| Q8_0 | 8.5 GB | 11.0 GB | 14.4 GB | Doesn't fit | Fits comfortably | Fits comfortably | Fits comfortably | Fits comfortably |
| Q5_K_M | 5.7 GB | 7.7 GB | 11.2 GB | Fits comfortably | Fits comfortably | Fits comfortably | Fits comfortably | Fits comfortably |
| Q4_K_M | 4.9 GB | 6.8 GB | 10.3 GB | Fits comfortably | Fits comfortably | Fits comfortably | Fits comfortably | Fits comfortably |
Fit verdicts use the 8K context preset. Estimated from published weight sizes plus standard KV-cache math — see methodology.
Runs on
Unified-memory machines — Apple Silicon Mac Studio/MacBook Pro (Max/Ultra chips) and AMD Strix Halo mini-PCs — share system RAM and GPU memory in one pool, so a 64–128 GB configuration can hold models that would otherwise need multiple discrete GPUs. Throughput is bandwidth-bound like any GPU; it isn’t modeled in the table above yet.
Recommended builds
Three ways to size a machine for Llama 3.1 8B, from the smallest GPU that fits to the no-compromise option with headroom for the next model up.
Tokens/sec expectations
| GPU | Bandwidth | Est. tok/s (Q4_K_M) |
|---|---|---|
| RTX 5090 | 1792 GB/s | ~255 |
| RTX 4090 | 1008 GB/s | ~143.4 |
| RTX 6000 Ada | 960 GB/s | ~136.6 |
| RX 7900 XTX | 960 GB/s | ~136.6 |
| RTX 5080 | 960 GB/s | ~136.6 |
| RTX 5070 Ti | 896 GB/s | ~127.5 |
Estimated from memory bandwidth ÷ active weight size — a ceiling, not a measured figure. Real throughput depends on the inference engine, batch size, and context length.
Build it, or rent the compute?
Owning the hardware is a one-time cost that keeps paying off the more you actually use it — every run after the first is free, the data never leaves your machine, and the GPU still does everything else a PC does (gaming, editing, the rest of your workload) between AI sessions. Renting cloud GPU time trades that upfront cost for pay-as-you-go pricing with no idle hardware to maintain, which wins for occasional, bursty, or much-larger-than-consumer-VRAM jobs. For a model that fits comfortably in the VRAM tiers above and gets used regularly, owning the GPU tends to be the cheaper path over time — for anything larger than a single card can hold, renting is often the more practical starting point.
Frequently asked questions
Can I run Llama 3.1 8B on an 8 GB GPU?
Yes — Q4_K_M weights are about 4.9 GB, leaving room for a few thousand tokens of context on an 8 GB card. For the full 128K context or Q8 quality you want 16 GB or system-RAM offload at reduced speed.