Can I Run This LLM?
Pick your GPU (or type your specs) and see which open models actually fit — before you download 40 GB to find out the hard way.
| Model | Params | FP16 | Q8 | Q4 | Best you can run |
|---|
Legend: GPU fits fully in VRAM (fast) · offload fits in VRAM+RAM via CPU offload (slow but works) · no doesn't fit. Estimates include the model weights plus KV cache at your chosen context; actual usage varies by runtime (llama.cpp, Ollama, LM Studio, vLLM).
How the math works
A model's memory footprint is roughly parameter count × bytes per parameter, plus KV-cache overhead that grows with context length:
- FP16 (full precision as shipped): ~2.0 GB per billion parameters
- Q8 (8-bit quantization, near-lossless): ~1.1 GB per billion
- Q4 (4-bit, e.g. Q4_K_M — the sweet spot for local use): ~0.6 GB per billion
So a 70B model needs ~140 GB at FP16 — but only ~42 GB at Q4, which is why quantization is what makes local LLMs possible on consumer hardware at all. Quality loss from Q8 is negligible; Q4 is usually a fair trade; below Q4 degradation gets noticeable.
Mixture-of-experts (MoE) models like Mixtral or GPT-OSS-120B are sized here by their total parameters (that's what must sit in memory) even though only a fraction is active per token — which is why they run faster than their size suggests, but still need the RAM of a big model.
If a model doesn't fit
- Drop one quantization level (Q8 → Q4).
- Reduce context length — KV cache at 64K can add many GB.
- Use partial CPU offload (works in llama.cpp / Ollama / LM Studio out of the box) — expect a large speed penalty once a meaningful share of layers leaves the GPU.
- Pick the smaller sibling: a 32B that fits fully in VRAM usually beats a 70B crawling on CPU offload.
FAQ
Can I run Llama 70B on a 24 GB GPU (RTX 4090/3090)?
Not fully in VRAM: 70B at Q4 needs ~42 GB plus KV cache. With CPU offload and 64 GB of system RAM it runs, but expect a low single-digit tokens/second. On 24 GB the practical ceiling for fully-GPU inference is roughly a 32B model at Q4/Q5.
Do Apple Silicon Macs work differently?
Yes — unified memory means the "VRAM" is your total RAM (minus what the OS is using). A 64 GB M-series Mac can comfortably run 70B-class models at Q4. Pick the closest option in the GPU dropdown or enter your RAM as VRAM.
Is Q4 quality actually acceptable?
For chat, coding assistance and summarization, Q4_K_M is widely used and the gap versus FP16 is small. For tasks that are sensitive to subtle reasoning errors, prefer Q8 or a smaller model at higher precision.
How much speed do I lose with CPU offload?
It scales with the share of layers on CPU: offloading a few layers costs a little, offloading half the model typically lands you at 1–5 tokens/second — usable for patience-tolerant tasks, painful for chat.