Local LLMs are a VRAM problem, not a model problem
On this page
Most guides to running local models start by comparing models. That is the wrong end of the problem. On your own hardware the question is not which model is best, it is which models fit, and everything else follows from that.
The rule that matters: a model that does not fit entirely in memory is not slow, it is unusable. Spill even a few layers to system RAM and throughput falls off a cliff. There is no gentle degradation.
The arithmetic
At full precision, budget roughly 2 GB of VRAM per billion parameters. Quantisation cuts that down predictably, because it is just storing each weight in fewer bits.
Ad space, reserved
| Precision | Bytes per parameter | An 8B model needs |
|---|---|---|
| FP16 | ~2 | ~16 GB |
| Q8 | ~1 | ~8 GB |
| Q4_K_M | ~0.5 | ~4.5 GB |
Add roughly 1 to 2 GB on top for the context window and general overhead. So an 8B model at Q4_K_M wants about 6 GB of headroom in practice, which is why 8 GB cards handle it comfortably and 6 GB cards do not quite.
Q4_K_M is the default in Ollama for a reason. It is the point where the memory saving is large and the quality loss is small enough that most people cannot pick it out in normal use.
What fits where
| VRAM | Realistic ceiling | What that gets you |
|---|---|---|
| 8 GB | 7B to 8B at Q4 | Summarising, classification, structured extraction. The workhorse tier. |
| 12 GB | 8B comfortably, 13B at Q4 | Same jobs with a much longer context window. |
| 16 to 24 GB | 30B class at Q4 | Noticeably better instruction following and longer reasoning chains. |
| 48 GB | 70B at Q4 | Approaching hosted-model quality, at workstation cost. |
The step everyone wants to skip is 8 GB to 24 GB, and it is worth being blunt: for the jobs a self-hoster actually automates, the 8B tier is usually enough. Summarising a transcript, extracting fields into JSON, classifying an inbox, drafting a first pass. None of those are reasoning-heavy, and a 70B model does them slightly better at several times the hardware cost.
Check what is actually happening
Ollama will silently split a model across GPU and CPU rather than refusing to load it. That is helpful and it is also how people end up concluding local models are hopeless. Check the split before you judge the speed.
# What is loaded, and where
ollama ps
# PROCESSOR column tells you the truth:
# "100% GPU" -> good
# "48%/52% CPU/GPU" -> this is why it is slowIf you see a CPU share, you have three options: a smaller model, a heavier quantisation, or a shorter context window. Context is the one people forget, and it is often the cheapest fix, because the KV cache grows with it.
# Pin the context to something sane for the job
ollama run llama3.1:8b
>>> /set parameter num_ctx 4096
>>> /save llama31-8b-4kA 4096 token context is plenty for summarising a lesson transcript. Loading a 128k context you never use costs real memory for nothing.
Quantisation, honestly
Going from FP16 to Q8 is close to free in quality terms. Q8 to Q4_K_M is a small, real loss that shows up mostly in long chains of reasoning and in exact recall of details buried in a long input. Below Q4 the losses stop being subtle.
For structured extraction, which is most automation work, Q4_K_M is fine. The task is bounded, the output is schema-constrained, and errors are caught by validation rather than by the model being clever.
For anything where the model has to hold a lot of context and reason across it, the quantisation drop is more noticeable than the parameter count. A 13B at Q8 will often beat a 30B at Q3 on the same hardware budget.
When local is the wrong answer
I run models locally for client work because footage and transcripts should not leave my network. That is a real constraint and it justifies the hardware.
It is not automatically the right call otherwise. Local is worse when you need frontier-level quality, when your volume is low enough that hosted API costs are trivial, or when the work is bursty and a GPU would sit idle. Frontier model prices fell substantially through 2026, which moved that line further toward hosted than it was.
The honest test: if the only reason you are self-hosting is cost, do the arithmetic including the hardware and the electricity. If the reason is that the data cannot leave, or that you want it to keep working without a subscription, the arithmetic stops mattering and local wins on its own terms.
Where to start
- Install Ollama, pull an 8B instruct model at the default quantisation.
- Run
ollama pswhile it is working and confirm 100% GPU. - Set
num_ctxto the smallest value your job actually needs. - Give it one real task from your own work, not a benchmark prompt.
- Only then decide whether you need more hardware.
Most people who conclude local models are not good enough tested a model that was half on the CPU, with a context window ten times larger than the task needed. Fix those two things before you buy anything.
Sources
- Ollama FAQ, memory and GPU behaviour
- Ollama VRAM requirements, 2026 guide
- Best local LLMs by VRAM tier, 2026
Ad space, reserved
