Local AI Hardware That Actually Runs Models: Memory First 8GB–24GB Tiers
For local AI, buy memory first and speed second. What decides which models you can run is whether your graphics card's memory (VRAM), or a Mac's unified memory, can hold the model plus the conversation it's working on. Roughly: 8 GB runs small 7B-class models, 16 GB runs 13B to 20B models comfortably, 24 GB runs 30B-class models, and 70B models need two big cards or a Mac with a lot of memory. Small models also run with no graphics card at all, just more slowly.
The tiers
"Local AI" means very different things to someone summarising PDFs on a laptop and someone running a 32B coding model all day. These four tiers cover most people. All sizes assume 4-bit quantization, the usual way models are shrunk to run at home.
| Tier | Memory | Runs well | Example cards |
|---|---|---|---|
| Starter | 8 GB VRAM, or a Mac with 16 GB | 7B-class models and smaller | RTX 4060 8 GB, RTX 3060 12 GB |
| Everyday | 16 GB VRAM, 32 to 64 GB system RAM | 13B to 20B models | RTX 4060 Ti 16 GB |
| Power user | 24 GB VRAM, 64 GB+ system RAM | 30B to 32B models | RTX 4090, used RTX 3090 |
| Workstation | Two 24 GB+ cards, or 128 GB+ unified memory | 70B models and up | Multi-GPU rigs, high-memory Mac Studio |
A used 24 GB card is a sensible upgrade if you're serious, because memory, not raw speed, is what lets you run the next size up. Multi-GPU builds add real complexity (drivers, PCIe lanes, software that splits a model across cards), so most people are better off with one bigger card first.
Not sure your PC is up to it? Check what runs on yours
How much memory a model actually needs
Start with the weights. At 16-bit precision each parameter takes 2 bytes, so a 7B model needs about 14 GB just to load. At 8-bit that halves to about 7 GB, and at 4-bit (the common GGUF builds) it's about 4 GB.
| Model size | 16-bit | 8-bit | 4-bit |
|---|---|---|---|
| 7B | ~14 GB | ~7 GB | ~4 GB |
| 13B | ~26 GB | ~13 GB | ~7 GB |
| 32B | ~64 GB | ~32 GB | ~18 GB |
| 70B | ~140 GB | ~70 GB | ~40 GB |
Weights are the floor, not the total. The model also needs room for its working memory of the conversation (the KV cache), which grows with the length of the context: double the context and that part roughly doubles. That's why a model that fits fine on a short chat can run out of memory on a long document. A rough rule is to take the 4-bit weight size, add a third to a half for the cache, plus a couple of GB for the runtime. If the total is more than your VRAM, use a smaller quantization, a smaller model, or let part of it run from system RAM (slower). More in how much VRAM you need and LLM quantization.
Apple Silicon or NVIDIA
Macs with Apple Silicon share one pool of memory between the processor and the graphics, so a Mac with 64 GB can give much of it to a model, more than any consumer graphics card at that price. They're quiet and efficient. The limits: you can't add memory later, and the Mac tools (such as MLX) differ from the CUDA ecosystem most AI software targets first.
NVIDIA cards have the broadest software support, from llama.cpp to server tools like vLLM, and you can upgrade the card or add a second one later. AMD and Intel cards are cheaper per GB and work with llama.cpp, but support for the newest tools tends to arrive later. If you want one quiet box for chat and documents, a high-memory Mac is appealing. If you want to grow over time, a PC with an NVIDIA card gives you more room.
CPU, RAM and storage
- System RAM: about double your VRAM is comfortable, so 32 GB or more next to a 16 GB card. With less, anything that spills out of the card ends up swapping.
- CPU: if a graphics card does the work, a mid-range 6 to 8 core processor is plenty. Without a card, the CPU and especially memory bandwidth decide your speed.
- Storage: a fast NVMe SSD with room for several models. Files run from a few GB to 40 GB each, and a slow drive makes every model switch slow.
- Power and cooling: AI runs keep the card busy for long stretches, unlike gaming. An undersized power supply or poor airflow shows up as random crashes or slowdowns that look like software bugs.
Which software suits which machine
llama.cpp is the engine under much of local AI and the best option for machines without a card or with little memory. Ollama wraps it in simple commands and works on almost anything. LM Studio adds a graphical app for downloading and chatting with GGUF models (see Ollama vs LM Studio). vLLM is for NVIDIA workstations serving many requests at once, not for a single laptop.
When a model runs out of memory
Before buying hardware, try these in order: shorten the context length, use a smaller quantization (4-bit instead of 8-bit), move fewer layers onto the graphics card so the rest runs from system RAM, or pick a smaller model. Read past the first line of the error log; it usually says whether loading the model or allocating the context failed. Only if none of that works is it time for a bigger card.
If you don't want to do the maths
Plenty of people want an AI on their own PC without choosing cards or quantizations. That's the idea behind GreenCube, which I build, so weigh this with that in mind. It checks your Windows PC on first launch and picks the model that fits: Cube 2B on smaller PCs, Cube 4B on PCs with a graphics card, both built on Google's Gemma. Here's how fast it writes on the PCs we tested:
| Your PC | Writes |
|---|---|
| 8 GB laptop, no graphics card | 5–7 tokens/s (about reading speed) |
| 16 GB laptop, no graphics card | 14–18 tokens/s |
| Gaming PC with a 6 GB+ card | 37–76 tokens/s on most cards |
Everything works on an 8 GB laptop, just slower; making images needs 12 GB of RAM. It costs €9.99 once and works offline after a one-time sign-in and download. The trade-off is the same as any small local model: it's much less capable than the big cloud AIs on the hardest problems. To check your own PC, try Can my PC run AI?
Questions
What hardware do I need for AI?
For small models, any recent PC with 8 GB of RAM, slowly. For mid-size models in comfort: a graphics card with 8 to 16 GB of VRAM (or a Mac with 16 GB or more), 32 GB of RAM and an NVMe SSD. Bigger models need 24 GB of VRAM or more.
What is the best AI tool to run locally?
It depends how hands-on you want to be. Ollama and LM Studio suit people happy to pick models, vLLM suits NVIDIA servers, and GreenCube suits people who want it set up for them.
Which CPU do I need for local AI?
With a graphics card, a mid-range 6 to 8 core CPU is enough. Without one, more cores and faster memory help, since the CPU does all the work.
Can I run an AI model locally without a dedicated GPU?
Yes, small quantized models run on the processor alone. On the laptops we tested, GreenCube wrote at about reading speed with 8 GB of RAM and two to three times faster with 16 GB.
How much does GreenCube cost?
€9.99 (US$9.99) once, everything included, no subscription. There's a 15-day refund if it doesn't suit your PC.