GreenCube
Get GreenCube

GreenCube blog

Local AI Hardware That Actually Runs Models: Memory First 8GB–24GB Tiers

For local AI, buy memory first and speed second. What decides which models you can run is whether your graphics card's memory (VRAM), or a Mac's unified memory, can hold the model plus the conversation it's working on. Roughly: 8 GB runs small 7B-class models, 16 GB runs 13B to 20B models comfortably, 24 GB runs 30B-class models, and 70B models need two big cards or a Mac with a lot of memory. Small models also run with no graphics card at all, just more slowly.

The tiers

"Local AI" means very different things to someone summarising PDFs on a laptop and someone running a 32B coding model all day. These four tiers cover most people. All sizes assume 4-bit quantization, the usual way models are shrunk to run at home.

TierMemoryRuns wellExample cards
Starter8 GB VRAM, or a Mac with 16 GB7B-class models and smallerRTX 4060 8 GB, RTX 3060 12 GB
Everyday16 GB VRAM, 32 to 64 GB system RAM13B to 20B modelsRTX 4060 Ti 16 GB
Power user24 GB VRAM, 64 GB+ system RAM30B to 32B modelsRTX 4090, used RTX 3090
WorkstationTwo 24 GB+ cards, or 128 GB+ unified memory70B models and upMulti-GPU rigs, high-memory Mac Studio

A used 24 GB card is a sensible upgrade if you're serious, because memory, not raw speed, is what lets you run the next size up. Multi-GPU builds add real complexity (drivers, PCIe lanes, software that splits a model across cards), so most people are better off with one bigger card first.

Not sure your PC is up to it? Check what runs on yours

How much memory a model actually needs

Start with the weights. At 16-bit precision each parameter takes 2 bytes, so a 7B model needs about 14 GB just to load. At 8-bit that halves to about 7 GB, and at 4-bit (the common GGUF builds) it's about 4 GB.

Model size16-bit8-bit4-bit
7B~14 GB~7 GB~4 GB
13B~26 GB~13 GB~7 GB
32B~64 GB~32 GB~18 GB
70B~140 GB~70 GB~40 GB

Weights are the floor, not the total. The model also needs room for its working memory of the conversation (the KV cache), which grows with the length of the context: double the context and that part roughly doubles. That's why a model that fits fine on a short chat can run out of memory on a long document. A rough rule is to take the 4-bit weight size, add a third to a half for the cache, plus a couple of GB for the runtime. If the total is more than your VRAM, use a smaller quantization, a smaller model, or let part of it run from system RAM (slower). More in how much VRAM you need and LLM quantization.

Apple Silicon or NVIDIA

Macs with Apple Silicon share one pool of memory between the processor and the graphics, so a Mac with 64 GB can give much of it to a model, more than any consumer graphics card at that price. They're quiet and efficient. The limits: you can't add memory later, and the Mac tools (such as MLX) differ from the CUDA ecosystem most AI software targets first.

NVIDIA cards have the broadest software support, from llama.cpp to server tools like vLLM, and you can upgrade the card or add a second one later. AMD and Intel cards are cheaper per GB and work with llama.cpp, but support for the newest tools tends to arrive later. If you want one quiet box for chat and documents, a high-memory Mac is appealing. If you want to grow over time, a PC with an NVIDIA card gives you more room.

CPU, RAM and storage

Which software suits which machine

llama.cpp is the engine under much of local AI and the best option for machines without a card or with little memory. Ollama wraps it in simple commands and works on almost anything. LM Studio adds a graphical app for downloading and chatting with GGUF models (see Ollama vs LM Studio). vLLM is for NVIDIA workstations serving many requests at once, not for a single laptop.

When a model runs out of memory

Before buying hardware, try these in order: shorten the context length, use a smaller quantization (4-bit instead of 8-bit), move fewer layers onto the graphics card so the rest runs from system RAM, or pick a smaller model. Read past the first line of the error log; it usually says whether loading the model or allocating the context failed. Only if none of that works is it time for a bigger card.

If you don't want to do the maths

Plenty of people want an AI on their own PC without choosing cards or quantizations. That's the idea behind GreenCube, which I build, so weigh this with that in mind. It checks your Windows PC on first launch and picks the model that fits: Cube 2B on smaller PCs, Cube 4B on PCs with a graphics card, both built on Google's Gemma. Here's how fast it writes on the PCs we tested:

Your PCWrites
8 GB laptop, no graphics card5–7 tokens/s (about reading speed)
16 GB laptop, no graphics card14–18 tokens/s
Gaming PC with a 6 GB+ card37–76 tokens/s on most cards

Everything works on an 8 GB laptop, just slower; making images needs 12 GB of RAM. It costs €9.99 once and works offline after a one-time sign-in and download. The trade-off is the same as any small local model: it's much less capable than the big cloud AIs on the hardest problems. To check your own PC, try Can my PC run AI?

Questions

What hardware do I need for AI?

For small models, any recent PC with 8 GB of RAM, slowly. For mid-size models in comfort: a graphics card with 8 to 16 GB of VRAM (or a Mac with 16 GB or more), 32 GB of RAM and an NVMe SSD. Bigger models need 24 GB of VRAM or more.

What is the best AI tool to run locally?

It depends how hands-on you want to be. Ollama and LM Studio suit people happy to pick models, vLLM suits NVIDIA servers, and GreenCube suits people who want it set up for them.

Which CPU do I need for local AI?

With a graphics card, a mid-range 6 to 8 core CPU is enough. Without one, more cores and faster memory help, since the CPU does all the work.

Can I run an AI model locally without a dedicated GPU?

Yes, small quantized models run on the processor alone. On the laptops we tested, GreenCube wrote at about reading speed with 8 GB of RAM and two to three times faster with 16 GB.

How much does GreenCube cost?

€9.99 (US$9.99) once, everything included, no subscription. There's a 15-day refund if it doesn't suit your PC.

Click around GreenCube right here

This is the app’s own window. Send one of the prompts, open what it makes and change it by chatting, or make an image in Images.

Good evening.

What do you want to make?

Message GreenCube…
Cube 2B
On your computer · Cube 2B
SlidesCauses of World War I

Example answers to give you a feel; nothing here runs AI. In the app, Cube 2B writes its own on your PC, offline.

Will GreenCube run on your PC?

  • Seed8 GB of RAM, no graphics cardEverything works, just slower.
  • Sprout16 GB of RAM, or 8 GB with a cardSteady speed, writes slide decks.
  • BloomA graphics card with 8 GB, or a 4-6 GB card with 16 GB of RAMFast, pictures with no waiting.
  • ThriveA card with 8 GB or more and 24 GB of RAM or moreThe fastest.

GreenCube looks at your PC and sets up the AI model that fits it: no terminal, no settings. €9.99 (US$9.99), paid once, and it's yours.

Check your PC Get GreenCube