Visit Website →
← Back to blog

Local AI Hardware That Actually Runs Models: Memory First 8GB–24GB Tiers

September 26, 2026
Local AI Hardware That Actually Runs Models: Memory First 8GB–24GB Tiers

If you plan to run AI locally, buy for memory first and clock speed second. Every use case, from a lightweight 7B chatbot to a 70B coding assistant, comes down to whether your GPU's VRAM or your Mac's unified memory can hold the model plus its working data. Starter setups need 8GB, everyday use wants 16GB, serious hobbyists should target 24GB or more, and workstation-class work means multi-GPU rigs or a high-memory Apple Silicon Mac. Apple Silicon offers a simpler unified-memory path; NVIDIA offers an expandable, upgradeable one.

TL;DR:

  • Running larger models requires at least 16GB of VRAM for 13B models and 24GB or more for 30B-class models, with multi-GPU setups or high-memory Apple Silicon Macs for the biggest workloads.
  • Memory headroom must account for KV cache, activations, and runtime overhead; doubling the model's weight size provides a practical estimate for GPU requirements.
  • Apple Silicon offers simplified unified memory, suitable for smaller models and lighter workloads, while NVIDIA's ecosystem enables more expandability and multi-GPU scaling.
  • To prevent out-of-memory errors, reduce context length, lower precision, apply quantization, and manage memory fragmentation before upgrading hardware.
GreenCube
Try Private AI on Windows
GreenCube runs AI models entirely on your Windows PC, with offline chats and no subscription, using a one-time purchase.
Visit GreenCube

Table of Contents

Hardware for Local AI: Picks by Tier and Budget

The right hardware for local ai depends entirely on what "local AI" means to you. Someone summarizing PDFs on a laptop has completely different requirements than someone fine-tuning a 32B model overnight. Four tiers cover almost everyone.

Starter tier targets 7B and smaller models at 4-bit quantization. You need 8GB of VRAM (an RTX 3060 12GB or RTX 4060 8GB works fine) or a Mac with 16GB of unified memory. This tier runs quantized 7B chat models, basic coding assistants, and short-context summarization at usable speeds. It's also the tier where running local AI on older laptops with 2 to 5GB of free memory becomes realistic, since a well-chosen small model doesn't need a dedicated GPU at all.

Everyday tier is the practical sweet spot for most enthusiasts. A 16GB VRAM card, such as an RTX 4060 Ti 16GB or RTX 4070 Super, handles 13B to 20B models comfortably at 4-bit quantization, according to community hardware benchmarking in the Local AI Hardware Guide. Pair it with 32 to 64GB of system RAM and a 1TB NVMe drive, since model files alone can run 8 to 15GB each and you'll want room for several.

Power user tier moves to Around 24GB of VRAM is a practical threshold for running 30B and 32B-class models at 4-bit precision. Cards like the RTX 4090 or used RTX 3090 commonly offer this capacity, and the used 24GB card market has become a legitimately smart upgrade path rather than a compromise. Retailer roundups now treat used 24GB cards as a standard recommendation for anyone serious about local inference on a budget. Pair this tier with 64 to 128GB system RAM.

Workstation tier is for 70B-plus models, heavy image or video generation, or multi-model pipelines. Here you're looking at either multiple 24GB+ GPUs with tensor parallelism, or a high-memory Apple Silicon Mac Studio with 128GB, 256GB, or larger unified memory pools.

  1. Starter: 8GB VRAM or 16GB unified memory, 7B models
  2. Everyday: 16GB VRAM, 32 to 64GB system RAM, 13B to 20B models
  3. Power user: 24GB VRAM, 64 to 128GB system RAM, 30B to 32B models
  4. Workstation: multi-GPU or 128GB+ unified memory, 70B+ models

Multi-GPU builds add real complexity: you need matching drivers, adequate PCIe lanes, and software that supports tensor parallelism. Before committing to that, most people are better served by a single used 24GB card or a memory upgrade on their current rig.

How Much VRAM Does a Model Actually Need?

VRAM math starts with a simple rule: model weights are the floor, not the ceiling. A model's file size at a given precision tells you the minimum memory to hold it, but you also need room for the KV cache, activations, and runtime overhead.

Here's the arithmetic. At FP16 (2 bytes per parameter), a 7B model needs roughly 14GB just for weights. Drop to 8-bit quantization and that halves to about 7GB. Drop to 4-bit (common in GGUF or AWQ formats) and you're down to roughly 3.5 to 4GB. The same math scales linearly: a 13B model at 4-bit lands around 7 to 8GB, a 32B model around 18 to 20GB, and a 70B model around 38 to 40GB.

Weights alone aren't the whole story. NVIDIA's own troubleshooting documentation for out-of-memory errors treats the model file size as a baseline and instructs developers to add headroom for KV cache proportional to context length, plus activations and a runtime reserve.

Statistic callout: Community hardware guides converge on tiered targets: 8GB VRAM comfortably runs small 7B models, 16GB is the practical minimum for 13B models with headroom, and 24GB or more is recommended for 30B-class models, according to the Local AI Hardware Guide.

Longer context windows change this math fast. Doubling your context length roughly doubles your KV cache footprint, which is why a model that fits comfortably at 4K tokens can throw an out-of-memory error at 32K tokens on the same card. If you're planning to work with long documents or extended conversations, budget extra VRAM headroom before you buy, not after.

To quickly diagnose whether a GPU will fit a model: take the quantized weight size, multiply by 1.3 to 1.5 for typical KV cache and activation overhead, then add 2GB for runtime. If that number exceeds your VRAM, your options are a smaller quantization level, a smaller model, or splitting the model across multiple GPUs.

How Much VRAM Does a Model Actually Need? — overview diagram

Apple Silicon or NVIDIA: Which Fits Your Workflow?

The core tradeoff is unified memory versus discrete VRAM, and neither one wins outright. It depends on whether you value simplicity or expandability more.

Apple Silicon Macs pool RAM and GPU memory into a single unified space, meaning a Mac with 64GB of unified memory can technically use much of that for a large model, something no consumer discrete GPU at that price point can match. Apple's Core AI developer materials recommend smaller, task-specific models for mobile-class Apple devices, but larger unified-memory Macs can run substantially bigger models by pulling in the CPU, GPU, and Neural Engine together. The newest M6 and M5 Ultra chips push this further, with larger unified memory pools and higher memory bandwidth that meaningfully expand which local models a Mac Studio can handle at usable speeds.

  • Apple Silicon strengths: unified memory scales with total RAM, quiet and energy-efficient operation, no separate GPU purchase needed.
  • Apple Silicon limits: maximum unified memory caps out per chip generation, and macOS toolchains (MLX, Core AI conversions) differ from the CUDA ecosystem most research code targets first.
  • NVIDIA strengths: the CUDA ecosystem has the broadest tooling for quantization, multi-GPU tensor parallelism, and production-grade inference servers.
  • AMD/Intel considerations: worth considering for cost-conscious Linux builds, though software compatibility lags CUDA for cutting-edge quantization formats.

If you want a single quiet machine that "just works" for chat and document tasks, an Apple Silicon Mac with generous unified memory is genuinely compelling. If you want to scale up over time, add a second card, experiment with different quantization formats, or run inference servers, NVIDIA's ecosystem gives you more room to grow without replacing the whole machine.

What CPU, RAM, and Storage Actually Support a GPU?

Your GPU or unified memory gets the attention, but a mismatched CPU, RAM, or storage setup will bottleneck it anyway. Here's what actually matters.

  1. System RAM: aim for roughly double your VRAM tier. A 16GB GPU pairs well with 32 to 64GB system RAM; a 24GB GPU wants 64GB minimum. RAM below VRAM capacity means the OS and any CPU-offloaded layers start swapping, and that swap traffic runs at a fraction of GPU memory bandwidth.
  2. CPU: for GPU-based inference, a mid-range 6 to 8 core CPU is plenty since the GPU does the heavy lifting. If you're running CPU-only inference with llama.cpp, prioritize core count and memory bandwidth instead.
  3. Storage: a 1TB NVMe SSD is the practical minimum. Model files run 4 to 40GB each, quantized variants multiply that, and slow storage directly delays model load times every time you switch models.
  4. PSU: budget 650 to 850 watts for a single high-end GPU build, and 1000 to 1200 watts if you're running dual GPUs. Undersized PSUs cause random crashes under sustained load, which is easy to misdiagnose as a software bug.
  5. Cooling: sustained inference workloads run GPUs at high utilization for minutes or hours at a time, unlike gaming's bursty loads. Case airflow with at least two intake and one exhaust fan keeps thermal throttling from silently cutting your throughput.

Pro Tip: Monitor GPU temperature and power draw during your first long inference session with a tool like HWiNFO or nvidia-smi. If temperatures climb past 80°C under sustained load, your case airflow needs work before you blame the model or the software.

Which Quantization Level Fits Your GPU?

Quantization is the single biggest lever for fitting bigger models on smaller hardware, and it matters more than almost any other buying decision. NVIDIA's NeMo documentation confirms that formats like FP8, INT8, and INT4 reduce model memory footprints and let larger models run on smaller GPUs with minimal quality loss when properly calibrated.

Two approaches get you there. Post-training quantization (PTQ) compresses an already-trained model, is fast to apply, and works well for most local use cases. Quantization-aware training (QAT) bakes precision reduction into the training process itself, which preserves more accuracy but requires access to training infrastructure most local users don't have. For running existing open models, PTQ is what you'll use almost every time.

Statistic callout: NeMo's model optimization documentation shows that INT4 with proper PTQ calibration can significantly shrink model memory while keeping validation loss close to the original full-precision model.

In practice, here's what common model sizes look like across precisions: a 7B model runs about 14GB at FP16, 7GB at INT8, and 4GB at INT4 or Q4. A 13B model runs roughly 26GB, 13GB, and 7GB across the same three precisions. A 32B model spans about 64GB, 32GB, and 18GB. A 70B model spans roughly 140GB, 70GB, and 40GB. That last number is why even power users with a single 24GB card need aggressive quantization or a multi-GPU split to touch 70B models at all. When one card can't hold the quantized weights plus KV cache, tensor parallelism across two or more GPUs becomes the only path forward, short of dropping to a smaller model.

Model memory requirements by precision

Which Local AI Tools Work Best on Your Hardware?

Software choice should follow hardware, not the other way around. Each tool has a natural fit.

  • Ollama wraps model management and serving into simple commands, and it runs well on nearly anything from a modest laptop to a multi-GPU workstation. It's the easiest starting point for learning how local AI works without wrestling with configuration files.
  • LM Studio gives you a graphical interface for downloading and chatting with GGUF models, which suits users who don't want a terminal.
  • vLLM is built for server-grade throughput and batching, best suited to workstation-tier NVIDIA setups running multiple concurrent requests.
  • NeMo and TensorRT provide NVIDIA-specific optimizations like the FP8/INT8/INT4 quantization paths already discussed, aimed at squeezing maximum throughput from CUDA hardware.
  • llama.cpp is the engine underneath much of the ecosystem and remains the best option for CPU-only or memory-constrained machines.

On format compatibility: GGUF is the preferred format on Apple Silicon and CPU-only setups, while AWQ and safetensors, or TensorRT-exported models, tend to run faster on NVIDIA hardware. Before buying anything, confirm your GPU's CUDA version matches what your chosen runtime expects, since driver mismatches cause more failed setups than actual hardware shortfalls.

What Do Real Local AI Builds Cost and Run?

Numbers matter more than specs on paper, so here's what four real budget tiers look like when you shop for them.

  1. Starter build (roughly $600 to $900 total): an RTX 4060 8GB or a Mac mini with 16GB unified memory, 32GB system RAM, 1TB NVMe. This runs quantized 7B models at solid conversational speed, handles document summarization, and covers basic coding help. Power draw stays modest, around 150 to 200 watts under load.
  2. Everyday build (roughly $1,200 to $1,600): RTX 4070 Super or RTX 4060 Ti 16GB, 64GB system RAM, 1 to 2TB NVMe. This comfortably handles 13B to 20B models at 4-bit quantization with room for multitasking. See which models fit which hardware tier for specific model recommendations at this level.
  3. Enthusiast build (roughly $1,800 to $2,400, less with a used card): RTX 4090 or used RTX 3090 with 24GB VRAM, 64 to 128GB system RAM, 2TB NVMe. This tier runs 30B to 32B models well and handles longer contexts without immediately hitting memory walls. The used GPU market is a legitimate way to hit this tier for hundreds less than new pricing, though expect more fan noise and higher idle power draw from older cards.
  4. Workstation build (roughly $4,000 and up, or a high-memory Mac Studio): dual 24GB GPUs with tensor parallelism, or 128GB+ unified memory on Apple Silicon, for 70B-plus models and heavy image or video generation. Expect 500 to 800 watts sustained under dual-GPU load, and plan for a case with genuine airflow, not just a tower with fans bolted on.

Why Do Local Models Run Out of Memory, and How Do You Fix It?

Out-of-memory errors almost always trace back to one of three stages: loading the model weights, allocating the KV cache, or running activations during generation. NVIDIA's NIM troubleshooting guide points to a mismatch between model profile and available GPU memory as the most common root cause, and the backend logs usually name the exact stage that failed if you read past the first error line.

Before assuming you need new hardware, work through these fixes in order:

  • Lower the context length, since KV cache scales directly with it.
  • Drop to a lower precision (INT8 or INT4 instead of FP16).
  • Apply PTQ if you're currently running an unquantized model.
  • Set allocator options like PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce fragmentation.
  • Reduce the gpu-memory-utilization parameter if your runtime exposes it.
  • Disable CUDA graph capture, which reserves a fixed memory block upfront.

Pro Tip: Many OOM errors are fragmentation, not a genuine capacity shortfall. NVIDIA's own troubleshooting docs note that allocator settings alone frequently recover enough memory to load a model that otherwise appeared too large for the card.

Only after exhausting these should you consider a second GPU or tensor parallelism. If your workload genuinely exceeds what any local setup can handle, a hybrid approach with a cloud API for the largest jobs remains reasonable.

GreenCube: Local AI That Matches Your Hardware, Not the Other Way Around

Not everyone wants to build a rig around memory math. GreenCube is a private AI assistant that runs entirely on your own Windows or Mac computer, with no cloud and no server involved. At setup, you pick one of two local models: "Quick" (Llama 3.2 3B, about 2GB, fast, text only) for modest hardware, or "All-rounder" (Gemma 4 E4B, about 4.2GB, reads images and builds study guides, needs 8GB RAM) for machines with more headroom. Chats and files never leave your machine, and a one-time Google or Microsoft sign-in just verifies your license. If you want a working example of these principles before building your own rig, installing local AI with GreenCube's Quick model is a low-stakes way to feel the difference unified memory and VRAM make firsthand.

The Pragmatic Case for Buying Memory Before You Buy Speed

Local hardware makes the most sense when privacy matters, your workload is predictable, and you're willing to accept somewhat slower output than a frontier cloud model delivers. It makes less sense for genuinely massive reasoning tasks or one-off jobs that don't justify owning hardware at all. For those, a cloud API remains the more sensible tool, and there's no shame in mixing both.

If you're upgrading, buy incrementally: a used 24GB card or a RAM upgrade almost always beats jumping straight to a multi-GPU workstation. Pick a VRAM tier, download a small model first, and confirm it runs well before spending on anything bigger. For getting your workflow and instructions right on that first model, a resource like this guide on refining AI instructions is worth a read too.

— Hector Gras

Want Local AI Running Today Without Building a Rig?

GreenCube skips the memory math entirely. It's a one-time purchase with everything included, no subscription and no ongoing fees. It installs on Windows, downloads its model once during setup, and then works completely offline. If your computer is on the modest side, the Quick model (Llama 3.2 3B) handles fast text chat without needing a dedicated GPU. If you've got more RAM and unified memory to spare, the All-rounder model (Gemma 4 E4B) reads images and helps build study guides and documents, at a somewhat slower pace in exchange for more capability.

GreenCube

GreenCube's tiers simply describe what your particular computer handles well, not a paid plan structure. A slower machine will still answer questions and build documents, just more slowly than a newer one. Your chats and files stay on your computer the entire time, and only features you actively switch on ever touch the internet. A one-time Google or Microsoft sign-in unlocks the license, nothing more. If you'd rather try before deciding, start at the GreenCube landing page, or go straight to the GreenCube Lifetime purchase page to get set up in the next few minutes.

Sources

FAQ

What hardware do I need for AI?

For local AI, the essentials are a GPU with at least 8GB of VRAM (or a Mac with 16GB or more of unified memory), 32GB or more of system RAM, and a fast NVMe drive for model storage. The exact target scales with model size, and VRAM tiers map fairly predictably to model sizes, from 8GB for small 7B models up to 24GB or more for 30B-class models.

What is the best AI tool to run locally?

There's no single best tool since it depends on your hardware and comfort level. Ollama and LM Studio suit most people who want a simple setup, vLLM fits server-grade NVIDIA workstations, and GreenCube offers a turn-key desktop option with no configuration for readers who want offline privacy without managing model files themselves.

Which CPU do I need for local AI?

For GPU-based inference, a mid-range 6 to 8 core CPU is sufficient since the GPU handles the actual computation. If you're running models without a GPU, on a CPU-only setup with a tool like llama.cpp, prioritize core count and memory bandwidth instead, since the CPU becomes the primary bottleneck.

Can I run an AI model locally without a dedicated GPU?

Yes, smaller quantized models run acceptably on CPU-only systems or older laptops with as little as 2 to 5GB of free memory, though response speed will be noticeably slower than on a GPU. GreenCube's Quick model, built on Llama 3.2 3B at about 2GB, is designed specifically for this kind of modest hardware.

How much does GreenCube cost?

GreenCube is a one-time purchase for $9.99, with everything included and no subscription. You can buy it directly on the GreenCube Lifetime page.