GreenCube
Get GreenCube

GreenCube blog

LLM Quantization: 6 Step PTQ Workflow for Practitioners

LLM quantization stores a model's numbers with fewer bits, for example 4 or 8 instead of 16, so the model needs less memory and runs faster, at some cost in accuracy. Done well, the loss is small: 8-bit formats come close to the original, and 4-bit is the usual choice when a model has to fit on one GPU or a laptop. Most people use post-training quantization (PTQ), which takes minutes to hours, rather than retraining the model.

Below is how it works in plain terms, the methods worth knowing, and a six-step workflow for quantizing a model yourself. If you only want to run a quantized model, not make one, skip to the section near the end about that.

What quantization does

A language model is billions of numbers, its weights, normally stored as 16-bit floating point. Each one takes two bytes, so a 7-billion-parameter model needs about 14 GB just for its weights. Quantization rounds every number onto a much smaller set of values. At 4 bits there are only 16 possible values per number, and the same model shrinks to roughly 4 GB once you add the small amount of extra data quantization needs.

It makes the model faster as well. To write each token, the model has to read all of its weights, so on most PCs speed is limited by how fast memory can feed the processor. Fewer bytes to read means more tokens per second.

Scale and zero-point

The usual mapping is linear. You choose a scale (how much real value each integer step covers) and a zero-point (which integer stands for zero), store q = round(x / scale + zero_point), and recover an approximation with x ≈ scale × (q - zero_point). Symmetric quantization fixes the zero-point at zero. That's simpler and suits weights, which are spread fairly evenly around zero. Activations, the values passed between layers, are often lopsided, which is one reason they're harder to quantize.

Groups and outliers

One scale for a whole matrix is crude, so most 4-bit formats give each small group of weights its own scale. Groups of 128 are a common default; smaller groups are more accurate but store more scales.

The hard part is outliers. In large models a few channels carry values far bigger than the rest, and a naive quantizer either clips them or wastes most of its few levels covering them. The usual fixes are group-wise scales, keeping a few sensitive parts at higher precision, and non-uniform formats such as NF4 (introduced with QLoRA), which place their levels where weights actually cluster.

Not sure your PC is up to it? Check what runs on yours

Post-training quantization or quantization-aware training?

Post-training quantization takes a finished model and quantizes it, usually with a small calibration set and no training. It runs in minutes to a few hours on one GPU. Quantization-aware training (QAT) simulates quantization during training or fine-tuning so the model learns to cope with it, which needs a training pipeline, data and far more compute.

For most deployments PTQ is the right call. QAT earns its cost at very low bit widths (below 4 bits), on tasks that are unusually sensitive to precision, or when you're fine-tuning anyway and can fold it in.

The main PTQ methods

Which bit width for which hardware

One of the largest public comparisons is a 2025 study that quantized the Llama 3.1 family and tested it on A6000, A100 and H100 GPUs (Kurtic et al., ACL 2025). In short, FP8 weights and activations (W8A8-FP) were effectively lossless, INT8 weights and activations (W8A8-INT) lost about 1 to 3 percent average accuracy, and 4-bit weights with 16-bit activations (W4A16) held up well and was the most cost-efficient for serving one request at a time. W8A8 did best for busy, batched serving.

Your hardware decides what's practical. NVIDIA's Ampere generation (A100, RTX 30 series) handles INT8 well but has no native FP8; Ada Lovelace (RTX 40 series) and Hopper (H100) add it. A format with no fast kernel on your hardware is no use, so check what your inference engine supports first. Taking activations down to 4 bits as well is riskier: a 2023 study found that decoder-only models, which is what chat models are, lose noticeably more accuracy at full INT4 than encoder models do (Wu et al., 2023).

Long chats add one more cost. The KV cache, which holds the conversation so far, grows with context length and can take as much memory as the weights. Many engines can store it in 8 bits too, but validate your weight format first, since each lossy step adds error.

On a home PC: GGUF and "Q4_K_M"

If you run models with llama.cpp, Ollama or LM Studio, you'll meet quantization in GGUF file names. Q8_0 is 8-bit. Q5_K_M and Q4_K_M are 5- and 4-bit "k-quants", which give small blocks of weights their own scales and keep some sensitive tensors at higher precision. Q4_K_M is the common default balance of size and quality; move up to Q5 or Q8 if you have memory to spare. llama.cpp can also use an "importance matrix" built from sample text, which does the job of a calibration set. More on the format in GGUF vs GGML.

Calibration data

Methods like GPTQ and AWQ look at sample inputs to decide what to protect, so the samples matter more than their number. Use text like what the model will see in use (real chat turns for an assistant, real code for a coding model) in the right languages and a spread of lengths. A few hundred sequences, 128 to 512 being a common range, is usually enough for 4-bit weight-only quantization. Random text is a poor substitute, especially for code and maths.

A six-step PTQ workflow

  1. Start from the original BF16 or FP16 checkpoint and check that it reproduces its published scores on a small held-out set. That's your baseline.
  2. Decide what "good enough" means (which metric, and how big a drop you'll accept) and which hardware the model will run on.
  3. Pick a method: GPTQ or AWQ for 4-bit weight-only, SmoothQuant or FP8 for 8-bit weights and activations, k-quants for llama.cpp.
  4. Build a calibration set that matches your real traffic.
  5. Quantize with a maintained tool, such as llm-compressor, GPTQModel or llama.cpp's own quantize tool, and keep the per-layer error logs.
  6. Evaluate against the baseline, then roll out to a small share of traffic before everyone else.

Catching regressions

Perplexity is a quick sanity check, but it can look fine while specific skills get worse. Add tests that match your use, such as HumanEval for code or a set of maths problems, and read real answers side by side with the original model's. Quantization damage usually shows up as repetition loops, small factual slips or broken code rather than gibberish. Small models tend to suffer more than large ones at the same bit width, so test them harder.

When something regresses you rarely need to start over. Try a smaller group size, then recalibrate with data closer to the failing cases, then keep the worst layers at higher precision. Switch methods or raise the bit width only if those don't help.

If you just want to run a quantized model

You don't need any of this to use one. Ollama and LM Studio download models that are already quantized, and you choose the size. GreenCube, the app I build, goes a step further and picks the model for your PC: Cube 2B (built on Gemma 4 E2B) on smaller PCs and Cube 4B (built on Gemma 4 E4B) on PCs with a graphics card. It's a Windows app that costs €9.99 once and works offline after the first download. If you'd rather choose by hand, see the best small LLMs and how much VRAM you need.

Questions

What is LLM model quantization?

Storing a model's weights, and sometimes its activations, in fewer bits (8 or 4 instead of 16), so it's smaller and faster at some cost in accuracy.

What is the best quantization method for LLMs?

It depends on your hardware. On GPUs with FP8 support, 8-bit FP8 is close to lossless; for tight memory, 4-bit weight-only GPTQ or AWQ is the standard, which on a home PC usually means a Q4_K_M or Q5_K_M GGUF file.

When should you quantize a model?

When memory, speed or cost stops you running it, like fitting a model on one consumer GPU or a laptop. If you have plenty of all three, quantizing adds risk for little gain.

How do I quantize a model?

Measure the full-precision model first, decide how much drop you'll accept, and pick a method. Then quantize with a maintained tool like llm-compressor or llama.cpp's quantize tool, using a calibration set that looks like your real traffic, and compare with the original before you ship.

Click around GreenCube right here

This is the app’s own window. Send one of the prompts, open what it makes and change it by chatting, or make an image in Images.

Good evening.

What do you want to make?

Message GreenCube…
Cube 2B
On your computer · Cube 2B
SlidesCauses of World War I

Example answers to give you a feel; nothing here runs AI. In the app, Cube 2B writes its own on your PC, offline.

Will GreenCube run on your PC?

  • Seed8 GB of RAM, no graphics cardEverything works, just slower.
  • Sprout16 GB of RAM, or 8 GB with a cardSteady speed, writes slide decks.
  • BloomA graphics card with 8 GB, or a 4-6 GB card with 16 GB of RAMFast, pictures with no waiting.
  • ThriveA card with 8 GB or more and 24 GB of RAM or moreThe fastest.

GreenCube looks at your PC and sets up the AI model that fits it: no terminal, no settings. €9.99 (US$9.99), paid once, and it's yours.

Check your PC Get GreenCube