GreenCube
Get GreenCube

GreenCube blog

Measure First: Decide RAG vs Fine-Tuning in an Afternoon for Practitioners

Use RAG (retrieval-augmented generation) when the model needs to know things: facts that change often, private documents, answers that must cite a source. Use fine-tuning when the model needs to behave differently: a fixed format, a consistent tone, a narrow task done the same way every time. If you're not sure which problem you have, start with plain prompts, add RAG for the knowledge gaps, and fine-tune only what's still wrong after that.

Below is what each one does, how they compare, and an afternoon's worth of measuring that tells you which you need.

What RAG is

RAG fetches relevant text at the moment a question is asked and puts it into the prompt, so the model answers from that text. The model's weights never change. The idea comes from a 2020 paper that paired a language model with a searchable index of documents (Lewis et al.).

Most of the work is in the retrieval, not the model. A typical pipeline:

  1. Split your documents into chunks, small enough to retrieve precisely but big enough to keep their meaning.
  2. Turn each chunk into a vector with an embedding model, and store the vectors in a vector database (or use a keyword index, or both).
  3. When a question comes in, find the closest chunks, optionally reorder them with a reranker, and pass the best few to the model with their sources.

Each extra step can improve the results and also adds time to every question, so measure both.

Want one that sets itself up? GreenCube picks the right AI for your PC

What fine-tuning is

Fine-tuning changes the model's weights by training it further on examples of what you want. Instead of telling the model how to answer each time, you teach it once. AWS's guidance makes the same split: fine-tuning suits tasks like summarisation or consistent output, while question answering over changing documents is better served by starting with RAG (AWS Prescriptive Guidance).

Full fine-tuning updates every weight and needs the most data and compute. Parameter-efficient methods are the usual choice now. LoRA trains small add-on matrices and leaves the original weights frozen; QLoRA does the same on top of a quantized base model, so it fits in much less memory. They cost far less, and you can keep several adapters for one base model. Full fine-tuning still has a place for deep changes across many tasks, but it's more prone to overfitting your examples.

RAG vs fine-tuning at a glance

RAGFine-tuning
Best forCurrent, private or citable knowledgeConsistent behaviour, tone or format
Updating itAdd or swap documents, no retrainingA new training run
SourcesCan point to the passages it usedNo built-in trail
Data you needYour documentsCurated examples, often hundreds to thousands
Speed per questionSlower: retrieval runs every timeNo extra step once trained
Typical failuresWrong chunk retrieved, missing contextOverfitting, out-of-date knowledge

AWS's comparison notes that RAG can take in a new document in minutes, while fine-tuning takes hours to days depending on model size. For fast-changing knowledge, that alone usually decides it.

A decision checklist you can run in an afternoon

  1. Write a small test set, around 30 real prompts that include your hardest cases, and run them with plain prompting. That's your baseline.
  2. Read the failures and sort them: is the model missing knowledge, or behaving wrongly? Knowledge gaps point to RAG; format and tone problems point to fine-tuning.
  3. How often do the facts change? If weekly or daily, fine-tuning can't keep up.
  4. Must answers cite a source? RAG gives you that trail.
  5. Do you have good labelled examples? Without them, fine-tuning isn't an option yet.
  6. How tight is your latency budget? If retrieval makes answers too slow, fine-tuning avoids that step.
  7. Still unsure? Ship RAG, measure what's still failing, and fine-tune only that.

What to measure before you touch the weights

Retrieval and generation fail in different ways, so measure them separately. A decent overall score can hide a retrieval miss that the model is papering over with a plausible guess.

PartMeasureWhat it catches
RetrieverRecall@kThe right passage wasn't found at all
RerankerMean average precisionThe right passage was found but ranked too low
ModelGroundedness, or exact match against reference answersThe answer isn't supported by the retrieved text
Whole systemLatency and cost per questionWhether it's usable in practice

Keep every failed answer together with the passages it was given. Many "hallucinations" turn out to be retrieval misses. If you do fine-tune, rerun the same test set after every training run, since a retrain can quietly break something that used to work.

The staged workflow

  1. Ship the prompt-only baseline and score it.
  2. Add retrieval where it fails on missing or outdated knowledge, and score again.
  3. Look at what's left. If it's behaviour (wrong format, wrong tone, ignored instructions), that's a fine-tuning job.
  4. Fine-tune the smallest thing that fixes it, often a LoRA adapter or the embedding model used for retrieval, before considering a full retrain.

The two also combine well. In an AWS case study with Amazon Nova models, customising the model and adding RAG each improved answers, and using both together helped most, though results varied by dataset (AWS Machine Learning Blog).

When you need neither

If what you actually want is a private assistant that reads your own files, you may not need to build a pipeline at all. Local AI apps run a ready-made model on your computer and let you hand it documents directly; how local AI works explains the pieces. I build one, GreenCube, so weigh my opinion with that in mind: it reads your PDFs, Word files and photos on your PC, works offline after the first download, and costs €9.99 once for Windows. It won't match the big cloud models on the hardest problems, and it isn't a platform for building your own retrieval system. For a team that needs RAG over thousands of documents, the workflow above is still the way. For more on fitting long documents into a model, see context windows explained.

Questions

What makes a tune a RAG?

Nothing: RAG isn't a kind of tuning. It fetches relevant text when you ask and adds it to the prompt, and the model's weights don't change at all.

What is better than RAG?

Nothing, in general: RAG and fine-tuning fix different problems. RAG is usually better for adding knowledge, especially facts that change or are rarely seen, and fine-tuning for behaviour; many systems use both.

Is fine-tuning still relevant?

Yes. Methods like LoRA make it cheap enough for changing behaviour, like a fixed output format, a steady tone or a narrow task, where retrieval alone doesn't help.

Can I run a private AI without RAG or fine-tuning?

Yes. A local AI app runs a ready-made model on your computer, and you just give it your documents. GreenCube is one for Windows (€9.99 once): it reads PDFs, Word files and photos on your PC and works offline after the first download.

Click around GreenCube right here

This is the app’s own window. Send one of the prompts, open what it makes and change it by chatting, or make an image in Images.

Good evening.

What do you want to make?

Message GreenCube…
Cube 2B
On your computer · Cube 2B
SlidesCauses of World War I

Example answers to give you a feel; nothing here runs AI. In the app, Cube 2B writes its own on your PC, offline.

An AI you own, set up for you

  • Slides
  • Images
  • Diagrams
  • Web pages

GreenCube picks and installs the AI that fits your PC, then writes, makes slides and draws diagrams, all in one app. €9.99 (US$9.99), paid once, with a refund within 15 days.

Get GreenCube Check your PC