RAG vs Fine-Tuning: When to Use Each
The Fundamental Question
You have a domain-specific problem. You need an LLM that knows about your company's products, your industry's regulations, your codebase's conventions. How do you get there?
Two main approaches exist: Retrieval-Augmented Generation (RAG) and fine-tuning. They're often discussed as if they're competing solutions, but they solve different problems. Understanding that distinction is the first step to making the right call.
What RAG Actually Does
RAG keeps the base model frozen. Instead of teaching the model new information, you retrieve relevant information at inference time and include it in the prompt.
The flow looks like this:
- User asks a question
- Your retrieval system searches a knowledge base for relevant chunks
- Those chunks are prepended to the prompt
- The model answers using both its training knowledge and the retrieved context
The knowledge lives in your database, not in the model's weights. This means you can update it without retraining anything — add a new product, update a policy, and it's immediately available.
RAG is best for:
- Large, frequently updated knowledge bases
- Information that changes over time (pricing, policies, product specs)
- Long-form documents (contracts, reports, documentation)
- Cases where you need to cite sources
- Situations where factual accuracy is critical and hallucination is costly
What Fine-Tuning Actually Does
Fine-tuning continues the training process on your specific data. You're adjusting the model's weights so it behaves differently — producing outputs in a specific style, following a specific format, applying domain-specific reasoning patterns.
Fine-tuning is not primarily a knowledge injection mechanism. It's a behavior modification mechanism.
Fine-tuning is best for:
- Teaching the model a consistent output format
- Replicating a specific writing style or brand voice
- Domain-specific reasoning patterns (legal analysis, medical triage protocols)
- Reducing refusals on legitimate domain-specific queries
- Efficiency: smaller fine-tuned models can outperform larger base models on narrow tasks
The Decision Tree
Start here: do you need the model to know different things, or behave differently?
If you need it to know different things → RAG
If the gap between what the model knows and what you need is primarily factual — it doesn't know your product catalog, your company's history, your customer's account details — RAG is the right tool. The model's reasoning capabilities are fine; it just lacks data.
If you need it to behave differently → Fine-tuning
If the model keeps producing the wrong format, uses the wrong tone, fails at a reasoning pattern your domain requires, or consistently refuses to engage with content it should handle — fine-tuning is the right tool. The issue is behavior, not knowledge.
The Hybrid Approach
For most production systems, the answer is both — and in sequence.
Step 1: Build a RAG system. It's faster to build, easier to update, and handles the knowledge problem well.
Step 2: Identify the behavioral gaps that RAG doesn't fix. Where does the model still produce bad outputs even with good retrieval?
Step 3: Fine-tune on examples of correct behavior for those specific gaps.
This sequencing matters. Fine-tuning before you've identified the real behavioral gaps means you're optimizing for the wrong thing.
Common Mistakes
Trying to fine-tune in facts: If you add product names, dates, or statistics to your fine-tuning data, the model will learn to produce plausible-sounding text in that domain — but it won't reliably recall specific facts. RAG is dramatically better for factual recall.
RAG on small, stable knowledge bases: If you have 50 documents that never change, just include them in the system prompt. Vector databases and retrieval pipelines add complexity and latency. Earn that complexity before adding it.
Fine-tuning on too little data: A few hundred examples will barely move the needle on a large model. You typically need thousands of high-quality examples to see meaningful behavioral change. If you don't have that data, consider prompt engineering or RAG instead.
Ignoring evaluation: Both approaches require rigorous evaluation. Build your eval suite before you start, so you can measure whether changes are actually improvements.
Practical Costs
Fine-tuning isn't free. Beyond the compute cost (which has dropped significantly), you need labeled training data — which means human review time. Plan for 1,000-10,000 examples for meaningful fine-tuning, and account for the time to create and quality-check them.
RAG's costs are primarily in inference (more tokens per call due to retrieved context) and infrastructure (vector database, embedding model). At scale, these add up — but they're more predictable than fine-tuning's upfront investment.
The right answer is almost always: start with RAG, measure what's still broken, then decide if fine-tuning is worth the investment for those specific gaps.
