Someone offers to «train a model on your data» and it sounds like finally making the AI yours, tailored, unbeatable. It sounds expensive, which is why it sounds serious. Before signing, it helps to know that in the vast majority of company cases you’re about to pay the expensive fix for a problem the cheap one solves better. The underlying decision is called RAG or fine-tuning, and the right answer is almost always RAG. Here’s why, no smoke.
RAG vs fine-tuning in a company: what each one does
They’re two different things the jargon mixes on purpose. RAG (retrieval-augmented generation) doesn’t touch the model: it gives it a library. When a question comes in, the system pulls the relevant fragments from your documents and hands them to the model along with the question, so it answers with your information in front of it. The knowledge lives outside the model, in a base you can update at any time.
Fine-tuning is the opposite: you take a base model and retrain it on thousands of examples so it adjusts its internal weights. You don’t give it a library; you change its reflexes. It’s for teaching a format, a tone or a very repetitive kind of task —not for feeding it data that changes. And that’s where the costly confusion starts: almost everyone who asks for fine-tuning actually wants the AI to «know their stuff», which is exactly what RAG does better.
| Dimension | RAG | Fine-tuning |
|---|---|---|
| What changes | The context the model receives | The model’s internal weights |
| What it’s for | Answering with YOUR knowledge | Adopting a fixed format/tone/task |
| Updating a fact | Edit a document, done | Retrain the whole model |
| Startup cost | Low | High (labeled data + compute) |
| Traceability | Cites the source it used | Black box: no idea where it came from |
| Typical risk | Poorly built retrieval | Hallucinating confidently with stale data |
Why they’ll sell you fine-tuning (and why you almost never need it)
Fine-tuning sells well for three reasons, and none of them is that it’s what you need. First: it sounds bespoke —«a model trained on your data» is a sentence that closes meetings—. Second: it’s more expensive, and in consulting expensive gets confused with good. Third: it’s hard to undo, so it locks the client in. The problem is it solves the wrong problem: the company doesn’t need a model with a different personality, it needs one that answers with the correct, current information. That’s a retrieval problem, not a retraining one.
The proof is in the data’s shelf life. Your vacation policy, your catalog, your prices and your procedures change. If you bake them in via fine-tuning, every change demands a retrain —and until it lands, the model asserts the old version with total confidence. With RAG you edit the document and the next question already answers with the new one, citing where it got it. For living knowledge, retraining is like printing Wikipedia every morning.
When it IS fine-tuning
It’s not that fine-tuning is never useful. It is, but for a narrow and fairly identifiable set of cases:
- You need a very rigid, repetitive output format that prompting can’t stabilize (say, classifying into your own schema with thousands of examples).
- You want a very specific, constant tone or style, not a fact: teaching it to write «like your brand» when the system prompt has run out of road.
- You have a very high-volume task where trimming context tokens on every call outweighs the cost of training.
- And —key— that behavior you teach does NOT change every week. If it changes, you go back to RAG.
Notice the pattern: fine-tuning is for teaching a stable skill, not for injecting changing knowledge. The moment what you want it to «know» has an expiry date, the answer swings back to RAG. And in a real company, almost everything you want it to know has an expiry date.
The 90% of company cases: why RAG wins
For the question a company actually asks —«answer with our stuff, up to date, without making things up»— RAG wins on the three things that matter: it’s cheaper to start, it updates by editing a document and, above all, it cites the source, so you can audit where each answer came from. A serious agent in 2026 is almost always a good RAG system with evals measuring whether it answers well, not a retrained model. We cover it in detail in the guide on how to train an AI agent: data, evals and governance, and in the one on building an AI chatbot that uses your knowledge base, where the RAG architecture is laid out step by step.
RAG’s other advantage is that failure is cheap and visible. If the AI answers badly, it’s almost always because retrieval fetched the wrong fragment —and that’s fixed by improving how you index and search, not by retraining anything. With fine-tuning, when it fails, you have an expensive black box and a days-long correction cycle. Building the retrieval layer right —which documents, how they’re chunked, how they’re searched, what’s logged— is infrastructure, and it’s where the project is won or lost: we treat it as exactly that in the enterprise AI infrastructure service.
The expensive mistake: fine-tuning to «inject knowledge»
If you keep one idea, make it this: fine-tuning changes how the model talks, not what it reliably knows. Using it to inject data is the most common and most expensive mistake, because the model learns to sound like your documents without guaranteeing it reproduces their facts —and it does so without citing, so you don’t even notice when it’s wrong. Always start with RAG. Get to fine-tuning only if, with retrieval already built right, you’re left with a real format or style problem context can’t solve. In that order you spend on what’s needed, when it’s needed.