How to train an AI agent: data, evals, governance

Training an AI agent isn't "upload a PDF" or "fine-tune the model". In 2026, it's almost always building a good RAG system, writing the evals that measure if it responds well, and designing the loop that improves based on real data. The rest is confusing vocabulary — and it costs money.

The 4 levels of "training": prompt, RAG, fine-tune, agent training

Level	What it is	When
Prompt engineering	Writing good instructions (system prompt)	Always — the foundation of everything
RAG	Connecting the model to your knowledge base	When the agent must use your specific info
Fine-tuning	Adjusting the model with your examples	Rarely — only very specific cases
Agent training	Iterating on the full agent with evals	Always — continuous cycle

Which one applies to your case

90% of cases: prompt + RAG + agent training. No fine-tuning.
You need a very specific tone or style the prompt can't reach: consider fine-tuning on a small model.
Latency/cost constraints: fine-tuning on Llama or a similar model to run faster and cheaper.
Highly specialized data (medical, legal): combination of strong RAG + selective fine-tuning.

How to build evals (the part almost nobody does)

Evals are what separate a serious agent from a pretty demo. And almost nobody builds them. The process:

Gather 50-200 representative inputs of the real cases the agent will handle.
Define the expected output for each — or the range of acceptable outputs.
Define automated evaluation criteria — measurable metrics (factual correctness, format, absence of hallucinations).
Run after every change to the agent (prompt, RAG config, model). If the score drops, it doesn't ship.
Iterate the set — add edge cases you spot in production.

The continuous-improvement loop

Production captures real interactions with feedback (CSAT, detected errors).
Weekly human review: identify error patterns.
Update the KB / prompt / config based on what you found.
Run evals to check there are no regressions.
Deploy the change.
Back to step 1.

Governance and sensitive data

Business-plan APIs. OpenAI API and Anthropic API on business plans don't train on your data. Confirm it in your DPA.
Anonymize when possible. Patterns help the model; names don't.
Encrypted logs. If you store conversations, encrypt at least the ones containing personal data.
Minimum retention. Don't keep what you don't need. Clear deletion policy.
Regular audit of what data enters the model and from where.

Frequently asked questions

In 2026, almost never. Base models are good enough and RAG is flexible enough that 90% of cases get solved with prompt + RAG done well. Fine-tuning helps when: (1) you need very specific tone/style you can't get from prompting, (2) you have latency/cost constraints that justify a smaller specialized model. If they offer fine-tuning as first option, ask why not RAG.

With evals: a set of representative inputs + expected outputs + automated evaluation criteria. Evals run on every agent change and produce a score. If your agent "works" but you don't have evals, you don't know — you have a hunch. The difference between a serious agent and a pretty demo is almost always whether evals exist.

To start: Promptfoo or LangSmith. For serious production: Braintrust or Galileo. Very technical team and you want open-source: DeepEval. The pick matters less than the habit — the problem isn't which tool, it's that most teams don't do evals at all.