AI Reliability Engineer (LLMOps)

Also known as

The market uses several titles for the same role:

How it changes by seniority

• Junior — Instruments logs and dashboards and handles alerts by runbook.
• Mid — Defines alerts, fallbacks and per-agent spend limits; diagnoses incidents.
• Senior — Owns the agent fleet’s reliability and prevents regressions at scale.

Interview questions

1. 1. A production agent degrades in quality without anyone touching the code. How do you diagnose it?

   What to look for: Thinks drift (provider model, input data), not just bugs; uses observability.

2. 2. What do you instrument in an LLM system to operate it with confidence?

   What to look for: Traces inputs/steps/cost/latency and alerts on the abnormal; dashboards for the owner, not just logs.

3. 3. The provider is retiring the model you use. What’s your plan?

   What to look for: Re-runs evals with the new model, versions and prepares rollback; doesn’t trust "it’ll behave the same".

4. 4. How do you set SLAs on something probabilistic like an LLM?

   What to look for: Defines targets by accuracy/latency/cost and escalates to a human when they slip.

Scorecard

• Drift detection — Separates code failure from a changing world; catches degradation in time.
• LLM observability — Instruments inputs, steps, cost and latency; useful alerting, not noise.
• Model-change management — Versions, re-evaluates and prepares rollback for provider deprecations.
• Operable reliability — Defines measurable targets and escalation paths for something probabilistic.

Red flags

• — Only logs the final output: when it fails, there’s no way to know why.
• — Switches models without re-running the evals.
• — Alerts on everything until nobody heeds the alerts.