Solution · AI Operations
Your prompts are in production and nobody knows for sure which version is running
The prompt is the single biggest lever on how your AI behaves, and in most companies it gets edited by hand: no review, no history, no way back. We build and run the function that turns it into a versioned artifact, tested before it ships and reversible in a minute.
The problem
The artifact with the most control over your AI is the only one nobody governs
- Nobody can say for sure which version of the prompt is serving right now, who changed it last, or why.
- A one-line change ships straight onto production, because there's no environment to test it against real cases first.
- When quality drops, you can't go back to the previous version: it isn't saved anywhere, or it's buried in the history of a chat conversation.
- The same prompt lives duplicated in the code, in a tool and in somebody's clipboard, and each copy has been drifting on its own for months.
- When the provider updates the model you have to review prompts, but nobody knows which ones exist, which depend on that model, or what they were validated against.
- The team has stopped touching the prompts. Not because they're good: because changing them is scary and nobody wants to be the one who breaks production on a Thursday.
Cost of staying the same
You're running a system whose highest-impact component has no change control. The cost arrives from two directions at once. First, the regression nobody saw coming and a customer finds. Then, more expensive, the paralysis: when changing the prompt is a gamble, the team stops touching it and the system stops improving right as the market starts moving. On top of that comes a third front, the moment somebody asks from outside: with no history you can't reconstruct which instructions governed your system the day it made a specific decision.
The solution
We build and run change management for your prompts: version, test, phased rollout and rollback
- 1We inventory and consolidate. We pull every prompt out of wherever it lives —code, tools, documents, heads— and turn it into an artifact with an ID, an owner, a version and a history, with a single source of truth every environment reads from.
- 2We separate environments and put a gate in. A change is born outside production, runs through the case suite, and only ships if it clears the agreed threshold. Same gate for everyone, no exceptions for urgency: urgency gets a documented fast lane, not a way around the control.
- 3We build the prompt's test suite: a set of real and adversarial cases, frozen and versioned, run against every candidate and compared with the output of the version that's live. Without the comparison against the live version, a good result means nothing.
- 4We roll out in phases. The new version goes in on a slice of traffic first, with the previous one running in parallel on the same inputs, and only then to all of it. Rolling back is a pointer change: one minute, no code deploy, no meeting.
- 5We run the function day to day: who can change what, what gets reviewed before shipping, what stays on record for every change so it can be reconstructed later, and the scheduled review every time the provider moves the model underneath you.
What changes
What you stop losing
The question “which version is running” goes from an investigation to a lookup: there is an ID, a history and an owner for every prompt in production.
Mechanism
A change stops being a gamble: it's compared against the live version on the same cases before shipping, and it ships in phases over part of the traffic.
Mechanism
Rolling back stops depending on somebody remembering the previous text: it is a pointer change to a saved version, with no code deploy.
Mechanism
What we measure: time from a regression being detected to it being reverted, % of changes that go through the gate, reverted changes as a share of the total, and number of prompts in production with no identified owner.
What we measure
Spec sheet
- Work it removes
- editing prompts by hand straight onto production, with no history, no comparison against the live version and no way back when quality drops
- Typical setup
- 4–8 weeks
- Input
- a proposed change to a prompt in production: an instruction tweak, a model swap or a fix after an incident
- Output
- a new version tested against the case suite, compared with the live one, rolled out in phases and reversible in a minute, with its record of who, when and why
- Works with
- OpenAIAnthropicAzure OpenAIGoogle Vertex AILangSmithLangfusePromptfooBraintrustGitHub ActionsGitLab CI
- Can connect to
- Tu repositorio y tu circuito de revisión de códigoTus registros de producción, de donde salen los casos de pruebaTu función de evaluación continua de calidad, si ya la tienes montadaTu registro de trazabilidad y tu inventario de sistemas de IA
- What we measure
- time from a regression being detected to it being reverted% of changes that go through the gate before production% of changes reverted out of everything deployedprompts in production with no identified owner
- Good fit for
- companies with AI in production and several people touching prompts —CIO, COO or the AI lead— who need to change fast without breaking things and to show what governed the system at any given moment
- Not a fit for
- anyone with a single stable prompt nobody has touched for months, or anyone looking to measure whether the agent's answer is good: that's quality evaluation, a different and complementary function
Frequently asked questions
In what they govern. Continuous evaluation measures whether your agent's answer is good today: it samples real conversations, scores them against a criterion and catches quality dropping. This function governs change to the artifact that produces that answer: where the prompt lives, who can touch it, what test it has to pass before shipping, how it rolls out and how it rolls back. They're complementary and they work better together —the case suite and the quality metrics feed off the same material— but they solve different problems: one tells you something broke, the other makes breaking it reversible and rare.
Git is the foundation and we use it, but on its own it solves half the problem. It gives you history and review, and that's already more than most have. What it doesn't give you: an environment to test the candidate against real cases before shipping, the automatic comparison against the version that's live, the rollout over a slice of traffic, and the way back with no code deploy —which is what you need at eleven at night. On top of that, in plenty of teams the person who writes the best prompts doesn't work in the repository, and making them open a pull request to change one sentence ends in the usual parallel copy.
The opposite: what slows you down today is fear. In teams with no change control, prompts get touched rarely and with held breath, because any tweak could break something nobody will catch until a customer complains. When testing costs minutes and going back costs a minute, the price of being wrong drops so far that people start experimenting again. The gate isn't there to add bureaucracy: it's there to make change cheap. And for the genuinely urgent there's a documented fast lane, with mandatory review after.
Want it running in your business?
You’ve pinned the problem. We ship the fix and leave it measured.