Change control for AI agents: treat every prompt, tool and model swap as a release
An agent changes when its prompt, tools or model change. Here is how to version, test and roll back those changes so the operation around it keeps working.
An AI agent in production is not a fixed program. It changes whenever someone edits its prompt, adds a tool or switches the model underneath it, and each of those edits can change how work gets done. Treat every one of them as a release: versioned, tested against a fixed set of cases, rolled out to a slice of the work and reversible. Most teams do this for code and skip it for agents.
What actually changes an agent
An agent's behavior is the sum of four things, and any of them can move without a line of application code changing:
- The instructions: the prompt, the procedure, the escalation rules.
- The tools: what it can read, write or trigger, and how those tools describe themselves.
- The model: a provider update or a deliberate swap.
- The data: the records, documents and context it reads at run time.
Anthropic's engineering guidance on building agents makes the same point from the other side: start with the simplest approach, measure performance, and add complexity only when it is needed. It also recommends extensive testing in sandboxed environments with appropriate guardrails, because autonomous agents can compound errors. Change control is how you keep that discipline after launch.
Step 1: write down the agent's version
Give each agent a single record that lists the exact prompt, tool list, model name and the procedure document it follows. Store it where changes are visible: a repository, not a chat. Every run logs the version it used. When something goes wrong on a Tuesday, you can say which version did it.
Step 2: keep a fixed set of cases
Collect real examples from the operation: ordinary requests, awkward ones and the exceptions people had to fix by hand. Record what a correct result looks like for each. This is your regression set. A new version must handle all of it before it touches live work.
Keep a baseline from the live operation as well. Without one, you cannot tell whether a change helped. Sentynel's Rodo case is a useful example of why: its record says the 82% resolution figure covers a first measured window with no comparable earlier baseline, so it can only anchor what comes next.
Source: Rodo operational report, 24 September 2026. The resolution cohort covers 31 August–24 September. Figures describe pipeline records, not all conversations.
Once a window like this exists, every later version is compared against it, not against a feeling.
Step 3: classify the change
Not every edit needs the same care. Decide in advance:
- Minor: wording fixes that do not change what the agent may do. Run the regression set, then ship.
- Standard: a new tool, a changed rule or a new data source. Regression set plus a partial rollout.
- Major: a model swap, new write access or a new kind of customer-facing action. Regression set, partial rollout and a named person who approves.
Anything that widens what the agent can do is at least standard, whatever the size of the edit.
Step 4: roll out to a slice, with a way back
- 01ChangeNew version recorded
- 02Regression setFixed cases must pass
- 03Small sliceA share of live work
- 04CompareAgainst the baseline
- 05Promote or revertOne step back
Every run logs the version that produced it.
Route a small share of live work to the new version while the rest stays on the old one. Compare the same measures you already track: resolution, time to resolve, the rate of escalations and the hours people spend reviewing. Define in advance what counts as worse, so the decision to revert is mechanical. Reverting must be one step: switch the record back to the previous version.
Step 5: assign an owner and a cadence
Every agent has a named owner who approves major changes and reads the comparison. Set a regular review of the regression set too: when the operation meets a new exception, add it. The set should grow with the work.
The test to run this week
Pick your most important agent and answer three questions. Which exact version ran yesterday? What would you run to check a new version before it goes live? How long would it take to go back? If any answer is "I would have to find out", that agent changes in production with no control, and the fix starts with a record, not a better model.