Skip to content
Sentynel

#opinion3 min readBy Sentynel

A passing agent eval is not a working operation

Salesforce now ships evals and cost limits for agents in production. They test the agent. The thing that has to work is the operation around it.

Share

On 27 October Salesforce announced Agent Evals and cost controls for its Agent Fabric, aimed at agents that "can fail silently, delivering wrong answers and racking up costs before anyone notices." It is a real problem and a useful product. But an agent can pass every eval and the operation can still fail, because the eval looks at the agent and the customer looks at the outcome.

What the new tools measure

Salesforce says Agent Evals checks "agents, LLMs, and MCP servers in production" against defined quality thresholds and triggers predefined interventions when results fall short. A Model Wallet adds "hard limits that stop requests before costs exceed budget." The same announcement notes that "fewer than half of organizations with agents evaluate them on live traffic."

That last line is the honest news: most teams are not measuring at all, so any measurement is progress. Start there.

Agent evaluation in production, as reported by Salesforce
< 50%of organizations with agents evaluate them on live traffic

Source: Salesforce, 27 October 2026

The unit of evaluation is the wrong one

An eval scores a response: was it accurate, was it on policy, was it cheap enough. Those are properties of one agent step. A company does not sell steps. It delivers things: the order that arrived, the invoice that was collected, the lead that was called back.

Picture an agent that classifies incoming requests with a high accuracy score. The eval is green. But if the "refund" queue it feeds has no owner, the request is classified correctly and then sits for nine days. Every score was good and the customer still waited. The failure lives between steps, where no eval is pointed.

Silent failure is mostly a design gap

"Fail silently" sounds like a model problem. Often it is an operation problem: nobody defined what a finished piece of work looks like, so there is nothing to compare the agent's output against. If "done" is not written down, the best eval in the world has no target.

Where the check should sit
  1. 01Work arrivesone owned queue
  2. 02Agent stepscored by the eval
  3. 03Human or system handoffowner named
  4. 04Outcome recordedresolved, or not

Measure the last box. The eval only covers the second.

What to do on Monday

  • Define "done" for one flow in a sentence a customer would accept: resolved, delivered, paid.
  • Record the outcome of every item, not only the agent's answer, so a case that stalls after the agent is visible.
  • Put a number on time-to-resolution next to the eval score. When they disagree, believe the one the customer feels.
  • Name an owner for each queue an agent feeds. No owner, no go-live.
  • Keep the cost limit. A hard budget stop is good engineering. Just do not mistake it for a quality control.

Sentynel's view is that companies run on an operating system made of habits, and that agents belong inside a redesigned one, never on top of the old one. Evals and cost controls are the instruments on that dashboard. They are worth buying. They do not draw the road.

A question for your next review: if every agent in your company passed its eval tomorrow, would you know whether the work got done?

Sources

  1. https://www.salesforce.com/news/stories/agent-fabric-catch-failures-control-ai-costs/
Share
← All sygnals