Skip to content
Sentynel

#technical4 min readBy Sentynel

A retry is not free: how to stop agents from re-doing work during an overload

New research shows retries keep an overloaded model engine from recovering. Here is how to set retry budgets and idempotency for agents in your operation.

Share

A paper posted to arXiv on 5 November 2026, "Cancellation Is Not Enough", tested what happens when an LLM engine is overloaded and the clients in front of it keep retrying. The authors report that with retries enabled, cancelling abandoned work did not restore service in any run, and that only 12–38% of the calls arriving after the overload were delivered. The lesson for operations is not about one inference engine: every automatic retry in an agent flow creates new work, and that work competes with the cases you are trying to finish.

What the research found

The setup is simple. A request times out, a client or proxy sends it again, and the engine now holds the old attempt and the new one. The authors ran overload experiments on vLLM with three open models and traced each client attempt to the execution it created. Their findings, as stated in the abstract:

  • Retries can stop an engine from recovering after a transient overload, even when the incoming load is below its capacity.
  • Cancelling abandoned work was not enough on its own.
  • A retry budget restored service by reducing retries, though many calls from the overload were still never delivered.
  • Request coalescing delivered every call from the overload onward, although some arrived late.
Retries enabled, cancellation only
12–38%of calls arriving after the overload were delivered
0 of all runsrestored service with cancellation alone

Source: Zhang and Wang, "Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine", arXiv 2611.08684, 5 November 2026. Experiments on vLLM with three open models; results are the authors' reported measurements.

Why this matters outside the engine room

You may not run your own inference engine. The pattern still applies one layer up. An agent calls a model, a CRM, a payments API or a messaging service. Something is slow, the step times out, and the agent or the platform tries again. Each attempt costs tokens, takes capacity from other cases and, worse, may repeat a side effect.

Three consequences show up in the operation:

  1. Cost rises exactly when quality falls. An overloaded system is the moment retries multiply.
  2. Cases queue behind duplicates. Work that nobody is waiting for keeps running.
  3. The same action can happen twice. A message sent, a record created, a ticket reassigned.

Four rules for retries in an agent flow

1. Give every flow a retry budget. Decide how many retries a step may use per case and per hour for the whole flow. When the budget is spent, the case goes to a named human or queue, not to another attempt. This mirrors the paper's finding that a budget restored service.

2. Make every side effect safe to repeat. Attach an idempotency key to each action that sends, creates, charges or deletes, so a second attempt returns the first result instead of acting again. If a step cannot be made idempotent, mark it "never retry automatically".

3. Collapse duplicates before they run. If the same request is already in flight, wait for its result instead of starting a second one. The paper reports that coalescing delivered every call from the overload onward.

4. Back off, and stop retrying what cannot succeed. Space attempts out with a growing delay and a ceiling, and separate errors that can clear (a timeout) from errors that cannot (a rejected input).

A step that fails safely
  1. 01AttemptWith an idempotency key
  2. 02Timeout?Back off, check the budget
  3. 03Join or retryReuse an in-flight request
  4. 04Budget spentStop retrying
  5. 05Hand to a personA named queue

Every exhausted retry budget ends in a human decision, never in silence.

Measure what retries add

The authors conclude that overload controls should be judged by the extra executions retries create and by the calls clients actually receive. That is a good operating metric too. For each flow, track:

Metric What it tells you
Attempts per completed case How much work is repeat work
Cases that exhausted the retry budget Where the flow is fragile
Duplicate side effects detected Whether idempotency actually holds
Cases completed during a slowdown Whether the flow degrades or collapses

Pair this with a cost per resolved case, not with the token bill alone: a flow can look cheap per call and expensive per outcome.

Rehearse the slowdown

Once a quarter, make a dependency slow on purpose in a test flow. Check that attempts stay within budget, that no side effect repeats, and that every stuck case lands with a named person. If you cannot see attempts per case today, that is the first thing to add.

A test for this week

Pick the agent flow with the most steps. For each step, write down two answers: what happens if it is run twice, and who gets the case after the last allowed retry. If either answer is "I don't know", that step is not ready for an overload.

Sources

  1. https://arxiv.org/abs/2611.08684
Share
← All sygnals