The hidden cost of retries in agentic AI systems

Retries are the most common reliability tool in distributed systems and the least examined line in an AI bill. In an agent workflow every layer of the stack is allowed to retry independently, each one has a reasonable default, and nobody owns the product of those defaults.

The result is a system that "usually works", with a tail of requests that take forty seconds and cost ten times the median. Those requests are not anomalies. They are the retry policy behaving exactly as configured.

Four layers of retry, none of them talking

In a typical LLM or agent service there are at least four places a call can be retried:

1. Provider SDK. Most client libraries retry 429 and 5xx responses two or three times with exponential backoff, silently.
2. Application call site. A wrapper around the model call adds its own retry for timeouts or malformed output.
3. Tool layer. External tools — CRM, pricing, search — have their own timeouts and retries, often inherited from an HTTP client default.
4. Workflow or agent loop. When a step fails, the orchestrator re-plans or re-runs the step, sometimes the whole chain.

Each layer is defensible in isolation. Multiplied, they turn one failed downstream call into a dozen billed inference calls.

What a trace shows

The trace below is an illustrative reconstruction of the pattern. A single external tool times out; the reasoner and the tool are retried twice with no backoff; more than half of the request's total cost is spent after the first failure.

WORKFLOW #A83Ftrace 7f3c…e91a

STATUS / DEGRADED

STEP0s4s8s11.8sCOST

01 CLASSIFIER
gpt-x 420ms €0.012

02 RETRIEVER
vector search 860ms —

03 REASONER
gpt-y · 6.1k tokens 3.4s €0.083

04 EXTERNAL TOOL
✕ timeout 4000ms 4.0s —

05 RETRY ×2
reasoner + tool, no backoff €0.116

TOTAL

11.8s

COST

€0.211

RETRY SHARE

55%

FINDING

▲ HIGHRetry amplification

Figure 1 — AI workflow showing classifier, retrieval, reasoning model, failed tool call and two retries increasing cost and latency. Illustrative, not a client result.

The arithmetic

Assume a workflow whose happy path makes one reasoner call and one tool call. Now give each layer its default retry count and look at the worst case for a single tool outage:

Two things are worth noticing. The cost is paid for requests that ultimately fail — the user got nothing. And the amplification is self-inflicted load on the provider, which is exactly the condition that produces the 429s the SDK is retrying.

A retry is a decision to spend more money on a request that has already failed once. Make it on purpose.

Retry budgets, not retry counts

The fix is not "fewer retries". It is one owner for the retry decision, expressed as a budget per user request rather than a count per layer:

request_budget:
  max_total_attempts: 4        # across all layers
  max_wall_time: 12s
  max_inference_cost: 0.25     # EUR, hard stop

layers:
  provider_sdk:   retries: 0   # budget owned above
  tool_client:    retries: 1, timeout: 3s, backoff: jitter
  workflow_step:  retries: 1, only_if: budget.remaining > 1
  agent_loop:     max_iterations: 3

With the budget in place the tail collapses. Failed requests fail fast and cheap, the provider sees no amplification, and — importantly — the trace for every request records how much of the budget was consumed, which turns retry behaviour into a metric you can alert on.

Checklist

• List every layer that can retry, including SDK defaults you did not configure.
• Compute the worst-case attempt count for one downstream failure. If it is above 4, you have amplification.
• Move retry ownership to one place and express it as a per-request budget of attempts, time and cost.
• Add jitter; remove retries on non-idempotent tool calls.
• Record attempts and budget consumed on the trace, and report cost per successful workflow rather than per request.