Agent retries and error handling
The retry that charged twice
Agent frameworks retry failed tool calls by default. If the tool moved money, sent mail or created a record, the retry did it again. Idempotency keys, from scratch.
· 8 min read
Every agent framework retries. Transient failures are the common case, the model often recovers when handed an error, and a retry is the obvious default. It is also the default that quietly turns one action into two.
The shape of the failure
agent tools/call charge_card { amount: 4900 }
tool -> payment provider: charge created, id ch_123
-> response times out on the way back
agent sees a timeout, which its retry policy treats as retryable
agent tools/call charge_card { amount: 4900 }
tool -> payment provider: charge created, id ch_456
Two charges. Nothing errored. Both calls were "successful".Nothing here is a bug in the ordinary sense. The tool worked, the retry policy did what it was configured to do, and the model was never involved in the decision. The system did exactly what it was told, twice.
Agents make this worse than a normal service call for two reasons. Retries happen at several layers at once (HTTP client, framework, and the model itself re-calling a tool it thinks failed), and the model is an unusually enthusiastic retrier: hand it error: timeout and it will frequently just try again.
Idempotency keys, from scratch
The fix is to make the second attempt recognisable as the same intent, so it can return the first result instead of repeating the work.
type ToolCall = { id: string; name: string; args: Record<string, unknown> };
// The key must be derived from the intent, not generated per attempt:
// a fresh uuid on every try is a new key, which is no protection at all.
function idempotencyKey(runId: string, call: ToolCall): string {
return sha256(`${runId}:${call.name}:${stableStringify(call.args)}`);
}
// Stable stringify matters more than it looks: {a:1,b:2} and {b:2,a:1} are the
// same intent, and JSON.stringify gives them different keys.
function stableStringify(value: unknown): string {
if (value === null || typeof value !== "object") return JSON.stringify(value);
if (Array.isArray(value)) return `[${value.map(stableStringify).join(",")}]`;
return `{${Object.keys(value as object).sort()
.map((k) => `${JSON.stringify(k)}:${stableStringify((value as any)[k])}`)
.join(",")}}`;
}Deriving the key from the arguments is what makes this work across layers: a model re-calling the tool, a framework retry and an HTTP retry all produce the same key, because they all describe the same intent.
Claiming before working
The subtle part is ordering. Recording the key after the work leaves exactly the window that caused the incident.
const inflight = new Map<string, Promise<string>>();
async function callToolOnce(runId: string, call: ToolCall, tool: Tool, store: Store) {
const key = idempotencyKey(runId, call);
const done = await store.get(key);
if (done) return done; // already ran: replay the result
// Two retries racing the same key must not both execute.
const existing = inflight.get(key);
if (existing) return existing;
const work = (async () => {
// Claim the key BEFORE doing the work. Claiming afterwards leaves the
// window that caused the double charge in the first place.
const claimed = await store.claim(key);
if (!claimed) return store.waitFor(key);
const result = await tool.run(call.args);
await store.complete(key, result);
return result;
})();
inflight.set(key, work);
try {
return await work;
} finally {
inflight.delete(key);
}
}store.claim has to be atomic: a conditional insert, not a read followed by a write. Across processes, two retries landing on different instances will both pass a naive check-then-set, which is the same bug with more steps.
Classify the tools, not the errors
The usual instinct is to make the retry policy smarter about which errors are transient. That is the wrong axis: a timeout is genuinely ambiguous, and no amount of error classification resolves it. Classify the tool instead.
safe to retry read a record, search, compute, fetch a page
safe with a key create a charge, send mail, insert a row, post a message
never retry blindly anything the provider cannot deduplicate, and anything
whose effect you cannot observe afterwardsThen let the tool definition carry it. A tool that declares itself unsafe to retry gets one attempt and an honest error, and the model can tell the user rather than silently doing it again.
What to keep from this
None of this is specific to agents. It is ordinary distributed-systems hygiene, arriving in a place that does not look like a distributed system, through a default nobody chose.
The question to ask of any agent framework is not whether it retries, but whether it lets a tool say "do not", and whether it deduplicates across the model, the framework and the transport. Most answer the first and not the second, which is why the key belongs in your tool rather than in your configuration.
The same mechanism, with its tradeoffs
Every pattern page runs its code on the page, in TypeScript, Python and Rust.