Why your agent bill grows quadratically

Each step appends a tool call and a tool result, then sends the whole list back. With a modest 1,000 tokens added per step:

step  messages sent   input tokens this step   cumulative input
  1         2                    1,000                  1,000
  2         4                    2,000                  3,000
  3         6                    3,000                  6,000
  5        10                    5,000                 15,000
 10        20                   10,000                 55,000
 20        40                   20,000                210,000
 40        80                   40,000                820,000

Read the last column, not the third. A forty-step run has sent 820,000 input tokens to produce one answer, and 780,000 of those are text the provider has already been shown.

Every step resends everything before it.

  cost(n) = sum of k*T for k in 1..n
          = T * n * (n + 1) / 2
          ~ T * n^2 / 2

T = tokens added per step. Doubling the steps roughly quadruples the bill.

This is why an agent that works fine in a demo becomes alarming in production. Demos are short. The distribution of real tasks has a tail, and the tail is where the quadratic lives.

Provider-side caching bills a repeated prefix at a reduced rate, which genuinely helps. It has one requirement that agent loops keep breaking: the cached portion must be a byte-identical prefix.

An agent appends to the end, so the prefix does stay stable, and caching works well. Then somebody injects the current time into the system prompt, or reorders tool definitions, or the retrieved documents change per step, and the cache misses on every call. The saving disappears quietly, because nothing fails.

Caching also does not shorten anything. It reduces the price per token on the repeated part; the quadratic is still there, just discounted.

The direct fix is to keep the message list bounded, which means deciding what is worth carrying.

async function step(messages: Msg[], budget = 12_000): Promise<Msg[]> {
  if (estimateTokens(messages) < budget) return messages;

  // Keep the system prompt and the original request: those are the task.
  // Keep the last few turns: those are the working state.
  // Summarise the middle, which is where the transcript actually bloats.
  const head = messages.slice(0, 2);
  const tail = messages.slice(-4);
  const middle = messages.slice(2, -4);
  if (middle.length === 0) return messages;

  const summary = await summarise(middle);
  return [...head, { role: "system", content: `Earlier steps: ${summary}` }, ...tail];
}

Summarising costs a model call, so it pays for itself only once the transcript is genuinely large. Below the threshold it is pure overhead, which is why the budget check comes first.

Before summarising, look at what is actually taking the space. It is almost never the reasoning. It is a tool that returned a 40 KB JSON document because nobody thought to project the three fields the model needed.

// Cheaper than summarising, and usually enough: most tool output is read
// once, by the next step, and never referenced again.
function trimToolResults(messages: Msg[], keepFull = 3): Msg[] {
  let seen = 0;
  return [...messages].reverse().map((message) => {
    if (message.role !== "tool") return message;
    seen++;
    if (seen <= keepFull) return message;
    return { ...message, content: message.content.slice(0, 200) + " ...[trimmed]" };
  }).reverse();
}

Trimming old tool output is cheap, requires no model call, and usually recovers more than summarising does. The better version of this fix is upstream: make the tool return less. A tool that answers "is this user active?" should return a boolean, not the user record.

The step budget is usually treated as a safety valve, and it is also a cost control. Because cost is quadratic in steps, cutting a limit from 25 to 15 does not remove 40% of the worst case. It removes about 64% of it.

The same asymmetry is worth remembering when choosing a model. A stronger model that finishes in 6 steps can be cheaper end to end than a cheaper one that takes 15, even at several times the per-token price. Compare on cost per completed task, never on cost per token.

Cost per run is the wrong metric to alert on, because it hides which runs are pathological. Track steps per completed task and input tokens per step. When either drifts, cost moves quadratically and you will see it in the bill weeks before you see it in a dashboard built on averages.