Bounded autonomy: loop and budget limits for AI agents

An agent that can try again is useful. An agent that tries again without limit is a liability. Most of the interesting work in an agentic system happens in loops: write something, check it, fix it, check again. Those loops are where agents earn their keep, and they are also where they go wrong.

This note covers how loops drift, the limits we put on them, and why we treat those limits as part of the design rather than a safety net added later.

Why loops drift

Ask an agent to fix failing code and it will try. If the first attempt fails, it tries again, now with the previous attempt and its error in view. Each round adds context, and each piece of context nudges the next attempt.

After enough rounds, the agent is often no longer solving the original problem. It is solving the narrower problem of making the most recent error disappear. Fixes start to pile on top of fixes. The code passes a check, but nobody can say any more whether it still does what it was meant to do.

In research this is more than an inefficiency. An agent that keeps adjusting a strategy until it passes validation is not testing a hypothesis. It is fitting the strategy to the test. The longer the loop runs, the more the final result reflects the loop rather than the idea.

Hard limits on iterations

Every loop in Pragma has a maximum number of iterations, set before the run starts. When the loop reaches it, the loop ends.

The important word is hard. The limit is not an instruction in the agent’s context that the model may or may not follow. It is enforced by the pipeline around the agent. The agent cannot negotiate it, forget it or talk its way past it.

Hard limits on spend

Each run also has a budget for model usage. Agents work within it, and when a run reaches its budget, it stops cleanly and records where it stopped and why.

A spend limit protects more than cost. It caps how much effort can be poured into a single idea, which is itself a guardrail against fitting. If an idea needs unlimited attempts to survive, that is a finding about the idea.

Stop instead of retry forever

When a limit is reached, the run stops. It does not quietly start again with a fresh context, and it does not escalate to a bigger budget on its own.

A stopped run is a result. It tells you that, within the limits you set, the hypothesis could not be implemented or could not be validated. That is useful information, and it is recorded alongside the run so you can see exactly which stage stopped and at which point.

Retrying forever would hide that information. It would also erode the separation between building a strategy and judging it, because every retry is another chance to shape the strategy around the check that is supposed to evaluate it.

Limits as a design input

We decide the limits for a stage when we design the stage, not after something has gone wrong.

Knowing that an agent has a fixed number of attempts changes how we structure its task. We break work into smaller steps, each with a clear success condition. We define explicit failure states, so an agent can report that it could not finish rather than produce something that only looks finished. We make sure each attempt leaves a record, so a stopped loop can be understood afterwards.

When a stage regularly runs into its limits, we read that as a signal about the stage, not about the limit. A stage that cannot do its job within reasonable bounds usually needs a clearer task, a better contract or a different split of responsibilities. Raising the limit treats the symptom.

What limits do not do

Limits do not make an agent smarter. A run that finishes within its limits is not correct for that reason alone, and limits are not a substitute for validation.

What they do is make failure visible and keep it cheap. A bounded agent fails in a known place, at a known cost, with a record of what it tried. An unbounded one fails slowly, expensively and in ways that are hard to see. For research software, where the point is to learn what holds up and what does not, we will take the first kind of failure every time.