IBM researchers report that they made an AI agent more dependable on repeated tasks by giving it instructions derived from its own unstable decisions. In their AppWorld test, a GPT-4.1 agent completed all five attempts on 69.0% of tasks, up from 53.0%, without retraining the underlying model.
The starting numbers explain why this is worth attention. The agent averaged 77.4% success across individual attempts, yet passed every attempt on barely half the tasks. For someone delegating recurring work, those figures answer different questions: whether the agent usually succeeds, and whether it keeps succeeding when asked to do the same thing again.
Across 168 AppWorld tasks, five-for-five success improved more sharply than average success. These are IBM-authored benchmark results, not an independent replication.
| Agent configuration | Mean@5 | Pass^5 |
|---|---|---|
| Baseline | 77.4% [1] | 53.0% [1] |
| With consistency guidelines | 81.0% [1] | 69.0% [1] |
Method: Values come from Table 1 of IBM’s preprint. A GPT-4.1 ReAct agent attempted each of 168 AppWorld test_normal tasks five times. Mean@5 averages success across the five runs; Pass^5 counts tasks passed in all five runs. As of: 2026-09-17. Comparison: GPT-4.1 with the same ReAct architecture on 168 AppWorld test_normal tasks, comparing five fresh baseline runs with five fresh runs using same-task consistency guidelines. (percent; Preprint results retrieved 2026-09-17).
- [1] Closing the Consistency Gap: Self-Evolving Agents That Learn to Stay on Course (retrieved 2026-09-17)
With IBM’s consistency guidelines, average success also rose, reaching 81.0%. The gap between average success and five-for-five completion narrowed from 24.4 percentage points to 12.0. These are IBM’s own results, reported in an arXiv preprint, rather than an independent replication or a measurement of a live service.
The experiment covered 168 tasks in AppWorld, a controlled simulation of nine personal apps with fictitious users. Its tasks require agents to carry out multiple steps through software interfaces. IBM used a ReAct agent, which alternates reasoning with actions, backed by GPT-4.1 through Microsoft’s Azure service.
The researchers call average success Mean@5 and the fraction of tasks passed on every attempt Pass5. That second measure differs from Pass@5, which asks whether at least one attempt succeeded. The 24.4-point gap is a subtraction between two scores, not the share of tasks with mixed outcomes. IBM separately reports that about 32% of tasks had both successes and failures.
Instructions for the moments that go wrong
IBM’s intervention starts with a record of what the agent already did. Its Consistency Analyzer revisits individual decisions to identify where the model gives varying responses to the same recorded context. A separate generation step turns those unstable moments into guidance that the agent can consult later.
The analyzer resamples decisions from a recorded run offline, without repeating the task’s tool actions. An AI model turns flagged steps into reusable instructions, which enter memory and are retrieved into later prompts; the model’s weights stay unchanged. The experiment sampled 30 responses per analyzed step. Agreement identifies stability, but cannot establish that a decision is correct.
A counting example in IBM’s announcement makes the idea concrete. Asked how many bucket-list activities were complete in a note, an agent could count checkbox symbols that also appeared in the title or legend. The generated guidance told it to count markers at the beginnings of lines and verify that it had selected the right note. Such instructions can target a fragile decision even when the recorded attempt happened to succeed.
For the main result, IBM generated guidelines from one baseline run per task, then evaluated five fresh runs. The operational agent used temperature zero, a setting intended to reduce output randomness. The observed variation shows that this setting alone did not make the tested agent repeatable. IBM has made the analysis and guideline-generation machinery available in its ALTK-Evolve repository.
The improvement has a bill and a boundary
The published experiment used more analysis than the announcement’s default of five sampled responses per step might suggest. The paper specifies 30 responses at temperature 0.5 and estimates roughly 30 times a trajectory’s token cost for analyzer resampling. That is an offline analysis expense, not a claim that every subsequent task costs 30 times more.
The paper gives no aggregate dollar bill, processing time or number of later uses over which that expense could be spread. Guideline generation and their use in later prompts also belong in a complete cost calculation. The result therefore establishes an improvement in measured performance, without establishing whether it pays for itself.
There is also a distinction between consistent decisions and correct ones. The authors acknowledge that a confidently wrong decision looks consistent to the analyzer. Generated guidelines enter memory without a correctness-validation gate; testing them before storage is proposed as future work. The paper does not report that the intervention increased consistently wrong outcomes.
The demonstrated gains remain within AppWorld, ReAct and the two models IBM tested. Varied simulated tasks provide useful evidence, but do not establish how the method handles changing workplace tools or policies. Five successful repetitions also cannot tell a buyer how severe the next failure might be.
Our read is that IBM has identified a useful target for improvement: decisions that an agent gets right only intermittently. Before trusting the method with recurring work, we would want evidence that its guidance stays correct on new tasks and saves more in avoided failures than it costs to produce.