Your Agent Aced the Task. Will It Do It Again?

IBM Researchers Target AI Agent Unreliability with New Diagnostic Tool

Researchers from IBM have released a new method to address a critical flaw in AI agents: inconsistency. While an agent may succeed on a task once, it can often fail on subsequent identical attempts, a reliability problem that hinders enterprise adoption for mission-critical work. The team introduced the "Consistency Analyzer" to diagnose and fix this "consistency gap," moving beyond average accuracy benchmarks to measure and improve an agent's ability to perform correctly every time.

The system works by analyzing an agent's past actions to find "flip-prone" decision points—steps where the underlying language model had low confidence between multiple choices. The Consistency Analyzer resamples these specific decisions to score their stability without re-running the entire task. These diagnostics are then used by IBM's open-source ALTK-Evolve framework to generate "consistency guidelines," which are injected back to the agent to ensure more stable and predictable behavior in future runs. On the AppWorld benchmark with a ReAct agent using GPT-4.1, the technique proved highly effective.

• Consistency Gap Reduction: The gap between average success (Mean@5) and all-run success (Pass^5) was halved from 24.4 percentage points to 12.0.
• Reliability Boost: The rate of succeeding on all five repeated runs (Pass^5) rose from 53.0% to 69.0%.
• No Accuracy Trade-off: Average task success (Mean@5) was maintained or slightly improved, rising from 77.4% to 81.0%.
• Generalization: The generated guidelines also improved reliability on different but related tasks, demonstrating they capture reusable patterns.

This work from IBM Research challenges the industry's over-reliance on average success rates as the primary measure of agent capability. By introducing and improving the Pass^k metric—the rate of success across all attempts—the framework provides a pathway for building more dependable agents for functions like financial reconciliation or contract analysis. It suggests that achieving enterprise-grade reliability requires targeted engineering to stabilize decision-making, an issue that simply using larger, more capable models does not inherently solve.

The conversation around AI agent performance is shifting from raw capability to enterprise-grade reliability. Tools like IBM's Consistency Analyzer signal a market maturation, where predictable, repeatable outcomes—not just high average scores on a leaderboard—are becoming the prerequisite for deploying agents in high-stakes business processes.