Dispatch
IBM Research ships consistency guidelines for agents, lifting AppWorld pass^5 from 53.0% to 69.0% without losing mean accuracy
IBM's ALTK-Evolve post targets the gap between an agent's average accuracy and its reliability across repeated runs: a ReAct agent scored 77.4% mean accuracy but succeeded on all 5 runs for only 53.0% of AppWorld tasks, a 24.4-point consistency gap. Their Consistency Analyzer replays each decision step with k=5 controlled resampling to find flip-prone decisions without full task replays or ground truth, then converts those into guidelines injected at inference time. Results: pass^5 53.0% to 69.0%, mean@5 77.4% to 81.0%, consistency gap halved to 12.0pp, and +13.0pp transfer to similar tasks. Code is on GitHub at AgentToolkit/altk-evolve with a technical report at arXiv 2609.08832.
↳ Follow the thread