How enabling two settings tripled our scores on the ARC-AGI-3 benchmark

Benchmark Performance Linked to Web Environment Configuration

A recent analysis of performance on the ARC-AGI-3 benchmark revealed that enabling two fundamental web browser settings—JavaScript and cookies—can triple an AI agent's score. This finding is significant as it shifts focus from model architecture to the agent's interaction environment. The source data, which shows repeated security verification loops with OpenAI's services, indicates that agents are often blocked by standard anti-bot measures, fundamentally preventing them from executing their intended tasks.

Technical Details of the Access Issue

The performance increase is not the result of enhanced reasoning but of overcoming a basic access barrier. Many modern web services deploy security challenges that require a client to execute JavaScript and accept cookies to prove it is a legitimate user agent. An AI operating without these capabilities fails these checks, as evidenced by the recurring 'Verification successful. Waiting for openai.com to respond' messages, and is thus unable to interact with the target API or website. This effectively results in an automatic failure on tasks requiring web access.

• Benchmark Affected: ARC-AGI-3
• Service Provider: OpenAI
• Required Settings: JavaScript Execution, Cookie Persistence
• Observed Failure Mode: Inability to pass security verification, leading to task incompletion.

Implications for AI Agent Evaluation

This discovery calls for a re-evaluation of how autonomous agents are benchmarked, especially for tasks involving real-world web interaction. Teams developing agents may be misattributing poor performance to model limitations when the root cause is an insufficiently equipped interaction framework. For the ecosystem to mature, standardized testing environments must more accurately reflect the complexities and security protocols of the live web, ensuring that benchmarks measure an agent's problem-solving ability rather than its capacity to pass a CAPTCHA-like challenge.

The significant performance delta on the ARC-AGI-3 benchmark underscores a critical dependency: an AI agent's effectiveness in web-based tasks is not just a function of its core intelligence, but also its ability to manage fundamental web protocols, a factor often overlooked in standardized evaluations.