Arc-AGI-3 Preview: 30-day learnings

ARC-AGI-3 Preview: 30-day learnings

Highlighting the gap between humans and AI with Interactive Benchmarks

ARC-AGI-3, the first Interactive Reasoning Benchmark by ARC Prize Foundation

On July 17, we released a sneak peek of ARC-AGI-3, our first Interactive Reasoning Benchmark. These three preview games were our first contact with reality using a new format: video-game-like environments where agents and humans must perceive, plan, and act across multiple steps to achieve long-horizon goals.

Our goal was to gather data about human performance, learn how well AI systems perform on interactive tasks, and see how the community would engage with this new style of evaluation. To encourage further development, we hosted an ARC-AGI-3 Preview Agent competition.

• *30-Day takeaways:**
• *Interactive benchmarks are easy (even fun) for humans, but hard for AI**

Most humans beat the games, often enjoying them. Persistent test-takers "speed ran" to theoretical minimums. AI agents struggled to efficiently make progress.
• *Some preview games were too friendly to random search**
A few game designs could be brute-forced without reasoning. What we learn will help make future games more resistant to brute force and more reflective of intelligence.
• *Action efficiency provides a clear intelligence signal**
Measuring how efficiently environment information is converted into strategy reveals a clear divide between human-level and AI-level performance.

Why Interactive Reasoning Benchmarks
Traditionally, to measure intelligence, static benchmarks have been the yardstick. However, Interactive Reasoning Benchmarks (IRBs) test for a broader scope of capabilities:

• *On-the-fly Learning** - Like in ARC-AGI-1 & 2, the test-taker cannot simply memorize strategies to succeed at the games, it must recombine what it knows on the fly to make sense of novel situations.
• *Exploration** - Can the test-taker efficiently gather the information it needs from the…