Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3

Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3

AI benchmarks can be incredible tools, but they usually only tell you if a model passed or failed. With ARC-AGI-3, however, we can see the thought process behind the score, not just the outcome.

This week we went through 160 replays and reasoning traces from OpenAI’s GPT-5.5 and Anthropic’s Opus 4.7 attempting novel, long-horizon environments. The scores were just one data point, but the interesting story is how they achieved their score.

Today we’re open-sourcing our analysis package.

Model
ARC-AGI-3 Score*
Public Demo Replays

GPT-5.5
0.43%
Link

Opus 4.7
0.18%
Link

• \ Scores tested with the semi-private dataset

With ARC-AGI-3 we can replay every action alongside the model's reasoning to see where it formed a hypothesis, where it abandoned a correct one, where it locked onto a wrong idea and couldn't let go.

We found 3 common failure modes:

• *True Local Effect, False World Model** - The models understand which action produced a change, but they fail to translate the effect into a global rule
• *Wrong Level of Abstraction From Training Data** - The models mistake an ARC-AGI-3 environment for another game
• *Solved The Level, Didn’t Learn The Game** - Even if a model beat a level, it’s unable to use that reward signal to enforce the correct actions

ARC-AGI-3 as an analysis tool

ARC-AGI-3 is a series of 135 novel environments. Each was hand-crafted by a human to test the ability of AI models to adapt to novelty. Play them yourself or watch our launch video.

The test-takers, whether human or AI, are not given instructions on how to play an environment. To make progress they must:
Explore unfamiliar interfaces
Infer rules from sparse feedback (aka world model)
Form & test hypotheses
Recover from wrong assumptions
Transfer what they learned from one level to the next (aka continual…