GPT-6 Astra vs Claude and the Open Models: What the Benchmarks Don’t Tell You

OpenAI announced GPT-6 Astra with a line designed to travel: “Anything you can do on a computer, Astra can do for you. Fast.” The launch briefing went even further: “Welcome to the AGI era.”

OpenAI shipped GPT-6 Astra on September 3 with a line built to travel: “Anything you can do on a computer, Astra can do for you. Fast.” The launch briefing went further and called it the start of the “AGI era.” The benchmark table backed that up on paper: 99.9% on ARC-AGI-3, near-saturated math scores, a Critical rating on cybersecurity capability.

Look past the table and a different question shows up: how much of that number is the model, and how much is everything wrapped around it?

Two things shipped on September 3

OpenAI priced one thing and demoed another.

Astra costs $10 per million input tokens and $50 per million output tokens. Its API exposes a 1.05-million-token context window and up to 128,000 output tokens. GPT-5.6 Sol was cheaper at $4/$20. Claude Fable 5.1 lands at the same $10/$50 price point as Astra.

Those numbers compare cleanly because they belong to the model.

The harder part is everything around it. Alongside Astra, OpenAI also shipped an updated Codex harness with expanded async clarification workflows. The launch post says it directly: Codex updates “significantly improve the speed of computer use.” OpenAI credits that updated harness with a 1.9x faster task completion on Mind2Web, compared to the current Sol experience.

That’s not a pure model number.

OpenAI engineer Ari Weinstein added a detail the next morning: the same harness optimizations make GPT-5.6 Sol tasks roughly 60% faster too. The harness moves the score on its own, independent of which model sits behind it.

OpenAI hasn’t published the comparison that would isolate the two effects: Sol on the old harness, Sol on the new harness, Astra on the old harness, and Astra on the new harness. Without that 2x2, the 1.9x headline mixes model gains with product engineering and reports the sum as if it belongs to Astra alone.

The distinction that matters for the rest of this piece:

• The price page tells you what the model costs.
• The benchmark headline tells you what the model plus product engineering can do.

Where Astra actually stands on general intelligence

On OpenAI’s own September 3 table, Astra scored 61.2 on the Artificial Analysis Intelligence Index v4.1, close to Sol’s 60.9. Claude Fable 5.1 led at 65.7.

The launch-day story wasn’t “Astra is dramatically smarter than Sol.” It was closer to “Astra is roughly level with Sol on general intelligence, and Fable leads both.”

Artificial Analysis then released v4.2 under a revised methodology. Under the new index, Fable scores 57, Astra 55, Sol 51, Kimi K3 50, GLM-5.3 49, and DeepSeek V4 Pro 42. The benchmark changed. The conclusion didn’t: Astra doesn’t own the general intelligence leaderboard, at launch or after the methodology update.

Where Astra actually moved

Astra’s real story isn’t raw intelligence. It’s where Astra improved over Sol on specific, execution-heavy tasks:

• DeepSWE: 72.7% to 74.1%
• Terminal-Bench: 37.3% to 57.9%
• SRE-Bench: 55.9% to 88.0%
• MRCR (long-context retrieval): 73.8% to 96.3%
• FrontierMath Tier 4: 83.0% to 97.6%

The gains aren’t uniform. Math and long-context retrieval show the model itself clearly improved. The largest execution gains appear in tool-rich environments where Astra operates inside a terminal or a software system, exactly where the model and the harness around it are hardest to separate.

Comparing Astra to Claude requires reading the footnotes

OpenAI’s own table makes Astra look strong against Claude on computer use, cyber, math, and scientific terminal work. The comparison isn’t as clean as the table suggests.

Some Claude rows in OpenAI’s table use different evaluation settings. In several cases, standard Fable 5.1 refused the benchmark, so OpenAI substituted Mythos, a less-restricted Fable variant, to measure raw capability instead. In others, Claude results were omitted entirely or replaced with a fallback model because refusals made the comparison impractical.

That doesn’t make the results invalid. It means the label “Astra vs Claude” hides implementation detail that changes what the comparison is actually measuring.

Artificial Analysis makes this explicit on its Coding Agent Index. As pure model scores, Astra sits at 67 and Fable at 70. Inside their respective harnesses, GPT-6 Astra inside Codex scores 67, and Claude Fable 5.1 inside Claude Code scores 70. Artificial Analysis also reports the cost of running the complete agent systems: Astra uses roughly one-third of Sol’s tokens in Codex and costs about $4.72 per task, versus $9.18 for Fable 5.1 in Claude Code.

At the frontier, you’re no longer comparing models. You’re comparing models inside agent systems. Once the harness becomes part of the result, “Astra beat Claude” becomes far less precise than it sounds.

Open models: cheaper, closer, and just as harness-dependent

The strongest open-weight models make this uncomfortable for every closed lab.

On the current Artificial Analysis v4.2 index, Kimi K3 scores 58 and GLM-5.3 scores 49, against Astra’s 55. DeepSeek V4 Pro trails at 42. Their list API prices sit far below Astra’s: Kimi K3 at $3/$15 per million input/output tokens, GLM-5.3 at roughly $1.40/$4.40, and DeepSeek V4 Pro at $1.32/$3.96.

Kimi’s strength isn’t theoretical. It reached number one on Arena’s frontend coding leaderboard when it launched, and Artificial Analysis found it unusually strong on long-horizon agentic knowledge work. BenchLM currently estimates Astra at 81.88 and Kimi K3 at 79.65, with overlapping 90% confidence intervals: directional evidence for an Astra lead, not a settled win.

The open models don’t disappear the moment a task needs a harness. Kimi has real agentic evidence, and GLM runs explicitly through Claude Code-compatible tooling. Once Kimi runs inside its own agent framework, or GLM runs inside Claude Code, you’re comparing model-plus-system against model-plus-system, the same attribution problem that applies to Astra and Claude.

Astra still holds the strongest public evidence package in desktop computer use and frontier cybersecurity. But the frontier on raw intelligence is crowded, and the open-weight price gap is enormous.

Coding and computer use: where the benchmark stops measuring the model alone

Astra gained 20.6 points on Terminal-Bench and only 1.4 points on DeepSWE. Compress both numbers into one sentence, “Astra is better at coding,” and you lose the pattern that actually matters.

Terminal-Bench is explicitly an agent benchmark: the system inspects an environment, runs commands, sees what happened, recovers from mistakes, and keeps going. DeepSWE is repository repair; still agentic, but with a different task distribution and interaction pattern. The 20.6-versus-1.4 contrast doesn’t prove the harness caused the Terminal-Bench gain. It proves Astra’s coding improvement is highly task-dependent.

Context management tells a related story. Long Codex sessions used to rely on compaction: when context filled up, the session compressed earlier work into a summary so the agent could continue. Every compression risks losing something specific, like why a fix failed or which constraint mattered. With Astra, OpenAI introduced persistent notes that Codex can search directly, even when a detail was never copied into the summary. That’s a change in how the agent remembers, and it’s Codex behavior, not something every Astra API call automatically gets.

Computer use makes the same attribution problem sharper. On OSWorld 2.0, Astra reaches 72.6% against Sol’s 65.7%, with task time falling from roughly 75 minutes to 40 minutes. That’s real evidence the model itself is a better computer-use component. Then comes the bigger headline: 1.9x faster task completion on Mind2Web, which OpenAI credits to the Codex harness combined with Astra’s model-side efficiency. Separating the two effects would need the same 2x2 test described earlier: Sol on the old and new harness, Astra on the old and new harness, under identical settings. OpenAI hasn’t published it.

Astra may be a better computer-use model. The benchmark winner, on the evidence available, is the model plus the system around it.

Cybersecurity: the same model doesn’t mean the same capability

Coding shows how the harness changes what a model can accomplish. Cybersecurity raises a sharper problem: the shipped product can expose less capability than the model actually has.

Astra is OpenAI’s first model to cross the Critical cybersecurity capability threshold under its Preparedness Framework. With the right tools and access, OpenAI says it can find previously unknown vulnerabilities and build exploit paths across hardened systems. The benchmark numbers back that up: 100% on ExploitBench, sharp improvement on ExploitGym and SRE-Bench, and two previously unknown zero-day vulnerabilities discovered and used during evaluation.

Those aren’t normal-production capabilities. OpenAI’s own path-to-Astra documentation says the cyber results reflect Daybreak Blue access, not the default production configuration. Astra was also tested without production safeguards on some cyber benchmarks. The version available to normal users refuses advanced requests like proof-of-concept exploits, while Daybreak exposes broader defensive capability under tighter controls.

So “how capable is Astra at cybersecurity” now has three separate answers: what the model can do in evaluation, what standard production allows, and what authorized defenders can access under a different policy regime. Those aren’t the same product.

This changes how you should think about capability generally. The weights set one ceiling. Policy sets another below it. In high-risk domains, the better the model becomes, the stronger the restrictions around it may need to be. In ChatGPT or Codex, a risky task may pause for review. Through the API, it may stop outright. Under Daybreak, more capability can be exposed under tighter controls.

Same model family. Different policy. Different effective agent. For enterprises, the question isn’t “can Astra do this.” It’s “which version of that capability will my organization actually be allowed to use.”

Alignment: safer behavior, harder to watch

On OpenAI’s computer-use safety evaluation, unsafe behavior fell from 22.0% with Sol to 2.4% with Astra, before AutoReview is added. With AutoReview, Astra drops to 1.8%. The important change is the 22.0-to-2.4 shift: Astra’s training changed the model’s default behavior, not just the guardrails wrapped around it.

Better model behavior doesn’t remove the need for external safeguards. OpenAI still surrounds Astra with AutoReview, classifiers, monitoring, access controls, and automatic stopping mechanisms. Two layers do different jobs: safety trained into the model is more likely to travel when Astra gets used elsewhere; safety provided by AutoReview or OpenAI’s surrounding systems may not.

That distinction matters more once the model starts acting instead of just answering. A chatbot gives an answer and waits. An agent opens a file, runs a command, changes something, inspects the result, and continues immediately. A bad decision stops being bad text; it becomes an action that changes the environment. Safe reasoning alone isn’t enough for an agent. Safe execution matters too.

There’s one trade-off worth naming directly: OpenAI reports Astra behaves more safely across several evaluations, but its written reasoning can also be harder to monitor than Sol’s in some tests. Astra makes fewer unsafe decisions, and it may also reveal less about how it reached them. A single safety score doesn’t capture either fact on its own. The real questions are whether the model behaves better, whether the surrounding system catches what it misses, and whether you can still see what the agent is doing while it acts.

The number that captures the whole thesis

If you want one figure that proves the argument, it’s ARC-AGI-3.

OpenAI reports Astra at 99.9% on the benchmark. ARC Prize also tested Astra using its provider-neutral standard harness. Same model, same benchmark family, a completely different result:

• Standard harness: Astra’s best score was 62.7%.
• Provider Adapter harness: 99.9%.

That’s a 37.2-point difference. The conclusion isn’t that 99.9% is fake. It’s that the harness is part of the score, sitting right next to the weights in determining what number gets published.

Once you see that gap, the rest of the benchmark table reads differently. Some rows change the prompt; FrontierCode uses a Codex-style developer message. Some rows change the model; certain Claude results use Mythos or an Opus fallback when Fable refuses. Some rows change the evaluation rules entirely, with time limits, turn limits, and benchmark settings that aren’t always identical across models. OpenAI itself notes scores are reported at the best reasoning effort, and that research and API evaluations may differ from production ChatGPT because the tools and system prompts differ.

None of that makes the table invalid. It means the rows are often measuring different configurations, not one model against another on equal footing.

The old way of reading a benchmark table was simple: model to score. The accurate version now reads: model, plus prompt, plus reasoning effort, plus harness, plus tools, plus memory, plus limits, plus policy, equals score.

Independent evaluators are already adapting to this. Instead of treating an agent benchmark as belonging to the model alone, they increasingly name the system: Astra in Codex, Fable 5.1 in Claude Code. That’s more honest, because it tells you what was actually tested. The unit of competition is moving from the model to the system around it, which raises the real question: which parts of that system are you actually buying.

What you’re actually buying

Take Astra out of Codex, and what’s still yours?

The weights. The FrontierMath gain, the prompt-injection resistance work, the lower unsafe-behavior rate before AutoReview, and part of the long-context improvement are the strongest evidence that Astra itself changed. These are the capabilities most likely to travel when you move the model into another environment.

Inference configuration. Reasoning effort changes results. Token budgets change cost. Fast mode changes latency and price. OpenAI reports benchmark maximums across effort levels rather than one fixed production configuration, so the same weights can produce different cost, speed, and accuracy depending on how much compute you let them spend.

The harness. Codex contributes persistent notes, search over earlier context, computer interaction, tool orchestration, recovery behavior, and asynchronous work patterns. On long-running tasks, those features can matter as much as the model underneath them.

Policy. Cyber access, misalignment monitoring, AutoReview, pause-versus-stop behavior, and account-level permissions determine how much of the underlying capability actually reaches you.

This changes how you should read model migration. Suppose Astra scores 66 inside Codex and a different model scores 68 inside a different agent environment. Treating that as a two-point model-quality gap is tempting and wrong. Swapping one for the other also swaps context management, prompts, tool schemas, recovery logic, caching behavior, safety enforcement, token consumption, and how much human intervention the task needs.

At that point you’re not replacing an LLM endpoint. You’re replacing part of an agent operating system.

That’s the part of Astra’s launch that matters beyond the launch itself. Raw intelligence isn’t moving alone anymore. Claude, OpenAI, Kimi, GLM, and the other frontier systems are close enough on broad reasoning that infrastructure and product design increasingly decide who wins a real task. The more autonomous the task becomes, the more those upper layers matter.

Next time a benchmark table lands in your feed, don’t start with which model scored highest. Start with four questions:

• Which model version and reasoning setting actually ran?
• Which harness, prompts, tools, and memory system were in the loop?
• Which access and safety policy produced the published capability?
• How much of that exact system will exist in your production environment?

OpenAI may have built a better model; the FrontierMath gain and the drop in unsafe behavior before AutoReview back that up. But “better model” and “better system” stopped being the same claim. For the tasks that matter most, the system is increasingly what you’re buying.

Credits and Further Reading

1. OpenAI Developer Community, “Introducing GPT-6-Astra” https://community.openai.com/t/introducing-gpt-6-astra-the-most-intelligent-and-aligned-model-in-the-world/1394703

2. Axios, “Welcome to the AGI era, OpenAI says as GPT-6 Astra debuts” https://www.axios.com/2026/09/03/openai-astra-gpt-6-agi-brockman

3. OpenAI, “GPT-6 Astra: A new generation of intelligence” https://openai.com/index/gpt-6-astra/

4. OpenAI, “Path to Astra: critical capabilities and frontier safeguards” https://openai.com/index/path-to-astra/

5. OpenAI API, GPT-6 Astra model page https://developers.openai.com/api/docs/models/gpt-6-astra

6. OpenAI API, GPT-5.6 Sol model page https://developers.openai.com/api/docs/models/gpt-5.6-sol

7. Anthropic, Claude Fable 5.1 pricing https://www.anthropic.com/claude/fable

8. Ari Weinstein follow-up quoted in ThursdAI and mirrored social results: https://sub.thursdai.news/p/sep-3-part-2-welcome-to-agi-openai

9. Artificial Analysis, “Benchmarking GPT-6 Astra” — https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra

10. Artificial Analysis, GPT-6 Astra: https://artificialanalysis.ai/models/gpt-6-astra

Thanks for reading! If you have any questions or feedback, please let me know on Medium or LinkedIn

GPT-6 Astra vs Claude and the Open Models: What the Benchmarks Don’t Tell You was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.