OpenAI’s headline claim for its GPT-6 Astra model did not survive contact with the people who built the test. The ARC Prize Foundation, which designs the ARC-AGI-3 benchmark to resist memorization and gaming, ran Astra through its standard, provider-neutral harness and scored it at 62.7 percent — still the highest result on the benchmark, but 37 points below the 99.9 percent OpenAI touted at the model’s September launch.
The gap comes down to the test setup. OpenAI’s 99.9 percent figure was produced with a custom tools-enabled harness that preserves the model’s reasoning state between steps, a configuration reported to cost nearly $19,000 per run. ARC Prize said that score is not valid under the conditions every other lab’s model must clear, because the standard harness is the one used for all comparable results.
That does not make Astra’s result unimpressive. When ARC-AGI-3 launched in March 2026, frontier models scored near zero on it. A verified 62.7 percent is a genuine leap — and a reminder that the software wrapped around a model can swing its score dramatically. Reporting on the benchmark’s October revisions notes that improved orchestration alone has pushed public scores sharply higher without changing model weights.
The lesson for buyers is practical: a single benchmark headline is not a buying decision. Ask how a score was produced, what harness and tooling were used, and test candidates like ChatGPT, Claude, and Gemini on your own workloads before committing. Our ChatGPT vs Claude comparison breaks down where the two flagships differ in everyday use.
Source: Startup Fortune
