Benchmarks
Astra’s published scores, with the testing conditions that matter.
GPT-6 Astra
Agents' Last Exam
Complete professional projects in real software, judged against expert deliverables.
Launch evaluation; do not compare directly with the public leaderboard’s full-task pass rate.
59.3%reported scoreOSWorld 2.0
Carry out multistep work in desktop and browser applications.
Offline set v2026.08.08. Partial credit is not the same as completing every step of a task.
72.6%partial-credit scoreTerminal-Bench 4.0
Solve tasks using a terminal, real tools, and software environments.
Published leaderboard result: 58.2% ±2.8. The model and agent setup are evaluated together.
58.2%tasks resolvedFrontierMath Erdős
Prove or disprove open Erdős problems with verified Lean proofs.
One attempt per problem, with a $300 budget and 72-hour limit. Higher-budget follow-ups are separate.
3%2 of 68 problemsARC-AGI-3
Explore unfamiliar games, infer their rules, and learn to solve them.
99.9% with the Provider Adapter at high effort. The two setups manage context differently; neither proves AGI.
62.7%standard setupFrontierMath Tier 4 (v2)
Research-level mathematics.
97.6%v2 scoreGPQA Diamond
Graduate-level science questions.
96%accuracyARC-AGI-1
Infer abstract rules from examples of grid puzzles.
98.5%verified scoreARC-AGI-2
A harder generation of abstract-reasoning grid puzzles.
95%verified score
No published Astra result
METR Time Horizon
Task difficulty in human work hours, at a predicted 50% success rate on software tasks.
No published Astra estimate. METR flags estimates above 16 hours as unreliable with its current suite. This is human task time, not agent runtime.
~17.4 hrs50% success estimateSWE-bench Pro
Fix real software issues so hidden tests pass.
No Astra result listed on Scale’s public leaderboard. This is the highest listed score in that evaluation setup.
61.5%tasks resolved