SWE-bench2,294 tasksLite · Verifiedcost · ambiguityMultilingual · PolyBenchbeyond PythonSWE-bench Proprivate repos
2023
2025
Sep 2026
Mar 2026
ARC-AGI-3 launches
no instructions · no stated goal · no win condition
humans 100% · frontier 0.51%
six months later
62.7%
GPT-6 Astra, standard harness
PATCHING THE OLD RULER — BUILDING A DIFFERENT ONE
The SWE-bench family kept absorbing its own limitations while a benchmark of another kind appeared alongside it.