20%40%60%80%
~70%*~70%*SWE-bench Verifiedpublic, familiar repos23.3%23.1%SWE-bench Pro — publicunified scaffold14.9%17.8%SWE-bench Pro — privaterepos never seen
THE FURTHER FROM FAMILIAR CODE, THE LOWER THE SCORE
Same two models, three evaluation protocols.
GPT-5
Claude Opus 4.1
*approximate — the source does not separate the two models on Verified