Context in 2025
At the end of 2025, frontier models already had surprising performance in software engineering. Claude Opus 4.1, GPT-5, and Gemini 2.5 Pro solved between 64% and 75% of SWE-bench Verified, a benchmark built from real GitHub issues, in which the model receives a repository and a bug report and must produce a patch that passes the project’s own tests. Claude Sonnet 4.5, released in September of that year, reached 82% (with parallel computation), the highest mark reported up to that point [1]. The gap between that and producing reliable code in real repositories was rapidly narrowing. Operating these systems, however, was expensive, and fixing a single bug cost between $0.85 and $1.70 in the top agents on the leaderboard [2].
This cost came directly from the way they worked. A competitive agent at the time operated in long cycles of searching, reading, modifying, and running the test suite, repeating the cycle until the tests passed or the budget ran out. Since there was no guarantee that the relevant part of the repository was inside the context window, the practical strategy was to expand the reading until it was, and the performance these systems achieved was real and bought through brute force.
What consumed these cycles was exploration. Trajectory analyses from the period put file discovery precision between 0.08 and 0.12, with agents opening eight to twelve times more files than those they actually modified [3], and each file opened by mistake went back into the context window and was paid for again in the next iteration. The conclusion I drew from this was that most of that cost was not being spent writing the fix, but looking for where the fix should be made.
Seen this way, the bottleneck was not just model capability, but context retrieval, that is, the task of selecting, from within a repository too large to fit in the window, only the relevant portion for the problem at hand. This turned the observation into a measurable question: how much of the advantage of frontier models in software engineering was reasoning capability, and how much was brute force compensating for the difficulty of finding the right context?
That question led me to the first Kodah experiment.
Why RAG Does Not Solve Code
The problem of how to retrieve the right context already existed in the industry and had a name since 2020: Retrieval-Augmented Generation [4].
Classic RAG works through semantic similarity. You split the corpus into chunks, generate embeddings, project the query into the same vector space, and retrieve the closest chunks. In documentation and knowledge bases this works well, because natural language carries meaning in the words themselves, and two texts about the same subject tend to use similar vocabulary. Proximity in vector space approximates proximity in meaning.
The first major problem is that code does not behave like narrative text, the premise of RAG breaks down, and trying to apply it without understanding its essence results in systemic failure. Variable, function, and class names are arbitrary, and the same concept appears as user, usr, accountHolder, or payload depending on who wrote it, in what year, and under which convention, sometimes within the same repository. The inverse also happens, because two functions or variables with nearly identical or generic names such as x can do things that have nothing to do with each other. The embedding of a code fragment ends up saying a lot about the style of the person who wrote it and very little about what it does, so semantic similarity starts measuring surface resemblance instead of behavior.
A second problem comes from the structure of the code itself. A part relevant to fixing a bug may be connected to the initial fragment through a function call, an import, or inheritance. It may be in another file and have practically nothing in common, in terms of words, with the description of the problem. Even so, the relationship between the two fragments is explicit in the program itself.
This problem of retrieving information from the relationships between data also already had its own line of research. Knowledge Graphs and, more recently, approaches such as GraphRAG, proposed by Microsoft Research in 2024 [5], began incorporating relationships between data into the retrieval process, inferred from the corpus, rather than relying only on similarity between texts.
It was at this point that my previous experience with Knowledge Graphs changed the way I looked at the problem. In code, relationships are essential and appear in the structure of the program itself, in imports, function calls, inheritance, and other dependencies. There was no need to infer these relationships and then use them for retrieval. Code was already a graph by nature.
Graphs and LLMs
Having the graph solved half the problem. The other half was getting an LLM to use it, and I already had a history with that.
In previous projects, I had tried the two most obvious approaches. The first was asking the model to translate the question into a query, generating Cypher from natural language. The error rate was high, and the worst kind of error was the most common: the query was syntactically valid, but logically wrong. A filter could be swapped or the syntax could belong to a different version from the one that was running, producing a plausible result that would only be noticed by someone who already knew the answer. The second approach was fine-tuning a small model specialized in this translation, but the result also did not reach a usable level.
These attempts showed me a more specific problem: I was asking the model to operate an interface whose output it could not validate. A tool such as grep or read_file returns something the model can inspect directly. A graph query is different. If the query is wrong, it may return too little, too much, or simply something plausible, without the model realizing that it searched in the wrong way.
That was why, in Kodah, I made a different decision: the model would not need to know that it was using a graph.
Instead of teaching the agent to write queries or navigate the structure, I created a layer of tools that received simple intents from the model and handled retrieval behind them. The agent asked questions about the repository and received the relevant context. There was no Cypher, query language, or graph vocabulary in the prompt. The complexity remained hidden in the architecture.
With this, I did not need to choose between classic RAG, in which the model does not participate in retrieval, and agentic RAG, in which it needs to figure out how to navigate the structure on its own. The model continued to drive the investigation, but the structure supporting that investigation remained invisible to it.
The first version of Kodah was built in Python with this idea at its center: a small agent, with a few tools, capable of investigating a repository without needing to know how the structure behind them worked.
The First Result: SWE-bench Lite
I took a model much smaller than the frontier models of the time and put this architecture to work on a real bug-fixing benchmark.
On SWE-bench Lite [6], with all 300 issues running in the official Docker harness and using GPT-5-mini, the system solved 51.0%, at $0.045 per issue, $13.59 for the entire benchmark [7]. The leaderboard leader at that point was Claude Opus 4.6, with 62.7% at around $1.70 per issue [8].
That is approximately 81% of the frontier resolution rate at 1/38 of the cost. The result showed that a small model could recover a significant part of the performance gap when the architecture reduced the work required to find the relevant context.
The model remained responsible for reasoning, planning, and patch generation. What the architecture changed was how it found the code it needed to reason about, delivering selected context from the repository structure instead of simply delivering more context.
There was also a characteristic of this experiment that does not appear in the table: Kodah did not execute the code it wrote. There was no test loop, execution feedback, stack trace reading, or iteration based on a failed test. The system investigated the repository, decided where the defect was, wrote the patch, and submitted it blindly.
The most competitive coding agents of the time depended heavily on execution, and not only to verify what they wrote. Running the suite and reading the stack trace is also a form of navigation, because the failure points to a file and a line. Without this mechanism, all the work of reaching the right place had to come from the structure.
The 51% does not isolate retrieval as the only source of the difference from frontier models, and that was not the claim. What it shows is that a relevant portion of the work of solving those issues could be handled by the architecture before the model wrote the first line of the patch.
Beyond Python: SWE-PolyBench Verified
It remained to determine how much of that result depended on the models’ familiarity with Python. The original SWE-bench was heavily concentrated in that language, and multilingual benchmarks began showing a considerable drop in performance when the same type of problem was taken to other ecosystems. On SWE-bench Multilingual, Claude 3.7 Sonnet reached 43% resolution, compared with 63% on Verified [9].
This led me to the next experiment: SWE-PolyBench Verified [10], from Amazon Science, covering Python, Java, JavaScript, and TypeScript across 21 industrial repositories, with independent evaluation.
With GPT-5-mini, Kodah solved 28.27% of the complete set, placing 4th on the leaderboard at that point. The scores by language were 32.74% in Python, 26% in TypeScript, 25% in JavaScript and 28.99% in Java. Localization precision was 0.77, above recall at 0.67, at around $0.10 per issue [11].
The difference between languages also caught my attention. In Python, Kodah ranked fourth. In TypeScript and JavaScript, it ranked third, ahead of Amazon Q Developer Agent in both languages. On class-scope bugs, it ranked second globally, with 72.73% versus 54.55% for Amazon Q.
The drop in performance outside Python was expected. What interested me was that Kodah’s relative position did not follow this drop: it moved from fourth place in Python to third in TypeScript and JavaScript, precisely in languages where the models’ familiarity advantage was smaller.
Localization precision helped explain this result. Kodah’s 0.77 directly measured the step that had motivated the project: finding the relevant code. The trajectory analyses of agents from that period found precisions between 0.08 and 0.12 [3]. The contrast did not prove that localization alone explained the entire performance difference, but it showed that frontier systems were spending a huge amount of computation exploring the repository before reaching the code they needed to modify.
The experiment was beginning to show that the result of a coding agent was not an isolated property of the LLM, but rather the product of the combination of model, context, tools, search strategy, number of iterations, and execution environment. This combination was changing rapidly.
Evolution of the Models in 2026
During 2025 and early 2026, increasing model capability produced important leaps in coding and agentic reasoning. Claude Opus 4.5 established frontier results in software engineering, and a few months later 4.6 brought explicit improvements in debugging, code review, planning, and long-duration agentic tasks, in addition to a 1-million-token context window in beta.
At the same time, model architecture began to change in important ways. One of the main directions was Mixture-of-Experts (MoE): models with many total parameters, but which activate only a portion of them for each token. This makes it possible to increase model capacity without proportionally increasing the cost of each inference.
Qwen3-Coder 480B-A35B-Instruct [12] is a good example. It has 480 billion total parameters, but only 35 billion active per token, with an MoE architecture and a native context of 256K tokens. The model has a much greater total capacity than what needs to be activated at each step.
This idea appeared across different model families throughout 2026, with increasingly sparse architectures. Kimi K3 reached 2.8 trillion parameters with 104 billion active, activating 16 of 896 experts per token, and DeepSeek V4-Pro operates with 1.6 trillion and 49 billion active. GLM-5.3 showed that the gain did not always come from growing, delivering about a 50% improvement in coding over its predecessor through post-training alone, while maintaining 744 billion parameters and 40 billion active [13]. The gap between open-weight models and the proprietary frontier also began to narrow in programming and tool-use tasks.
This had an interesting relationship with what I had done in Kodah. I was trying to reduce cost by putting efficiency outside the model, in the architecture surrounding it. The labs were doing something similar inside the model, deciding where to spend capacity and where not to spend it. The difference is that when this efficiency enters the weights themselves, it becomes part of the capability of any system built on top of that model.
And the models were not only getting better at writing code. They were getting better precisely at the capabilities that Kodah had needed to compensate for: using tools, maintaining context, planning long tasks, and deciding which information to search for during execution.
The industry itself began treating this as a system-engineering problem. Anthropic began using the term context engineering to describe the selection of contextual state that should be available to the model at each stage [14], and introduced mechanisms such as Tool Search and Programmatic Tool Calling to avoid putting all tools and their definitions directly into the context [15]. The problem was no longer only which model to choose, but how to organize the context, tools, and state around it.
The change also appeared in the format of products. Claude Code, Codex, Antigravity, and GitHub Copilot began occupying the space that previously belonged to systems that received an issue and returned a diff. The object of evaluation became the complete system: model, context, tools, terminal, memory, execution, retrieval, planning, compaction, feedback, and interaction.
Evolution of the Benchmarks in 2026
When models change at this speed, the measuring sticks change along with them. A benchmark is not a neutral measure of capability, but a snapshot of the constraint that the field considered relevant at the moment it was built.
In a few years, SWE-bench multiplied into versions: Lite, Verified, Multilingual, PolyBench, and Pro. Each one emerged to address a limitation exposed by the previous one and, in doing so, changed somewhat what was being measured.
The accumulated effect appears in the results. On SWE-bench Verified, GPT-5 and Claude Opus 4.1 exceeded 70%. Under a unified scaffold on SWE-bench Pro [16], both fell to around 23%. On the private subset of Pro, consisting of repositories neither had seen, they fell again, to approximately 15% for both. At each step that moved the evaluation away from public and known code, the score decreased.
Contamination is only part of this problem. It is not necessary to train directly on benchmark data to adapt a model to it: similar data and selection among many configurations can also shift the evaluation. The Llama 4 case made this public, with 27 private variants tested before selecting the version presented on LMArena, which subsequently showed lower performance in independent evaluations [17].
Saturating a measuring stick, therefore, does not distinguish between two possibilities: the models became better at that task or learned to adapt to that specific set. The way to separate the two hypotheses is to change the evaluation.
In March 2026, the ARC Prize Foundation launched ARC-AGI-3 [18], radically changing the type of task. The agent enters an interactive environment without instructions, declared rules, a described objective, or a victory condition. It needs to explore, infer the mechanics, and discover on its own what counts as success. Humans solve 100% of the environments. Frontier models, at launch, scored 0.51%.
The difference from SWE-bench is fundamental. In SWE-bench, the problem reaches the agent already described and accompanied by the repository. In ARC-AGI-3, not even the objective is provided, and the operational definition of the problem needs to be constructed during execution. This characteristic also appears in evaluations launched in 2026, from Terminal-Bench to AutomationBench and Agents’ Last Exam: the context necessary to solve the task is not simply delivered ready-made.
ARC-AGI-3 itself showed how quickly a new measuring stick can age. The record went from 0.51% in March to 7.8% with GPT-5.6 Sol, 30.2% with Claude Opus 5 in July, and 62.7% with GPT-6 Astra in September, under the standard ARC Prize harness. OpenAI reported between 98.6% and 99.9% using its own adapter.
From that point on, the question is no longer just how much the model can do. It is also what the harness allows it to do.
Harness and Evaluation
A harness is the software that runs a model as an agent: it decides which tools the model can call, what goes into its context at each step, how its reasoning carries across turns, and when the task ends.
In July 2026, OpenAI itself published a result that makes this difference concrete [19]. GPT-5.6 Sol appears on ARC-AGI-3 with three results: 7.8% in the ARC Prize record, measured on the semi-private set, and, on the public set, 13.3% under the official harness in OpenAI’s measurement and 38.3% when that configuration enables reasoning preservation across turns and context compaction. The model was the same in all three measurements. What changed was how and where execution was conducted.
The difference between 13.3% and 38.3% came from how the agent preserved its state throughout the task. In the 38.3% configuration, reasoning was maintained across turns and the context was compacted to allow a longer execution. The agent’s working memory became an explicit part of the harness.
In August 2026, NVIDIA published SoL-Pi [20] with a different question: instead of manually deciding how an agent should execute a task, would it be possible to automatically discover which changes to the harness would improve its efficiency?
The system created and evaluated 152 possible modifications to the agent’s operation. The proposals changed different parts of execution, such as how actions were performed, how received observations were handled, and how context was managed. The modifications were tested on software-development tasks, and were only kept when they could reduce execution cost without exceeding the defined performance-loss threshold. In the end, four mechanisms were selected and combined into a new harness, SoL-Pi.
The final test was conducted on EdgeBench, a separate set of 51 long-duration coding-agent tasks, kept outside the search from the beginning. With GPT-5.6 Sol and Claude Opus 5, the resulting harness preserved 93.7% and 94.3% of the reference harness performance, reducing token traffic by 44.7% to 49.0% and API cost by approximately one third.
The important point is the change in object: the research was not only about which model to use, but how to organize the execution of that model. The harness itself had become part of the optimization problem.
The Current Bottleneck
Models became better at long context, tool use, planning, and execution of prolonged tasks. As a result, some of the limitations that motivated Kodah ceased to be the main constraints of coding agents. Context retrieval is a clear example.
In the first systems, a natural approach was to retrieve the most relevant fragments beforehand and deliver them to the model. Vector RAG did exactly that. In more advanced systems, retrieval began happening during execution: the agent itself decides what to search for, calls tools such as glob and grep, reads the results, and chooses the next fragment to investigate. Retrieval ceased to be a step preceding reasoning and became part of reasoning itself.
This change was possible because models became better at conducting long sequences of actions and because the cost of maintaining this context fell. Cache reading, which represents an important part of the cost of an agentic loop, became much cheaper. In September 2026, Anthropic reduced its price by 75%, from $1.00 to $0.25 per million tokens in model Claude Fable 5.1 [21]. Re-reading 200,000 tokens per turn went from costing $0.20 to $0.05.
The combined effect of these changes was to drastically increase the amount of work an agent can execute. And that is precisely where the next bottleneck appears.
In September 2026, Anthropic described its own software engineering [22]: Claude was already writing about 80% of the code reaching production, while engineers were producing approximately eight times more code per quarter than during the 2021 to 2025 period. The number of tests had grown tenfold and CI jobs 25-fold in six months, with only a nominal increase in team size.
The problem was no longer just making the agent produce code. Everything it was producing had to be verified.
A concrete example was Anthropic’s test impact analysis service. Its function was to determine, after a code change, which tests needed to be run. As the volume of changes increased, this service began to fail under the new scale. The team tried to fix it three times: the first solution lasted 70 days, the second 29 days, and the third less than a day. The definitive solution required a rewrite that consumed three weeks of one engineer’s work.
This case shows the change in scale. When agents produce little code, finding the relevant code and writing the change are central problems. When they begin producing code at a much greater volume, testing, verifying, and processing the consequences of those changes begin to consume a larger share of the system’s capacity.
And there is an interesting technical connection to the problem that opened the article. Test impact analysis also depends on the structure of the code: to know which tests may be affected by a change, it is necessary to traverse the dependencies leading from that change to the rest of the system.
The bottleneck, therefore, did not disappear. It shifted along the chain: first finding the code, then producing the change, and now verifying and absorbing what agents are capable of producing.
Agentic Systems in 2026
In 2026, the advancement of agentic systems ceased to be only a matter of better models. At the beginning of the period, it was reasonable to look at a coding agent primarily as a model to which we added context and tools. Today, that description is already too small. The result depends on the interaction between model, retrieval, memory, tools, harness, execution, and verification, and the boundary between these layers is moving.
This also changes how we decide what should be solved by the model and what should remain deterministic. There is no longer a stable division. What needed to be built externally yesterday may be absorbed by the model tomorrow. At the same time, when scale increases, operations that seemed too simple to justify their own infrastructure once again require deterministic mechanisms, because they become part of the system’s control and verification layer.
For this reason, I no longer see agent architecture as a choice between an LLM or external structure. The problem becomes deciding, for each stage, where the model’s flexibility adds value and where the predictability of a deterministic mechanism is indispensable. This boundary needs to be able to move without requiring the entire system to be rebuilt.
The same change appears in evaluation. Public benchmarks remain important for measuring the field’s progress, but they cease to be sufficient to define the quality of a system. As models saturate a measuring stick and harnesses begin to directly interfere with the result, the relevant unit ceases to be the isolated model. It is the complete system, subjected to tasks that represent the work it actually needs to perform.
In practice, this changes how a team should evaluate and invest in agentic systems. Across the systems discussed here, the cost of resolving an issue varied by a factor of almost forty for a difference of about ten points in resolution rate, and a harness change alone cut a third of the cost with the same models. Cost is as much a property of the architecture as of the model. Public benchmarks also say little about how a system will behave on a specific codebase: the same models that solved more than 70% of SWE-bench Verified fell to around 15% on repositories they had never seen. The measure that matters is the cost per task actually resolved, on the organization’s own code.
The same logic applies to where the investment goes. When agents write most of the code, the limiting resource stops being the number of developers or agent licenses and becomes the capacity to test, review and integrate what is produced. At Anthropic, the service that decides which tests to run was the first to break under that volume. Knowing which parts of a codebase depend on which, and which tests a change can affect, becomes infrastructure that serves twice: first to guide the agent to the right place, and then to verify what it wrote without running everything.
This is the system view I take from 2026. The challenge is no longer simply to make the model more capable. It is to build an architecture that knows where to place that capability, where to impose structure, and how to move that boundary as the model evolves.
References
[1] Anthropic. Introducing Claude Sonnet 4.5 (29 set. 2025). https://www.anthropic.com/news/claude-sonnet-4-5
[2] Silas Liu. Kodah: SWE-bench Lite. https://www.silasdata.com/kodah/
[3] Kim, M., Wang, D., Cui, S., Farmahinifarahani, F., Garg, S., Ray, B., Zhuo, T. Y., Mukherjee, R., & Kumar, V. (2026). TRAJEVAL: Decomposing Code Agent Trajectories for Fine-Grained Diagnosis. arXiv:2603.24631. https://arxiv.org/abs/2603.24631 — note: v2 (May 2026) was retitled “Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code”
[4] Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401. https://arxiv.org/abs/2005.11401
[5] Edge, D. et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research. arXiv:2404.16130. https://arxiv.org/abs/2404.16130
[6] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770. https://arxiv.org/abs/2310.06770
[7] Silas Liu. Kodah: SWE-bench Lite. https://www.silasdata.com/kodah/
[8] SWE-bench Lite leaderboard. https://www.swebench.com/lite.html
[9] SWE-bench Multilingual. https://www.swebench.com/multilingual.html
[10] Rashid, M. et al. (2025). SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents. Amazon Science. arXiv:2504.08703. https://arxiv.org/abs/2504.08703
[11] Silas Liu. Kodah: SWE-PolyBench (Python). https://www.silasdata.com/kodah-swe-polybench-python/ and Kodah: SWE-PolyBench (Overall). https://www.silasdata.com/kodah-swe-polybench-overall/
[12] Qwen Team. Qwen3-Coder-480B-A35B-Instruct (model card). https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct
[13] AlphaCorp. Top LLMs: benchmarks, pricing, picks. https://alphacorp.ai/blog/top-llms-benchmarks-pricing-picks
[14] Anthropic. Effective context engineering for AI agents. https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
[15] Anthropic. Introducing advanced tool use on the Claude Developer Platform (Tool Search Tool and Programmatic Tool Calling, Nov. 2025). https://www.anthropic.com/engineering/advanced-tool-use
[16] SWE-bench Pro (2025/2026). https://labs.scale.com/leaderboard/swe_bench_pro_public
[17] Almeida, D. Lies, damned lies, and benchmarks. The Complete Skeptic, Sep. 2026. https://www.completeskeptic.com/p/lies-damned-lies-and-benchmarks
[18] ARC Prize Foundation. ARC-AGI-3 (Mar. 2026). https://arcprize.org/
[19] OpenAI. How enabling two settings tripled our scores on the ARC-AGI-3 benchmark (29 jul. 2026). https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/
[20] Liu, H., Ye, T., Gao, S., Cao, Q., Li, Y., Zhuge, M., Wang, D., Zhang, R., Luo, P., Bian, J., Zhu, L., Zhu, L., Xie, E., & Han, S. (2026). SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness. NVIDIA, NTU, MIT. arXiv:2609.20519. https://arxiv.org/abs/2609.20519 Blog: https://nvlabs.github.io/SoL-Pi/ Code: https://github.com/NVlabs/SoL-Pi
[21] Anthropic. Introducing Claude Fable 5.1 and Claude Mythos 5.1 (Sep. 2026). https://www.anthropic.com/claude-fable-and-mythos-5-1
[22] Malhotra, S. Agentic coding is straining CI. Here’s how we scaled test impact analysis at Anthropic. Anthropic, Sep. 14, 2026. https://claude.com/blog/agentic-coding-is-straining-ci-heres-how-we-scaled-test-impact-analysis-at-anthropic