TwinEdgeAI

↑ Course home

← Previous: Week 9a, SWE-bench (Guest Lecture) ·  Next: Week 11, Proactive Agents →

Engineering AI Agents · Week 10

Coding & Software Agents

How a coding agent works from the issue to the submitted patch: the loop, the interface the agent uses, the scaffold around the model, the pipelines that skip the loop, the tests that serve as its eyes, the harnesses that keep it working across many sessions, what it costs, and what the evidence says about its effect on real developers. A lab builds a small version of each part with no model and no network.

Author: Grok Bot Version: 1.0, October 29, 2026 Study time: about 17 hours

The slides were not availableThis lecture is on Monday, November 30, and I wrote this pack on Thursday, October 29, 2026. I checked the course's slide folder once, at about 9:00 PM Pacific time. Its index listed lecture01.pdf to lecture11.pdf, and the four addresses I tried for the weeks after that, lecture18.pdf to lecture21.pdf, each returned "not found" (HTTP 404) [22]. So I have not seen the slides. This pack is built from the schedule's one-sentence description of the lecture, its two required readings and two additional readings, and the public sources that I list. I do not know what the instructors will say. Nothing in this pack is a quotation from them or a summary of their talk, and where I describe what a source says, the source is named.

A name and an order to checkThis file is called week10 because it is the tenth pack that I wrote. The schedule that I read on October 29, 2026 puts this lecture in its Week 11, on Monday, November 30, under the heading "Coding Agents & Proactive Agents". The lecture after it, on Wednesday, December 2, is on proactive agents, and it is not in this pack. The two Thanksgiving days, November 23 and 25, are listed as "No Class" under Week 10 [21]. The "Previous" link above goes to the previous page of my pack, Week 9a, which was a guest lecture on Monday, November 16. In date order, the previous lecture is Week 8b, Agent Safety, on Wednesday, November 18. The schedule's one deadline on the lecture's own day is that the two paper video peer reviews are due on Monday, November 30, at 11:59 PM Pacific time [21].

CreditThis coursework is adapted from, and inspired by, Stanford CS329Z: Engineering AI Agents (Diyi Yang, Michael Ryan, John Yang), the lecture "Coding & Software Agents" (Monday, November 30). The schedule describes it as: "How coding agents work end to end: the SWE-agent, Claude Code, and OpenHands architectures, scaffolds as design decisions, SWE-bench and the 4-tuple framework in practice, and the future of software development with agents." It lists two readings, the SWE-agent paper (NeurIPS 2024) and the OpenHands paper (ICLR 2025), and two additional readings, Anthropic's "Claude Code: Best Practices for Agentic Coding" (2025) and Young's "Effective Harnesses for Long-Running Agents" (Anthropic, 2025) [21]. The topic and the readings come from the course's public schedule. Everything else here is my own work: the notes, the choice of added sources, the lab, the quiz and the assignment. This is not an official Stanford document.

1. Week overview and learning objectives

Week 9a looked at the benchmark that most coding agents are scored on. This lecture looks at the agents themselves. The schedule's description has four parts, and the notes follow them [21]. The first is how coding agents work end to end: the loop of reading, editing, running and submitting, and the three named designs, SWE-agent, OpenHands and Claude Code (sections 2.1 to 2.5). The second is scaffolds as design decisions: the choices around the model, such as the interface, the loop, the context policy and the stopping rule, and the alternatives that skip the loop (2.2, 2.6 and 2.7). The third is SWE-bench and the 4-tuple framework in practice: how a task's request, environment, stopping criteria and scorer show up in a real coding agent and in the tests it uses to check itself (2.1 and 2.8). The fourth is the future of software development with agents: long-running work, cost, and what the evidence says so far about developers (2.9 to 2.12). I cannot say what the instructors will emphasize. The additional readings suggest that Claude Code's practices and long-running harnesses will come up [3] [4].

The lab uses the standard library only. It does not call any model, any API, Docker or the network. It builds on the Week 9a toolkit and adds: an agent-computer interface (a windowed file viewer, a capped search, a linter-gated edit command and a protected-path rule); a scripted agent that works only through that interface; an Agentless-style pipeline that localizes, samples patches, validates them with a reproduction test made from the issue, and picks one by voting; a repo map built from the ast module and ranked by links between files; a long-running harness with a feature list, progress notes, checkpoints and context-window sessions; and cost and turn accounting. Every task, agent, price, probability and budget in the lab is made up by me. The scripted agents are programs, not language models. The lab is a model of how the parts fit together. It tells you nothing about how well any real agent does.

Learning objectives

After this week you should be able to:

1. Describe the loop of a coding agent (find, read, edit, run, check, submit) and say where each part of the 4-tuple, the request, the environment, the stopping criteria and the scorer, appears in it. [2.1]
2. List the design decisions of a scaffold and say which ones a benchmark score silently includes. [2.2]
3. Name the parts of SWE-agent's agent-computer interface and say what the paper's ablations found about the editor, the search commands and the file viewer. [2.3]
4. Describe the three parts of OpenHands (the agent abstraction, the event stream and the runtime) and how it reuses ideas from CodeAct and SWE-agent. [2.4]
5. Explain the difference between an instruction that the model may ignore and a rule that the harness enforces, with examples from Claude Code, and say what AGENTS.md is for. [2.5]
6. Describe Agentless's three phases and AutoCodeRover's code search, and say why a fixed pipeline can be competitive with an agent loop. [2.6]
7. Explain a repo map, a file skeleton and a capped search as ways to spend a context budget. [2.7]
8. Explain how a reproduction test, a visible test suite and a vote work together to select a patch, and when voting does not help. [2.8]
9. Describe the two failure modes that the long-running harness post reports and the pieces it uses against them. [2.9]
10. Explain why the cost of an agent loop grows faster than its number of turns, and what caching, budgets and turn limits do about it. [2.10]
11. State what the METR study, its 2026 update, SWE-Lancer and the pull request studies each measure, and what each cannot tell you. [2.11]
12. Build, break and compare the parts of a small coding agent yourself. [lab]

How to use this pack

Time	What to do
3.5 hours	Read the notes (section 2). Skim the figures first.
3.5 hours	Read the two required papers with the reading guide questions, then the two additional readings.
5 hours	Do the lab and at least five exercises.
1.5 hours	Discussion and the quiz.
3 hours	The graded assignment.

2. Lecture notes

The slides were not available (see the top of the page), so these notes follow the schedule's topic sentence [21], the readings it lists and the public sources that I added. The section titles and the order are mine. I do not know which of these points the instructors will cover. Where a number comes from a source it is that source's own claim, and where it is my own arithmetic or from my lab, I say so.

2.1 A coding agent from issue to patch

A coding agent is a language model that is run in a loop with a computer. Each turn the model reads what it has seen so far, chooses one action (open a file, search, edit, run a command), the computer carries it out, and the result goes back to the model as the next observation. SWE-agent describes its loop this way: at each step the model produces a thought and a command, and the output of the command is added to what the model sees next [1]. Claude Code's documentation describes three phases that blend together: gather context, take action, and verify results, with the model reasoning and tools acting [20]. OpenHands describes an agent as a function that takes the current state and returns the next action [2]. The three descriptions are the same loop. In this pack I use six verbs for it: find, read, edit, run, check, submit.

Take a task of the kind that Week 9a covered. The request is an issue text, for example "startswith_any() looks at the end of the string". The agent searches for the function name, opens the file near the match, runs a one-line example to see the wrong answer, edits one line, runs the example again, runs the visible tests, and then submits. The submitted thing is a patch, the same unified diff that SWE-bench grades [5]. Demo section 1 of the lab runs exactly this task through a small interface, with one deliberate slip: the first edit has a syntax error, the interface refuses it, and the agent tries again.

The 4-tuple in a real coding agent

Week 7b described a task as four parts: a request, an environment, stopping criteria and a scorer (the 4-tuple). The table shows how the same four parts look in a benchmark run and in everyday use of a tool such as Claude Code. The right-hand column is what the lab uses.

Part	SWE-bench style run	Everyday use of Claude Code	In the lab
Request	The issue text, as given [5]	Your prompt, plus CLAUDE.md (or an AGENTS.md that it reads in its place), and what the agent reads in the repository [20]	An issue text for a toy task
Environment	The repository at the base commit in a container, with the agent's tools [1] [5]	Your project, your terminal and your git state, locally, in a cloud machine or through remote control [20]	An in-memory repository and the interface of section 2.3
Stopping criteria	The agent submits, or a limit is hit. SWE-agent used a $4 budget per instance and submitted the existing edits if a run went over [1]	"Claude stops when the work looks done." Limits are optional: --max-turns and --max-budget-usd with the non-interactive mode, or a spend limit [3]	Submit, a turn limit and a made-up price
Scorer	Hidden FAIL_TO_PASS and PASS_TO_PASS tests, all of which must pass [5]	None built in. Tests, a typecheck, a Stop hook, CI, and a person who approves the merge [3]	Hidden tests, as in Week 9a

The last row is the one to remember. A benchmark has a scorer that the agent never sees. In everyday use there is no hidden scorer, and the Claude Code best-practices page says what follows from that: "Claude stops when the work looks done. If it has nothing to run that tells it whether the work actually is done, 'looks done' is all it has to go on." [3]. So a large part of the practical craft is giving the agent something to check against, which is the subject of 2.8. The other half of the 4-tuple, the stopping rule, matters because the cost of a run is set by how long it goes on (2.10).

2.2 Scaffolds as design decisions

The scaffold is everything around the model: the tools it can call and how their output is shown, the loop that calls the model, what the model sees on each turn, where code runs, what stops the run, and what is kept between runs. A benchmark number such as "resolves 18% of SWE-bench Lite" is a number for a model and its scaffold, its budget and the harness that scored it. Week 7b made this point in general (scaffolds), and Week 9a showed how the SWE-bench leaderboards deal with it, with one view that fixes the scaffold and others that let each team bring its own [17]. This lecture's sources are mostly about the choices in the scaffold. I group them into seven decisions.

Decision	The question	Examples from the sources
Interface	Which actions exist, and what does each one return?	SWE-agent's few purpose-built commands [1]; bash only in mini-SWE-agent [17]; code actions in OpenHands and CodeAct [2] [8]; AST-based search in AutoCodeRover [7]
Loop	Does the model decide each step, or does a fixed program decide the steps?	A free loop in SWE-agent and Claude Code; a fixed three-phase pipeline in Agentless [6]
Context policy	What does the model see on each turn, and what is dropped?	SWE-agent keeps the last five observations in full [1]; Claude Code compacts the conversation into a summary when the window fills [20]; Aider builds a ranked map of the repository [9]
Environment	Where does code run, and what can it damage?	An OpenHands sandbox in Docker [2]; Claude Code locally, in the cloud or by remote control, with checkpoints and permission modes [20]
Verification	What tells the agent that it is done or wrong?	Visible tests; reproduction tests written from the issue in Agentless [6]; a Stop hook in Claude Code [3]
Memory across runs	What survives when a context window ends?	CLAUDE.md and auto memory [20]; a feature list, a progress file and git commits in the long-running harness [4]
Budget and stopping	What ends the run, and who pays?	A $4 budget in SWE-agent [1]; turn and spend limits in Claude Code [3]

Three consequences follow. First, comparisons are confounded. The OpenHands paper compares itself with SWE-agent on a code-repair benchmark and notes in its table that its own agent is run with no examples in the prompt ("0-shot") whereas the SWE-agent number is "1-shot", so the gap mixes the effect of the agent with the effect of the prompt [2]. That is a small example of a problem that is everywhere. Second, a small scaffold can be competitive: mini-SWE-agent is a roughly 100-line agent that gives the model only a bash tool, and its README claims a score above 74% on SWE-bench Verified, which I did not check [17]. The honest reading is that the scaffold matters, and also that the model matters, and that a single leaderboard row cannot separate the two. Third, every decision has a cost side: a richer interface costs the designer effort and the model tokens to read its documentation, a longer context costs money on every turn, and a looser stopping rule costs more when the agent wanders.

Questions to ask of any coding-agent result: Which model, and which scaffold? What was the budget or turn limit? What was in the prompt (examples, tips)? How many runs, and was the best one reported? Who ran it? These are the same questions that Week 7b asked of any evaluation.

2.3 SWE-agent and the agent-computer interface

Required reading [1]. SWE-agent's central idea is that a language model is a new kind of user of a computer, and that the interface should be designed for it, as a graphical interface is designed for a person. The authors call this an agent-computer interface (ACI). They write that a good one should help the model "understand the state of the application given previous changes, manage history to avoid unnecessary context from prior observations, and provide actions that models can use efficiently and reliably" [1]. They hold the model fixed and design the interface, using two methods on a development set: reading the agent's behaviour to find difficulties, and a grid search over interface settings.

The paper states four design principles that came out of this work [1]:

1. Actions should be simple and easy to understand. Many bash commands have dozens of options, and simple commands with concise documentation are easier to use.
2. Actions should be compact and efficient. Important operations such as navigation and editing should be a few actions, not many small ones that must be composed over several turns.
3. Feedback should be informative but concise. After an edit the agent is shown the changed region, but not unnecessary detail.
4. Guardrails mitigate error propagation and hasten recovery. A syntax check that rejects a bad edit is the example.

The concrete interface has four parts. Search and navigation commands (find a file, search a directory, search a file) return a list of matches, and the list is capped at 50 results, with a message that asks the agent to narrow the query if there are more. A file viewer shows a window of at most 100 lines, with commands to go to a line and to scroll, and a line that says how many lines are above and below. An edit command replaces a range of lines, runs a linter on the result, and if the file would no longer parse it discards the edit and shows the agent the error together with how the file would have looked. And a context policy keeps only the last five observations in full, collapsing older ones [1]. The agent can also run ordinary bash commands. The paper reports 12.47% of the full SWE-bench test set (2,294 tasks) and 18.0% of SWE-bench Lite with GPT-4 Turbo, with a $4 budget per task [1].

What the ablations found

The paper changes one part of the interface at a time on Lite (300 tasks, GPT-4 Turbo). Figure 2 shows the numbers from its table [1]. The default has linting, summarized search, a 100-line viewer and the last five observations, and resolves 18.0%.

Part	Default (18.0%)	Variants, as % resolved on Lite [1]
Edit	Edit with linting	Edit without linting 15.0 (3.0 lower); no edit command, the agent uses bash instead, 10.3 (7.7 lower)
Search	Summarized search, capped	Iterative search 12.0 (6.0 lower); no search commands 15.7 (2.3 lower)
Viewer	100 lines	30 lines 14.3 (3.7 lower); the full file 12.7 (5.3 lower)
Context	Last 5 observations	Full history 15.0 (3.0 lower); no demonstration 16.3 (1.7 lower)

Two readings are worth keeping. First, the interface can hurt as well as help: iterative search, which shows one result at a time, did worse than having no search commands at all, because the agent kept paging through results, which the authors say can use up the budget or the context [1]. Second, be careful with small gaps. Lite has 300 tasks, so a 3.0-point gap is 9 tasks. My own arithmetic: a 95% Wilson interval around 18.0% with 300 tasks is about plus or minus 4 points, which is wider than the gaps for linting, the context policy and the no-search variant. I do not know the run-to-run spread of each cell, because I read the table and not the paper's repeated-run analysis, so I would treat the large gaps (no edit, iterative search, full-file viewer) as more believable than the small ones. The lab's own ablation (below) is a toy in which the effect is built in, so it cannot settle the question either.

One more number shows why the guardrail is there: in the paper, 51.7% of the trajectories (1,185 of 2,294) had at least one failed edit, an edit that the linter rejected, and the odds of recovery fell as failed edits accumulated [1]. The paper also reports that 93.0% of the instances it resolved were submitted before the budget ran out, against 69.0% of all instances, which suggests that raising the budget would not rescue many of the failures [1].

The lab version

The lab's Workspace class implements the same four parts for the toy repositories: a windowed viewer, a capped search, a linter-gated edit and a protected-path rule that blocks edits to test files. The scripted AciAgent works only through it. The linter has three modes: none, syntax only (the check SWE-agent uses) and names (syntax plus undefined names, my own extension, a stand-in for a stricter linter). Because the agent is a program, I give it a made-up chance of 0.4 of producing a broken edit on any attempt, and compare the modes on 60 tasks (20 curated tasks from Week 9a and 40 synthesized ones) [5].

The made-up slips are of two kinds: an unclosed bracket (a syntax error) and a mistyped name (code that parses and fails when it runs). In the lab, with slips switched on and no linter, 20 of 60 tasks are resolved at $1.659 per task. With the syntax linter, which can catch only the first kind, it is 25 of 60 at $1.625. With the names linter, which catches both, it is 35 of 60 at $1.342, the same 35 of 60 as an agent that never slips, which costs $1.102 (the difference is the extra turns spent retyping). This is a toy. The size of the effect is set by my 0.4 slip chance and by my choice of slips, and the "names" mode is built to catch exactly the slips I planted. What carries over is the shape of the argument: a guardrail turns a silent corruption into an error message that costs one turn, and a check that is matched to the common mistakes recovers more of them. The viewer-window sweep in the same demo shows the cost side, with a 5-line window costing $0.630 per task and a 100-line window $0.730 on the 20 curated tasks, at the same 14 of 20 resolved.

2.4 OpenHands: a platform

Required reading [2]. SWE-agent is mainly about one interface. OpenHands is a platform, an open-source system that is meant to host many agents, many tasks and many users. The paper's title is "OpenHands: An Open Platform for AI Software Developers as Generalist Agents" [2]. The course schedule words it a little differently, as "OpenHands: An Open Platform for AI Software Developers" with no "as Generalist Agents", and I could not tell whether that is a shortened title or an older one, so check the schedule when you read. The paper has authors at Carnegie Mellon University and at All Hands AI, which maintains the platform [2].

Its design has three main parts.

• The agent abstraction. An agent is something with a step function that takes the current state and returns the next action. The state includes the history of actions and observations. This makes it easy to write a new agent without touching the rest of the platform [2].
• The event stream. Every action the agent takes and every observation it gets, including a user's messages, is an event in one chronological stream, which is what an agent reads to decide what to do next and what an interface can display [2].
• The runtime. Each session runs in a Docker sandbox. Inside it a small server exposes a REST interface for executing actions, and the sandbox has a bash shell, an IPython (Jupyter) server and a Chromium browser. Any Docker image can be used as the starting point [2].

The actions are few and general. The paper names a command action (CmdRunAction, a bash command), a Python action (IPythonRunCellAction) and a browser action (BrowserInteractiveAction), plus an action that delegates a subtask to another agent (AgentDelegateAction). The default agent, CodeActAgent, is built on the CodeAct idea that the agent's actions should be executable code, not a fixed menu of tool calls [8]. A library called AgentSkills adds editing and navigation utilities to the Python environment, and the paper says that its file-editing utilities were adapted from SWE-agent and Aider [2] [9]. So OpenHands is partly a re-use of the ideas in 2.3 and 2.7, put behind a different interface: code the agent writes, instead of commands the agent types.

On SWE-bench Lite the paper reports its CodeActAgent (version 1.8) at 26.0% with claude-3.5-sonnet, at an average cost of $1.10 per task, and 22.0% with gpt-4o at $1.72, and it lists other systems on the same table, including SWE-agent at 18.0% and AutoCodeRover at 19.0% [2]. Do not read this as "OpenHands beats SWE-agent", because the models differ, the prompts differ (2.2), and the table collects numbers that were reported by different teams at different times. The authors of the paper are also the maintainers of the platform. The paper is a good place to see how a general platform is organised, and it is not the place to learn which agent is best.

2.5 Claude Code, Codex CLI and AGENTS.md

Additional reading [3] [20]. Claude Code is a coding agent that runs in your terminal, and also in editors, a desktop app, a browser and CI [20]. Its documentation is useful because it says plainly what the scaffold is made of. The page "How Claude Code works" says that the loop is powered by two parts, "models that reason and tools that act", and that "Claude Code is the layer around the model that provides the tools and manages the context the model sees", which "is what the term agentic harness refers to" [20].

A note on the sources. The schedule lists "Claude Code: Best Practices for Agentic Coding" (2025). I could not retrieve a 2025 article under that name. The old web address now shows the current documentation page, "Best practices for Claude Code", which has no date and is a later revision that is longer than anything I can attribute to 2025 [3]. So everything below about Claude Code is from the live documentation as I read it on October 29, 2026, and it may differ from what the instructors will show. Anthropic wrote the documentation and sells the product, and I did not run the tool.

What the documentation says the scaffold contains

• Tools in five categories: file operations, search, execution (shell commands, tests, git), web, and code intelligence (type errors after edits, jumps to definitions, which needs plugins). It also has tools for starting subagents and for asking the user questions [20].
• Context management. The context window holds the conversation, file contents, command output, CLAUDE.md, memory, loaded skills and system instructions. When it fills, the conversation is compacted into a summary, and the page warns that "detailed instructions from early in the conversation may be lost", so persistent rules belong in CLAUDE.md. Skills load on demand. A subagent works in its own context window and returns a summary, so its tool calls stay out of yours [20].
• Sessions. Each message, tool use and result is saved to a plain JSONL file, which allows resuming and forking. Sessions are independent, and a new one starts with a fresh context window. Before an edit the files are snapshotted, so that you can rewind (checkpoints), but checkpoints do not cover actions on remote systems [20].
• Permissions. There are modes that range from asking before every edit and command, to accepting edits, to a planning mode that does not edit, to an "auto" mode in which a classifier reviews actions in the background [20].
• CLAUDE.md and AGENTS.md. CLAUDE.md holds project instructions that are loaded every session. If a repository has an AGENTS.md "for other coding agents", Claude can read that in place of a CLAUDE.md [20].

Instructions the model may ignore, and rules the harness enforces

The best-practices page draws the line that matters most for scaffold design. Everything you tell the tool is one of two kinds. A non-deterministic instruction, such as your prompt, CLAUDE.md or a skill, is read by the model, which "usually follows your commands", but "there's no guarantee". A deterministic one, such as a hook, a deny rule or a managed setting, is carried out by the program "every time, no matter what the model decided to do" [3]. The page gives a table that pairs the two for each need, and I list four rows of it.

Need	Model decides (may be ignored)	Program enforces
What it can touch	The auto mode classifier	Deny rules, the sandbox in strict mode, managed settings
When it is done	A request for evidence, /goal, /code-review	Tests, a typecheck, a Stop hook, CI
How much it spends	Effort level, the model's own pace	--max-turns or --max-budget-usd with the non-interactive mode, a spend limit
Who has the last word	The model's summary of what it did	Branch protection, a person who approves the merge

The page gives a worked example of the Stop hook: a short entry in the settings file runs the typecheck when the agent tries to stop, and an exit code of 2 blocks the stop and sends the compiler output back to the model as the reason [3]. It also advises that a review done "by someone who didn't write the code" is better, and that the /code-review command reviews the diff in a fresh subagent "that never saw your conversation" [3]. The lab's protected-path rule is the same kind of thing in miniature: an edit to a test file is refused by the interface, whatever the agent intended.

Codex CLI and the AGENTS.md convention

The Codex CLI README says that it is "a coding agent from OpenAI that runs locally on your computer", under an Apache-2.0 license, and that it can be used by signing in with a ChatGPT account [11]. I read the repository's front page and not the code, so I can say what it is, not how it works. The schedule also mentions Cursor's agent, but I could not verify any Cursor documentation for this pack, so I say nothing about it.

AGENTS.md is a file convention. The site describes it as "a simple, open format for guiding coding agents" and says to "think of AGENTS.md as a README for agents": a predictable place for build steps, test commands and style rules that would clutter a README for people [10]. It lists many supporting tools, among them Codex, Aider, Cursor, Gemini CLI, Zed and VS Code, and it says that over 60k open-source projects use it, which is the site's own claim and which I did not check [10]. Its example is a Markdown file with headings such as "Setup commands" and "Code style" and short bullet lists, and the lab has a parser for that shape (parse_agents_md) that pulls out the sections and the backticked commands. The file is an instruction and so, by the table above, it belongs on the model-decides side: it improves the odds that the agent runs the right test command, and it does not enforce anything.

2.6 Pipelines versus loops: Agentless and AutoCodeRover

Not every system that fixes issues is a free loop. Agentless asks whether the loop is needed at all [6]. It has three fixed phases and no decision about which tool to use next:

1. Localization, from coarse to fine. First, the model is shown a tree of the repository's file names and the issue, and ranks the most suspicious files; a simple embedding-based retrieval adds more files. Then it is shown a skeleton of each top file (the headers of its classes and functions) and names the classes and functions to examine. Last, it picks the exact lines to edit [6].
2. Repair: ask for the change as a Search/Replace edit and sample many candidates, 40 patches per bug in the reported setup [6].
3. Patch validation: write a reproduction test from the issue, run candidates against the existing tests and the reproduction test, and pick the best by a vote. The reproduction tests are labelled by their output ("Issue reproduced", "Issue resolved", "Other issues") and the most common normalized test is kept [6].

On SWE-bench Lite it reports 32.00% (96 of 300) at $0.70 per task with GPT-4o, and says that this is the highest among the open-source systems that it compared, at low cost [6]. The authors also examine Lite itself and report that 4.3% of its tasks include the exact patch in the issue text, 10.0% lack information and 5.0% are misleading, so a pipeline can be doing well on tasks that are easier than they look [6]. AutoCodeRover sits in between [7]. It runs an agent, but the agent's tools are code-aware: searches over classes and methods in the program's syntax tree, with a fault-localization step that uses test results. It reports 19% on Lite at about $0.43 per task with GPT-4 (ISSTA 2024), in the paper's table of 37k tokens on average [7].

Be careful comparing costs across papers. The Agentless paper's table gives $2.51 for SWE-agent with GPT-4, while the OpenHands paper's table gives $1.67 for SWE-agent [6] [2]. I did not find out why. The models, the dates and the way cost is counted may differ. So the cost of a system in one paper's table should be compared with another system in the same table only.

What can a fixed pipeline do better? It is easier to debug, because each phase can be checked on its own (a localization either named the right place or it did not), and it can be cheaper, because a phase is one or a few model calls and not thirty or forty turns of growing history. What can a loop do better? It can decide what to look at after seeing the first result, run the code, and handle the tasks that do not fit a fixed recipe. The Agentless authors argue that the complexity of agents is often unnecessary on SWE-bench-like tasks [6]. That is a claim about a benchmark and about the model at the time, and the authors of the paper wrote both the pipeline and the argument.

The lab builds both on the same 60 tasks as 2.3, so that the cost can be compared under one accounting (demo section 4). The pipeline localizes in two steps, files and then code units, both ranked by BM25 (a classic text-matching score, not a model), samples up to eight candidates from a made-up distribution, and validates them with the visible tests and a reproduction test made from the issue. The loop is the AciAgent with the best linter.

Lab system (60 tasks)	Resolved	Model calls or turns	$ per task	$ per solved task
Agent loop (ACI)	35	15.7	1.102	1.890
Pipeline, no selection	20	4.0	0.097	0.290
Pipeline, first passing patch	27	5.0	0.107	0.239
Pipeline, vote	27	5.0	0.107	0.239

In this toy the loop resolves more tasks (35 against 27), and the pipeline costs about a tenth as much per task. Both results are partly artifacts of how I built the toy. The loop in the toy can retry with different edits, which the pipeline can only do by sampling; and the loop's cost is high because I count every check as a turn and charge for the whole history on every turn (2.10), a charge that the pipeline avoids by making fewer calls. A real comparison would depend on the model, which the toy does not have. The sweep over the number of samples shows the more general point, that a pipeline can buy accuracy with samples: the count resolved goes from 20 of 60 with one sample to 36 of 60 with 32, and the cost per task from $0.051 to $0.299, in this toy world and with this sampler.

2.7 Context: search, skeletons and repo maps

A real repository is far bigger than a context window, and a model that is shown everything is slower, more expensive and, as SWE-agent's ablations suggest, sometimes worse (2.3). So every coding agent has a context policy: a way of deciding which parts of the repository the model sees. The sources in this pack use five different ones.

Policy	How it works	Source
Capped search	Search commands return at most 50 results and ask the agent to narrow the query if there are more	SWE-agent [1]
Skeleton	Show the headers of the classes and functions in a file, not their bodies, and ask the model which ones to open	Agentless [6]
Code-aware search	Search by class and method names over the program's syntax tree, and use test results to rank suspicious code	AutoCodeRover [7]
Repo map	A compact map of the most important classes and functions in the whole repository, with their signatures, fitted to a token budget	Aider [9]
Read on demand	The agent searches and reads for itself, and the harness compacts the conversation when the window fills. Claude Code says that it starts each task by reading the code, and that skills and subagents keep material out of the main context until needed	Claude Code [3] [20]

The Aider repo map is the most explicit of these, and the lab builds a small version of it. Aider's documentation says that the map shows "the most important classes and functions along with their types and call signatures", so that the model can see the shape of the whole repository and ask for the files it needs. To fit a large repository it runs a graph-ranking algorithm on a graph "where each source file is a node and edges connect files which have dependencies", and keeps the most important parts that fit the active token budget, which defaults to 1k tokens and is adjusted dynamically. It expands the map when no files have been added to the chat [9]. I read the documentation page, and not the code, so I know the idea but not the details of its ranking.

The lab's repo_map works in four steps. It reads each file with Python's ast module and lists the classes, functions and methods with their signatures (symbol_index). It counts which names are used in which other files, to make a graph of files (file_graph). It ranks the files with a PageRank-style score over that graph (pagerank), with a boost for files that you say you are working on. And it fills the map in rank order until the token budget runs out. On the synthetic repository, a budget of 40 tokens shows 3 of 13 files and 9 of 36 symbols, 120 shows 9 files and 26 symbols, and 400 shows 12 files and all 36 symbols (the thirteenth file has no symbols). The most central file by the score is pkg/m00.py (0.39), which fits how the generator is built: it makes early modules hubs that many later modules import. This is a stand-in: Aider handles many languages with a parser library, and mine handles Python only and uses my own weights.

Does a map help find the right place? The lab compares two indexes for finding the buggy function on the 20 curated tasks: an index of whole function bodies, and an index of signatures only, which is what a tight map shows. When the issue names the function, both find it 19 times of 20 (0.95), so the comparison says nothing. When I replace the function's name in the issue with the word "it", bodies find the right file first on 11 of 20 tasks (0.55) against 9 of 20 for signatures (0.45), and find the function in the top three on 9 of 20 against 7 of 20 (0.45 against 0.35). The gaps are 2 tasks of 20, which is within noise, and the tasks are small and the issue texts are mine. I read it as a direction, not a result: when the issue describes behaviour and not names, the bodies carry words that the signatures do not. That is the case for letting the agent open files after the map, and not stopping at the map.

2.8 Tests as the agent's eyes, and choosing a patch

An agent that edits code has to know whether the edit worked. There are four sources of that knowledge, in rising order of effort, and each is a different kind of check.

1. Run the issue's own example. If the issue shows a wrong output, run it before and after. It is cheap and it is what the lab agent does first.
2. Run the visible tests. They catch regressions (the idea of PASS_TO_PASS tests), but they usually do not test the new behaviour. The lab keeps the task's own FAIL_TO_PASS tests hidden, as in Week 9a, so the visible tests can pass while the issue is still unresolved.
3. Write a reproduction test. Agentless asks the model to turn the issue into a script that prints "Issue reproduced" on the buggy code and "Issue resolved" when it is fixed (with "Other issues" for anything else), and uses it as a second check on every candidate patch [6]. The lab's repro_script does it for toy tasks, and repro_quality asks the question that matters, whether the generated test fails before the gold patch and passes after. For 18 of the 20 curated tasks it does, and for 2 the issue has no runnable example, so there is no reproduction test to make (demo section 5).
4. Ask for evidence, or ask someone else. Claude Code's page says to "ask Claude for evidence": the command it ran and its output, not "all tests pass" by itself, and to have a fresh subagent or a second session review the diff [3].

With several candidate patches, tests can also choose one. Agentless first filters by the existing tests and the reproduction tests, then picks by majority vote over normalized patches [6]. Test-time scaling work trains a model to do this job: SWE-Gym (ICML 2025) releases 2,438 task instances with executable environments, trains agents and verifiers on them, and reports 32.0% on Verified and 26.0% on Lite with its fine-tuned agents [15]. I read its abstract and the sections on the data, not the verifier experiments. The lab compares seven ways of choosing, on the 20 curated tasks, with 5 seeds, and with a made-up sampler that skews toward early candidates (demo section 6). The last row is an oracle, the best possible choice among the candidates, which is the ceiling for any selector.

In the lab, always taking the first candidate resolves 6.0 of 20 and picking at random 4.2. Choosing by the visible tests raises that to 8.4. Adding the reproduction test raises it to 11.2, which is the oracle ceiling for those candidates, so the selector loses nothing. Voting did not beat "first passing" in my toy: 7.4 against 8.4 with visible tests, and 11.2 against 11.2 with the reproduction test (at 16 samples with a flat sampler, voting is 12.0 against 12.6). I report that as it came out. My explanation, which I did not test, is that the toy's wrong patches are popular, because the sampler repeats the same plausible edits, so votes reward them. The real papers argue the other way [6], and my toy is not evidence against them. What it shows is that voting rests on an assumption (right answers are repeated more than wrong ones) which should be checked, not assumed.

2.9 Long-running work: harnesses and clean state

Additional reading [4]. A task that takes hours cannot fit in one context window, so the agent has to work in sessions, and each new session begins with no memory of the last. The post "Effective harnesses for long-running agents" (Justin Young, Anthropic, published November 26, 2025) compares it to a project staffed by engineers in shifts, where each new engineer arrives with no memory of the previous shift [4]. (The schedule gives the author as "Young", and the post's author is Justin Young.) Compaction is not enough: the post reports that a coding model run in a loop on a high-level prompt such as "build a clone of claude.ai" tended to fail in two ways [4].

1. Doing too much at once. The agent tried to build the whole app in one go, ran out of context in the middle, and left a half-implemented, undocumented feature for the next session.
2. Declaring victory early. After some features existed, a later session looked around, saw progress and called the job done.

A third problem appears in the testing section: the agent marked features as complete without testing them end to end, for example with unit tests or curl calls that passed while the feature did not work in the browser [4]. The post's fix has two kinds of session. An initializer agent runs once and sets up an init.sh script, a progress file and an initial git commit, and writes a feature list: over 200 entries in the post's example, each marked as failing at first, with a field passes that later sessions are told to change and nothing else (it picked JSON because the model was less likely to damage it than Markdown). A coding agent then works in each later session on one feature, commits with a descriptive message, and writes a summary in the progress file, so that it can use git to revert a bad change. Each session starts by reading the progress file and the git log [4]. The post calls the target a clean state, the kind of code that would be appropriate for merging to a main branch [4]. For testing, it reports that the agent did much better once it was told to use browser automation and test as a human user would, with some bugs still invisible to the tools [4]. The post describes what the author's team tried, and it reports no numbers.

The lab turns this into a small experiment with a crash. A project has several planted bugs, each with a hidden regression test, and these are the "features". The work runs in sessions, each limited by a made-up context size, and with a made-up chance (0.3 per attempt) that a session is cut off in the middle of an edit and leaves a half-written file. Four setups are compared on 5 toy packages and 5 seeds (25 runs, 100 features in all).

Setup	Features passing (of 100)	Sessions per run	$ per run	Existing tests still pass
Bare: no feature list, notes or commits	0	1.0	0.01	25 of 25 runs
Feature list only	14	7.5	6.44	4 of 25 runs
Feature list and progress notes	14	4.3	3.17	4 of 25 runs
List, notes and commits	73	3.8	3.73	25 of 25 runs

The pattern matches the post's failure modes. The bare setup checks the visible tests, which pass, and declares the job done in the first session with nothing fixed: "declaring victory early". The feature list fixes that, since the agent now knows what remains, but a crash leaves a half-written file that the next session inherits, and in 21 of 25 runs the earlier passing tests were broken as well. Progress notes alone make the run cheaper (4.3 sessions instead of 7.5) because the agent stops retrying a feature that has failed twice, but they do not repair the broken file. Commits are what restore a clean state: a failed or cut-off attempt is reverted to the last good commit, and 73 of 100 features pass with the existing tests intact in every run. As the crash chance grows from 0 to 0.3 to 0.6, the full setup passes 80, 73 and 47 of 100 and needs 2.6, 3.8 and 5.7 sessions: a harness makes a crash cost more sessions, not fewer features, until the crash is very frequent. All of this is built into the toy. I chose the crash model and the limits, and the toy cannot show what the post's real agents do.

2.10 Cost, turns and budgets

In a chat-style loop, every turn sends the whole history to the model again. If turn 1 sends a prompt of P tokens and each turn adds g tokens (the model's output plus the tool's result), then turn t sends P + g(t-1) tokens, and the input over T turns is about P·T + g·T(T-1)/2. The second term grows with the square of the number of turns. This is why the cost of an agent run grows faster than its length, and it is one reason that context policies exist: SWE-agent keeps only the last five observations in full, and Claude Code compacts the conversation when the window fills [1] [20].

My arithmetic from the lab's settings (a 600-token starting prompt; each turn adds 310 tokens): after 10 turns the input total is 19,950 toy tokens, after 20 it is 70,900, and after 30 it is 152,850, so a run of three times as many turns costs 7.7 times as much input. Providers offer prompt caching, where a repeated prefix of the prompt is billed at a lower price. In the lab I model it with a made-up price of 0.1 times: the 30-turn run costs $6.40 with no caching, $3.65 with half of the input cached and $1.45 with 90% cached, at made-up prices. I did not check any provider's current prices or caching rules, and the Claude Code documentation has a page about it that I did not read, so use the figures as a shape and not as a price list.

Three controls follow, and the best-practices page has all three [3]. A turn limit and a spend limit (--max-turns and --max-budget-usd in the non-interactive mode) stop a run that is going nowhere. Separate worktrees give parallel sessions their own checkouts so that their edits cannot collide. And a subagent works in its own context window and returns only a summary [20], so its long history is not carried by the main session (it is still paid for). In the SWE-agent paper the $4 budget had a measurable effect on the way the run ends: 93.0% of the resolved instances were submitted before the budget ran out [1], which the authors take to mean that a larger budget would not rescue many of the failures.

Cost per solved task is more informative than cost per task, because a cheap agent that solves little is not cheap. In the lab's table in 2.6 the loop costs $1.890 per solved task and the pipeline $0.239, a ratio of about 8 (my arithmetic, 1.890 divided by 0.239). The ratio reflects my accounting choices. What generalises is the question to ask of any reported number: which price list, how many turns, and was the cost per task or per solved task? Cross-paper costs do not line up, as 2.6 showed.

2.11 What the evidence says about developers and tasks

A benchmark tells you what an agent does on a fixed set of tasks. The questions people care about are different: does it make developers faster, can it do paid work, does the code survive review? The sources here answer different parts of those questions, and none of them answers all of them. I read the METR sources and the SWE-Lancer paper in more detail than the two pull request studies, for which I read only the abstracts.

Source	What it measured	What it cannot tell you
METR randomized trial, early 2025 [12]	16 experienced open-source developers did 246 tasks in mature projects where they had on average 5 years of experience. Each task was randomly assigned to allow or disallow AI tools (mostly Cursor Pro with Claude 3.5 or 3.7 Sonnet). Developers forecast that AI would cut completion time by 24%, estimated afterwards that it had cut it by 20%, and the measured result was that it increased completion time by 19%	Anything about other developers, other projects or later tools. The authors say that the influence of experimental artifacts cannot be ruled out entirely, and test 20 properties of the setting as possible explanations [12]. Sixteen people is a small group
METR update, February 24, 2026 [13]	A second study from August 2025 with 57 developers (10 from the first study, 47 new), paid $50 an hour and not the earlier $150. Estimated change in task time: -18% (95% interval -38% to +9%) for the returning developers and -4% (-15% to +9%) for the new ones. In METR's sign, -18% means that tasks took 18% less time with AI. The first study's figure is +19% (+2% to +39%) [13]	METR says that this data is an "unreliable signal" of the current effect. Developers who would not work without AI dropped out, and 30% to 50% of the developers said that they did not submit tasks that they would not want to do without AI, so the estimate is likely biased toward too little speedup. The time measurements are also unreliable when a developer runs several agents at once. METR is changing the design [13]
SWE-Lancer (OpenAI) [14]	1,488 freelance jobs worth $1 million in real payouts, from one open-source repository (Expensify) posted on Upwork. It includes independent coding tasks that are graded by end-to-end tests, and management tasks in which the model chooses between proposals. A public Diamond set holds $500,800 of the tasks. The best model of the paper's time, Claude 3.5 Sonnet, scored 26.2% on the coding tasks and 44.9% on the management tasks and earned $208,050 of the $500,800 [14]	Anything about other codebases, or about the value of the work to a client. The paper's models are from early 2025, and the benchmark's authors work at OpenAI, which builds such models. Payouts weight the tasks by price, which is not the same as weighting them by difficulty
Terminal-Bench 2.0 [16]	89 hard command-line tasks verified by three reviewers. The authors report that frontier models and agents resolve fewer than 65% of them [16]	Anything about maintaining a repository. It tests a different thing from SWE-bench, which is working in a terminal on tasks such as building and configuring software
Automated code review in industry [18]	A study of an AI review tool (based on the open-source Qodo PR Agent) with about 238 practitioners; 1,568 of 4,335 pull requests in three projects had automated reviews. 73.8% of the tool's comments were resolved, and the average time to close a pull request rose from 5 hours 52 minutes to 8 hours 20 minutes, with different trends in different projects [18]	Whether the resolved comments were worth resolving. I read only the abstract
Agent-written pull requests on GitHub [19]	567 pull requests written with Claude Code in 157 open-source projects: 83.8% were eventually merged, and 54.9% of the merged ones had no further changes. The authors note that the rest needed human revision [19]	Quality. A merge says that a maintainer accepted the change, and the sample is of projects whose pull requests say they used the tool. I read only the abstract

Four lessons come from reading these together. First, a measured result can go out of date quickly. The METR slowdown is the most quoted number in this area, and the same group says that its newer data, with newer tools, is hard to read, and that developers are probably more sped up now (it calls the data "very weak evidence" on that) [13]. Do not cite the 19% as if it described tools today. Second, a study can break its own sample. When developers will not work without the tool, the people and the tasks that remain are the ones who gain least [13]. Third, the units differ. A benchmark counts resolved tasks, SWE-Lancer counts dollars, the pull request studies count merges and closure time, and METR counts minutes. A rise in one is not a rise in the others, and the 19% slowdown and a high merge rate can both be true. Fourth, who measured matters. The METR work is independent of the tool vendors, the SWE-Lancer authors build models, and the two pull request studies are from academic groups and an industrial study [12] [14] [18] [19].

The lab does not measure any of this, and it has no developers in it. It is relevant in one way only: it shows that the cost and the success of an agent depend on choices that a trial of "AI on or off" does not vary, such as the interface, the checks and the budget (2.3 to 2.10). A result for "AI tools" is a result for a particular bundle of those choices.

2.12 The future of software development: what to watch

The schedule ends the lecture's topic list with "the future of software development with agents" [21]. I do not know what the instructors will argue, and the evidence in 2.11 does not settle it. So instead of a forecast, here are six things in the sources that I would watch, each tied to a claim in this pack, and each with a sign that would change my mind.

What to watch	Why, from this pack	What would change my mind
Where the scorer comes from	In a benchmark the scorer is hidden and given. In practice there is none, and "looks done" is all that the agent has [3]. The tests, the reproduction scripts and the feature lists of 2.8 and 2.9 are ways of supplying one	Evidence that agents given a self-written check do about as well as agents given a trusted one
Harness over model	The long-running post says that the first fix for a failing agent was not a better model but a feature list, a progress file and commits [4], and the lab's harness comparison points the same way, at toy scale	Results where the same harness helps a stronger model much less
Review as the bottleneck	The industrial study found that closing a pull request took longer after an AI reviewer was added, and the agent pull request study found that nearly half of the merged ones needed human changes [18] [19]. Someone still has to read the code	Studies that follow reviewer time per change, and not only merge rates
What gets measured	METR's own redesign shows that a randomized trial is hard once everyone uses the tools [13]. Benchmarks drift as well (Week 9a)	A design that survives developers choosing which tasks to submit
Enforcement over instruction	The best-practices page separates what the model may ignore from what the program enforces, and tells you to enforce what must always happen [3]	Incidents with enforced rules that still failed, which I did not find in these sources
Conventions	AGENTS.md is an attempt to make instructions portable between tools [10], and Claude Code reads it in place of its own file [20]	A study of whether such files change what agents do. I found none, and the site's count of "over 60k projects" says that files exist and not that they help

Two cautions about this section. It is my reading, not a summary of any source, and the sources are mostly by people who build or study the tools. And the lecture is on November 30, so some of what I list here may be out of date by then.

2.13 Worked arithmetic

All of this is my own arithmetic from numbers in the sources or the lab.

1. Is a 3-point ablation gap real? On SWE-bench Lite (300 tasks), 18.0% is 54 tasks and 15.0% is 45 tasks, a difference of 9 tasks [1]. The 95% Wilson interval for 54 of 300 is 14.1% to 22.7%, and for 45 of 300 it is 11.4% to 19.5%. The two intervals overlap by a wide margin, so a single run of each cannot tell the two settings apart. The paper's no-edit ablation, 10.3% (31 tasks), is 23 tasks below the default. Its interval, 7.4% to 14.3%, only just touches the default's lower end of 14.1%, so that gap is more believable, though intervals alone do not settle it. How I would settle it: run both settings on the same tasks several times and compare them task by task (Week 9a's paired comparison).
2. The cost of history. With a starting prompt of P = 600 tokens and g = 310 tokens added per turn, the input over T turns is P·T + g·T(T-1)/2. For T = 10: 6,000 + 310 × 45 = 19,950. For T = 30: 18,000 + 310 × 435 = 152,850. The ratio is 152,850 / 19,950 = 7.66, or 7.7. If 90% of the input is cached and billed at 0.1 times, the effective input is 152,850 × (0.9 × 0.1 + 0.1) = 152,850 × 0.19 = 29,042, an 81% cut. The lab's price of $1.45 for the 30-turn run adds the output cost.
3. Why sampling needs a selector. If each sampled patch is correct with probability p = 0.10 and the samples are independent, the chance that at least one of n = 8 is correct is 1 - 0.9⁸ = 0.57. So eight samples can lift a 10% system to a 57% ceiling, but only if something can pick the right one out. That is the oracle row of Figure 9, and the reason that selection by tests matters. Samples are not independent in practice, so this is an upper bound for the idea and not a prediction.
4. Cost per solved task. $1.102 per task at 35 of 60 solved is 1.102 × 60 / 35 = $1.889 per solved task, which the demo prints as 1.890 because it divides unrounded numbers. For the pipeline, 0.107 × 60 / 27 = $0.238 (the demo prints 0.239).
5. The METR numbers as minutes. For a task that takes 60 minutes without AI: a 24% forecast cut is 45.6 minutes, and the measured 19% increase is 71.4 minutes, a difference of 25.8 minutes (43% of 60) between what developers expected and what was measured. The 2026 update's -18% would be 49.2 minutes, but METR says that this figure is unreliable [12] [13].
6. SWE-Lancer in dollars. $208,050 of $500,800 is 41.5%. That is above the 26.2% on coding tasks because the management tasks (44.9% solved) are in the dollar total, and because tasks are weighted by payout [14].

2.14 Week 10 in one page

• A coding agent is a model in a loop with a computer: find, read, edit, run, check, submit. In a benchmark the scorer is hidden. In everyday use there is none unless you supply one [3].
• The scaffold has seven design decisions: interface, loop, context policy, environment, verification, memory and budget. A benchmark score belongs to the model and its scaffold together [1] [2] [17].
• SWE-agent's ablations: a linter-gated edit, a capped search, a windowed viewer and a short history each helped, and one search design was worse than none. A 3-point gap on 300 tasks is within noise [1].
• OpenHands is a platform: an agent that maps state to action, an event stream and a Docker runtime. It reuses CodeAct's idea of code as action [2] [8].
• Claude Code's page separates instructions the model may ignore from rules the program enforces. Use hooks, deny rules and limits for what must always hold [3]. AGENTS.md is an instruction, not an enforcement [10].
• A fixed pipeline (Agentless) can match a loop on SWE-bench Lite at a lower cost. Do not compare costs across papers [6].
• Context is a budget: capped search, skeletons and repo maps decide what the model sees [1] [6] [9].
• Tests are the agent's eyes: the issue's example, visible tests, a reproduction test and a fresh reviewer. With samples, a selector is needed, and voting is an assumption to check [3] [6].
• Long-running work needs a feature list, a progress file and commits, to avoid doing too much at once and calling the job done early [4].
• A loop's input cost grows with the square of its turns, which turn and spend limits and prompt caching control [3].
• Evidence on developers is thin and goes out of date quickly: the 2025 METR result (+19% time) has been followed by an update that METR calls unreliable [12] [13].

3. Reading guide

The course lists two required readings and two additional readings for this lecture [21]. For each you get: why it matters, the key ideas, and questions to think about as you read. Read the questions first. The table at the end points to the sources that I added. Many authors are close to the subject: John Yang, one of the course's instructors, is the first author of the SWE-agent paper, whose co-authors include the two Week 9a speakers [1], and Anthropic wrote both additional readings [3] [4].

The sources that I added

Source	Ref	What I used it for, and what I read
Jimenez et al., SWE-bench (ICLR 2024)	[5]	The task and grading rule. Re-used from Week 9a, where I read the abstract, the main text and Appendix A. Co-authored by the SWE-agent authors.
Xia et al., "Agentless" (2024)	[6]	The three-phase pipeline. I read the abstract and the method and results sections.
Zhang et al., "AutoCodeRover" (ISSTA 2024)	[7]	Code-aware search. I read the abstract, the method and the results table.
Wang et al., "Executable Code Actions Elicit Better LLM Agents" (CodeAct, ICML 2024)	[8]	Code as action. I read the abstract and the introduction. Re-used from Week 4a.
Aider, "Repository map"	[9]	The repo map idea. I read the documentation page and not the code.
AGENTS.md site	[10]	The convention. I read the home page. The site's own claims.
OpenAI, Codex CLI README	[11]	What it says it is. I read the front page, not the code. Vendor.
Becker et al., METR randomized trial (2025)	[12]	Developer productivity. I read the abstract, the design and the discussion of bias. Independent of the tool vendors.
METR, "We are Changing our Developer Productivity Experiment Design" (2026)	[13]	The update that supersedes the headline number. I read the post.
Miserendino et al., SWE-Lancer (2025)	[14]	Paid freelance tasks. I read the abstract and the main results. The authors are at OpenAI.
Pan et al., SWE-Gym (ICML 2025)	[15]	Training agents and verifiers. I read the abstract and the data sections, not the verifier experiments.
Terminal-Bench 2.0	[16]	Command-line tasks. I read the abstract and the task construction. Re-used from Week 6a.
mini-SWE-agent README	[17]	A bash-only agent. I read the front page. Its score claim is the project's own.
Cihan et al., "Automated Code Review in Practice" (2024)	[18]	An industrial study of an AI reviewer. Abstract only.
Watanabe et al., "On the Use of Agentic Coding" (2025)	[19]	Agent-written pull requests. Abstract only.
"How Claude Code works" (Anthropic docs)	[20]	The loop, tools, sessions and context. I read the whole page. Vendor documentation.

4. Hands-on lab: an interface, a map, a pipeline and a harness

Time: about 5 hours. Goal: build and test a small toolkit that does what this week's notes describe: an agent-computer interface (windowed viewer, capped search, linter-gated edit, protected paths) and a scripted agent that works only through it; a reproduction-script generator; an Agentless-style pipeline that localizes, samples, validates and selects; a repo map with a graph ranking; a long-running harness with a feature list, progress notes and checkpoints; and cost and turn accounting. It imports the Week 9a toolkit (swe_bench_toolkit.py, which you need in the same folder) for the toy repositories, the 20 tasks, the test runner and the patch grading, so I do not repeat them. Everything uses the Python standard library, with no model, no API, no Docker and no network.

What you will learn

• How a windowed viewer, a capped search and a linter-gated edit turn a repository into something that a model can work through one turn at a time, and what each costs in turns and tokens.
• How a protected-path rule enforces "do not edit the tests" in the interface, not in a prompt.
• How a pipeline without a loop localizes, samples, validates and selects, and how it compares with a loop when cost is counted the same way.
• How a reproduction test is made from an issue, and what has to be true of it for it to be useful.
• How a repo map is built from the ast module and a graph ranking, and how a token budget cuts it.
• How a feature list, notes and commits each change what survives a crash between sessions.
• Why input cost grows with the square of the turns, and what a cache price does about it.

Map of the code

Part	Main names in the code	What it does, and what it stands for
1	Ledger, undefined_names, lint_source, Workspace	The agent-computer interface [1]. A ledger counts turns and tokens, and in a chat-style loop each turn re-reads the whole history. The workspace has open, goto, scroll_down, scroll_up, find_file, search_dir, search_file, edit (with a linter in three modes), python, run_tests, revert, diff and submit, and a deny list of protected paths. Every method returns text, which is all that the agent sees.
2	slip_edit, AciAgent	A scripted loop agent: find the code with BM25, open it through the viewer, try single edits, check each against the issue's example and the visible tests, submit the first that passes. It can make a typing slip with a made-up chance.
3	repro_script, run_repro, add_repro_test, repro_quality	Reproduction scripts from an issue's example, with the three labels "Issue reproduced", "Issue resolved" and "Other issues" [6], and a check that a generated test fails before the gold patch and passes after.
4	skeleton, normalized_key, Cand, AgentlessPipeline, selection_study	The fixed pipeline: localize files and then code units, draw sampled candidates, validate each against the visible tests and a reproduction test, and select by none, first or vote [6]. The study compares seven selectors with an oracle ceiling.
5	symbol_index, reference_counts, file_graph, pagerank, repo_map, synthetic_repo, localize_study	An Aider-style repo map [9]: a symbol index from the ast module, a graph of which file uses names from which other file, a PageRank-style score, and a map cut to a token budget. A synthetic repository with hub modules, and a study of bodies against signatures for localization.
6	project_files, HarnessConfig, run_long	The long-running harness [4]: a project with several planted bugs as features, sessions limited by a made-up context size, a crash that leaves a half-written file, and switches for the feature list, the progress notes and the commits.
7	four_tuple, parse_agents_md	The 4-tuple of one task, and a parser for an AGENTS.md style file [10].
8	demo	The ten-section demo.

Setup

1. You need Python 3.10 or later. There is nothing to install. I ran it on Python 3.13 only.
2. Save the Week 9a toolkit as swe_bench_toolkit.py in a folder. It is the same file as the one on the Week 9a page, and you can also download it here: swe_bench_toolkit.py.
3. Save the code below as coding_agents_toolkit.py in the same folder, or download it: coding_agents_toolkit.py.
4. Save the tests as test_coding_agents_toolkit.py in the same folder, or download: test_coding_agents_toolkit.py. The test code is not printed on this page, to keep the page shorter. It is in the download, and the names of all the tests are listed below.
5. Run python coding_agents_toolkit.py for the ten-section demo, and python -m unittest test_coding_agents_toolkit for the tests. Both are deterministic. The demo takes about ten seconds on my machine and the tests about fifteen.

The code: coding_agents_toolkit.py

"""coding_agents_toolkit.py: a small, stdlib-only lab for Week 10 (Coding & Software Agents).

It builds on swe_bench_toolkit.py from Week 9a (same folder) and adds:
  1. Ledger and Workspace: an agent-computer interface (windowed viewer, capped search, linter-gated edit,
     a deny rule) with turn and token accounting.
  2. AciAgent: a scripted agent loop that works only through that interface.
  3. AgentlessPipeline: localize, repair (sampled), validate with a reproduction test, select by voting.
  4. Reproduction scripts and tests built from the issue text.
  5. Repo map: an ast symbol index with a link-based ranking and a token budget.
  6. Long-running harness: feature list, progress notes, checkpoints, and context-window sessions.
  7. Small helpers: the 4-tuple view of a task, an AGENTS.md reader.

Everything is deterministic. The scripted agents are programs, not language models. Every number, price,
probability and budget below is made up by me for teaching. The runner is not a security sandbox.
Run `python3 coding_agents_toolkit.py` for the demo, `python3 -m unittest test_coding_agents_toolkit` for tests.
"""
import ast
import builtins
import os
import re
import sys
from collections import Counter

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import swe_bench_toolkit as S

mean, rng_for, est_tokens = S.mean, S.rng_for, S.est_tokens
SCALE = S.CONTEXT_SCALE


# ---------------------------------------------------------------- Part 1: ledger and interface
class Ledger:
    """Counts turns and tokens. In a chat-style loop every turn re-reads the whole history, so input tokens
    grow with the square of the number of turns. `call` is for independent single-shot calls (no history)."""

    def __init__(self, tier="mid", system_tokens=600):
        self.tier, self.context, self.turns, self.tin, self.tout = tier, system_tokens, 0, 0, 0

    def turn(self, out_tokens, obs_tokens=0):
        self.tin += self.context
        self.tout += out_tokens
        self.context += out_tokens + obs_tokens
        self.turns += 1

    def call(self, prompt_tokens, out_tokens):
        self.tin += prompt_tokens
        self.tout += out_tokens
        self.turns += 1

    def cost(self, scale=SCALE, cached_frac=0.0, cache_price=0.1):
        """Dollars at the made-up tier price. cached_frac of input tokens are billed at cache_price times the rate."""
        pin, pout = S.TIERS[self.tier]
        eff_in = self.tin * ((1 - cached_frac) + cached_frac * cache_price)
        return (eff_in * pin + self.tout * pout) * scale / 1e6

    def result(self, patch, log):
        res = S.AgentResult(patch, self.tin * SCALE, self.tout * SCALE, self.turns, log)
        res.peak_context = self.context * SCALE  # how full the context window got (tokens, scaled)
        return res


LINT_MODES = ("none", "syntax", "names")


def undefined_names(tree):
    """Names that are read but never bound anywhere in the module (a coarse check: one global scope)."""
    defined = set(dir(builtins))
    for n in ast.walk(tree):
        if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            defined.add(n.name)
        elif isinstance(n, ast.arg):
            defined.add(n.arg)
        elif isinstance(n, ast.Name) and isinstance(n.ctx, (ast.Store, ast.Del)):
            defined.add(n.id)
        elif isinstance(n, (ast.Import, ast.ImportFrom)):
            for a in n.names:
                defined.add((a.asname or a.name).split(".")[0])
        elif isinstance(n, ast.ExceptHandler) and n.name:
            defined.add(n.name)
    bad = [(n.id, n.lineno) for n in ast.walk(tree)
           if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Load) and n.id not in defined]
    return sorted(bad, key=lambda x: (x[1], x[0]))


def lint_source(src, mode):
    """Return an error message, or None when the source passes. mode: none, syntax (ast.parse), names."""
    if mode not in LINT_MODES:
        raise ValueError("unknown lint mode %r" % mode)
    if mode == "none":
        return None
    try:
        tree = ast.parse(src)
    except SyntaxError as e:
        return "SyntaxError: %s (line %s)" % (e.msg, e.lineno)
    if mode == "names":
        bad = undefined_names(tree)
        if bad:
            return "undefined name %s (line %d)" % bad[0]
    return None


def _lines(src):
    parts = src.split("\n")
    return parts[:-1] if src.endswith("\n") else parts


def _unlines(lines, trailing=True):
    return "\n".join(lines) + ("\n" if trailing and lines else "")


COMMANDS = ("open", "goto", "scroll_down", "scroll_up", "find_file", "search_dir", "search_file", "edit",
            "run_tests", "python", "revert", "diff", "submit")


class Workspace:
    """The agent-computer interface. Every method returns text, which is all the agent ever sees.
    The viewer shows at most `window` lines (SWE-agent: 100), search shows at most `search_cap` hits
    (SWE-agent: 50), and `edit` refuses a change that fails the linter (lint=syntax or names)."""

    def __init__(self, files, window=100, search_cap=50, lint="syntax", deny=(), ledger=None):
        self.base, self.files = dict(files), dict(files)
        self.window, self.search_cap, self.lint, self.deny = window, search_cap, lint, tuple(deny)
        self.cur, self.top = None, 1
        self.ledger = ledger or Ledger()
        self.stats = Counter()
        self.history = []
        self.last_tests = {}

    # ---- viewer
    def _view(self):
        lines = _lines(self.files[self.cur])
        n = len(lines)
        a = self.top
        b = min(n, a + self.window - 1)
        out = ["[File: %s (%d lines total)]" % (self.cur, n), "(%d more lines above)" % (a - 1)]
        out += ["%d: %s" % (i, lines[i - 1]) for i in range(a, b + 1)]
        out.append("(%d more lines below)" % (n - b))
        return "\n".join(out)

    def _place(self, line):
        n = max(1, len(_lines(self.files[self.cur])))
        top = (line if line is not None else 1) - (self.window // 2 if line is not None else 0)
        self.top = max(1, min(top, max(1, n - self.window + 1)))

    def open(self, path, line=None):
        if path not in self.files:
            return "Error: file not found: %s" % path
        self.cur = path
        self._place(line)
        return self._view()

    def goto(self, line):
        if self.cur is None:
            return "Error: no file is open. Use open first."
        n = len(_lines(self.files[self.cur]))
        if not 1 <= line <= n:
            return "Error: line %d is outside the file (1 to %d)" % (line, n)
        self._place(line)
        return self._view()

    def scroll_down(self):
        if self.cur is None:
            return "Error: no file is open. Use open first."
        n = len(_lines(self.files[self.cur]))
        self.top = max(1, min(self.top + self.window, max(1, n - self.window + 1)))
        return self._view()

    def scroll_up(self):
        if self.cur is None:
            return "Error: no file is open. Use open first."
        self.top = max(1, self.top - self.window)
        return self._view()

    # ---- search
    def find_file(self, name):
        hits = [p for p in sorted(self.files) if name in p.split("/")[-1]]
        if not hits:
            return 'No files named "%s".' % name
        if len(hits) > self.search_cap:
            return "More than %d files match. Please narrow your search." % self.search_cap
        return "Found %d file(s):\n" % len(hits) + "\n".join(hits)

    def _grep(self, term, paths):
        hits = []
        for p in paths:
            for i, ln in enumerate(_lines(self.files[p]), 1):
                if term in ln:
                    hits.append("%s:%d: %s" % (p, i, ln.strip()))
        if not hits:
            return 'No matches found for "%s".' % term
        if len(hits) > self.search_cap:
            return 'More than %d matches for "%s". Please narrow your search.' % (self.search_cap, term)
        return 'Found %d match(es) for "%s":\n' % (len(hits), term) + "\n".join(hits)

    def search_dir(self, term):
        return self._grep(term, sorted(self.files))

    def search_file(self, term, path=None):
        path = path or self.cur
        if path not in self.files:
            return "Error: file not found: %s" % str(path)
        return self._grep(term, [path])

    # ---- edit
    def edit(self, start, end, text):
        if self.cur is None:
            return "Error: no file is open. Use open first."
        if any(d in self.cur for d in self.deny):
            self.stats["blocked"] += 1
            return "Edit blocked: %s matches a protected path." % self.cur
        lines = _lines(self.files[self.cur])
        if not 1 <= start <= end <= len(lines):
            return "Error: bad line range %d to %d (the file has %d lines)" % (start, end, len(lines))
        new = text[:-1].split("\n") if text.endswith("\n") else (text.split("\n") if text else [])
        new_lines = lines[:start - 1] + new + lines[end:]
        new_src = _unlines(new_lines, self.files[self.cur].endswith("\n"))
        if self.cur.endswith(".py"):
            msg = lint_source(new_src, self.lint)
            if msg:
                self.stats["rejected"] += 1
                lo, hi = max(1, start - 2), min(len(new_lines), start + len(new) + 1)
                shown = "\n".join("%d: %s" % (i, new_lines[i - 1]) for i in range(lo, hi + 1))
                return "Your edit was not applied (%s).\nThis is how it would have looked:\n%s\nFix the edit and try again." % (msg, shown)
        self.files[self.cur] = new_src
        self.stats["edits"] += 1
        self.top = max(1, start - 3)
        return "File updated.\n" + self._view()

    # ---- run and submit
    def run_tests(self):
        res = S.run_tests(self.files)
        self.last_tests = res
        bad = sorted(t for t, s in res.items() if s != "PASS")
        head = "%d passed, %d not passing" % (len(res) - len(bad), len(bad))
        return head + ("\n" + "\n".join(bad[:5]) if bad else "")

    def python(self, expr, setup=""):
        if self.cur is None or not self.cur.endswith(".py"):
            return "Error: open a python file first."
        return "=> " + S.eval_snippet(self.files, self.cur, setup, expr)

    def revert(self, path=None):
        if path is None:
            self.files = dict(self.base)
        elif path in self.base:
            self.files[path] = self.base[path]
        else:
            self.files.pop(path, None)
        return "Reverted."

    def diff(self):
        return S.make_diff(self.base, self.files)

    def submit(self):
        return self.diff()

    def act(self, cmd, *args, think=30):
        """One agent turn: run a command, charge the ledger for the command and for the text sent back."""
        if cmd not in COMMANDS:
            raise ValueError("unknown command %r" % cmd)
        obs = getattr(self, cmd)(*args)
        out = think + est_tokens(cmd + " " + " ".join(str(a) for a in args))
        self.ledger.turn(out, est_tokens(obs))
        self.history.append((cmd, args, obs))
        return obs


# ---------------------------------------------------------------- Part 2: an agent loop on the interface
def slip_edit(text, kind):
    """Plant a typical slip in an edit. 'paren' leaves an unclosed bracket (a syntax error); 'name' mistypes the
    first parameter at its last use (passes ast.parse, fails at run time)."""
    if kind == "name":
        m = re.match(r"\s*def \w+\(((?:self, )?)(\w+)", text.split("\n")[0])
        if m:
            p = m.group(2)
            body = text.split("\n", 1)[1] if "\n" in text else ""
            spots = [x for x in re.finditer(r"\b%s\b" % re.escape(p), body)]
            if spots:
                x = spots[-1]
                body = body[:x.start()] + p + "_" + body[x.end():]
                return text.split("\n")[0] + "\n" + body
    return text.rstrip("\n") + " ("


class AciAgent:
    """Locate with BM25, open the code through the viewer, try single edits through the edit command, check
    each against the issue example and the visible tests, submit the first edit that passes.
    `slip` is the made-up chance that an edit is typed with a mistake. With lint=syntax the interface rejects
    a syntax slip and the agent retypes the edit (one more turn); with lint=none the bad edit goes in."""

    def __init__(self, name, repertoire=None, lint="syntax", slip=0.0, max_turns=60, window=15, k_units=3,
                 tier="mid", search_cap=50, use_tests=True):
        self.name, self.repertoire = name, list(repertoire or S.FULL_OPS)
        self.lint, self.slip, self.max_turns, self.window = lint, slip, max_turns, window
        self.k_units, self.tier, self.search_cap, self.use_tests = k_units, tier, search_cap, use_tests

    def solve(self, view, hint=None):
        led = Ledger(self.tier)
        ws = Workspace(view.files, window=self.window, search_cap=self.search_cap, lint=self.lint,
                       deny=("tests/",), ledger=led)
        probe = S.ScriptedAgent("probe", S.STD_OPS, k_units=self.k_units)
        units = probe._units(view)
        ranked = probe._rank(view, units, hint)[:self.k_units]
        repro = S.parse_repro(view.issue)
        log = []
        for kw in S.content_tokens(view.issue):
            if ws.search_dir(kw).startswith("Found"):
                ws.act("search_dir", kw)
                break
        base_pass = set()
        if self.use_tests:
            ws.act("run_tests")
            base_pass = {t for t, s in ws.last_tests.items() if s == "PASS"}
        tried = 0
        for key in ranked:
            path, name = key
            a, b, text = units[key]
            ws.act("open", path, a)
            if repro:
                ws.act("python", repro[1], repro[0])
            for fam, new_unit in S.candidates_for_unit(text, self.repertoire, repro[2] if repro else None):
                if led.turns >= self.max_turns:
                    log.append("turn budget used up; gave up")
                    return led.result("", log)
                tried += 1
                typed = new_unit
                if self.slip and rng_for("slip", self.name, view.id, tried).random() < self.slip:
                    kind = "paren" if rng_for("kind", self.name, view.id, tried).random() < 0.5 else "name"
                    typed = slip_edit(new_unit, kind)
                    log.append("slip (%s) on try %d" % (kind, tried))
                obs = ws.act("edit", a, b, typed, think=60)
                if obs.startswith("Your edit was not applied"):
                    log.append("edit rejected by the linter on try %d" % tried)
                    obs = ws.act("edit", a, b, new_unit, think=60)
                ok = obs.startswith("File updated")
                if ok and repro:
                    ok = ws.act("python", repro[1], repro[0]) == "=> " + repro[2]
                if ok and self.use_tests:
                    ws.act("run_tests")
                    ok = all(ws.last_tests.get(t) == "PASS" for t in base_pass)
                if ok:
                    patch = ws.act("submit")
                    log.append("submit %s edit in %s after %d tries" % (fam, name, tried))
                    return led.result(patch, log)
                ws.act("revert")
        log.append("no candidate passed after %d tries; gave up" % tried)
        return led.result("", log)


# ---------------------------------------------------------------- Part 3: reproduction scripts and tests
def repro_script(repro, import_line=""):
    """Source of a standalone reproduction script in the three-label style that the Agentless paper describes:
    it prints 'Issue reproduced', 'Issue resolved' or 'Other issues'."""
    setup, expr, expected = repro
    body = ["    " + ln for ln in setup.split("\n") if ln.strip()]
    out = ([import_line] if import_line else []) + ["", "", "def main():", "    try:"]
    out += ["    " + ln for ln in body]
    out += ["        got = %s" % expr, "    except Exception:", '        print("Other issues")', "        return",
            "    if got == %s:" % expected, '        print("Issue resolved")', "    else:", '        print("Issue reproduced")',
            "", "", "main()", ""]
    return "\n".join(out)


def run_repro(files, module_path, repro):
    """What the script would print on this version of the code (computed with the in-process runner)."""
    got = S.eval_snippet(files, module_path, repro[0], repro[1])
    if got.startswith("raises"):
        return "Other issues"
    return "Issue resolved" if got == repro[2] else "Issue reproduced"


def add_repro_test(files, test_path, repro, name="test_issue_repro"):
    """Append the issue's example to a test file as a test that asserts the expected value."""
    setup, expr, expected = repro
    body = "".join("    %s\n" % ln for ln in setup.split("\n") if ln.strip())
    src = files[test_path].rstrip("\n") + "\n\n\ndef %s():\n%s    assert (%s) == (%s)\n" % (name, body, expr, expected)
    out = dict(files)
    out[test_path] = src
    return out


def _test_path(files):
    tfs = S.test_files(files)
    return tfs[0] if tfs else None


def repro_quality(tasks):
    """For tasks whose issue has a runnable example: does the generated test fail on the base code and pass
    with the gold patch? Returns {task id: 'valid' | 'no repro' | 'passes before' | 'fails after gold'}."""
    out = {}
    for t in tasks:
        repro = S.parse_repro(t.issue)
        if repro is None:
            out[t.id] = "no repro"
            continue
        before = S.run_tests(add_repro_test(t.base, t.test_path, repro))
        gold, _ = S.apply_patch(t.base, t.gold_patch)
        after = S.run_tests(add_repro_test(gold, t.test_path, repro))
        key = t.test_path + "::test_issue_repro"
        if before.get(key) == "PASS":
            out[t.id] = "passes before"
        elif after.get(key) != "PASS":
            out[t.id] = "fails after gold"
        else:
            out[t.id] = "valid"
    return out


# ---------------------------------------------------------------- Part 4: localize, repair, validate, select
def skeleton(path, src):
    """Headers of the classes and functions in one file (the Agentless 'skeleton' idea, in my own simple form)."""
    out = [path + ":"]
    for name, (a, b, text) in S.code_units(path, src).items():
        out.append("  " + text.split("\n")[0].strip())
    return "\n".join(out)


def normalized_key(src):
    """Whitespace and comment-proof identity of a python file: the dump of its syntax tree."""
    try:
        return ast.dump(ast.parse(src))
    except SyntaxError:
        return "syntax-error:" + src


class Cand:
    def __init__(self, path, name, fam, new_src, rank):
        self.path, self.name, self.fam, self.new_src, self.rank = path, name, fam, new_src, rank
        self.key = normalized_key(new_src)


class AgentlessPipeline:
    """Fixed three-phase pipeline, no agent loop: (1) localize files then functions with BM25, (2) draw n
    sampled single-edit candidates, early candidates more often (a stand-in for sampling a model), (3) validate
    each distinct candidate against the visible tests plus a reproduction test made from the issue, then pick
    one. select: none (first sample, no validation), first (first that validates), vote (most frequent among
    those that validate; ties go to the earliest). If nothing validates the pipeline submits nothing (my choice)."""

    def __init__(self, name, repertoire=None, n_files=2, n_units=4, n_samples=8, select="vote", tier="mid", skew=2.0):
        self.name, self.repertoire = name, list(repertoire or S.FULL_OPS)
        self.n_files, self.n_units, self.n_samples, self.select, self.tier, self.skew = n_files, n_units, n_samples, select, tier, skew

    def localize(self, view, hint=None):
        probe = S.ScriptedAgent("probe", S.STD_OPS)
        units = probe._units(view)
        files = {}
        for (p, n), (a, b, text) in units.items():
            files.setdefault(p, []).append(text)
        docs = {p: S.content_tokens("\n".join(ts)) + S.content_tokens(p) for p, ts in files.items()}
        top_files = [p for _, p in S.bm25_rank(S.content_tokens(view.issue), docs)][:self.n_files]
        sub = {k: v for k, v in units.items() if k[0] in top_files}
        ranked = [k for k in probe._rank(view, sub, hint)][:self.n_units]
        return top_files, ranked, units

    def pool(self, view, ranked, units):
        repro = S.parse_repro(view.issue)
        out = []
        for key in ranked:
            path, name = key
            a, b, text = units[key]
            lines = view.files[path].split("\n")
            for fam, new_unit in S.candidates_for_unit(text, self.repertoire, repro[2] if repro else None):
                new_src = "\n".join(lines[:a - 1] + new_unit.split("\n") + lines[b:])
                out.append(Cand(path, name, fam, new_src, len(out)))
        return out

    def sample(self, view, pool, seed=0):
        """n draws from the pool. Draw 0 is the greedy one (pool[0]); later draws favour early candidates."""
        if not pool:
            return []
        rng = rng_for("sample", view.id, seed)
        draws = [pool[0]]
        for _ in range(self.n_samples - 1):
            draws.append(pool[min(len(pool) - 1, int(len(pool) * rng.random() ** self.skew))])
        return draws

    def validate(self, view, draws):
        """For each distinct candidate: (passes the visible tests that passed before, passes the reproduction test)."""
        repro = S.parse_repro(view.issue)
        tp = _test_path(view.files)
        base_pass = {t for t, s in S.run_tests(view.files).items() if s == "PASS"}
        out = {}
        for c in draws:
            if c.key in out:
                continue
            files = dict(view.files)
            files[c.path] = c.new_src
            use_repro = repro is not None and tp is not None
            if use_repro:
                files = add_repro_test(files, tp, repro)
            res = S.run_tests(files)
            reg_ok = all(res.get(t) == "PASS" for t in base_pass)
            rep_ok = res.get(tp + "::test_issue_repro") == "PASS" if use_repro else True
            out[c.key] = (reg_ok, rep_ok)
        return out

    def choose(self, draws, valid, how=None):
        how = how or self.select
        if not draws:
            return None
        if how == "none":
            return draws[0]
        good = [c for c in draws if all(valid[c.key])]
        if not good:
            return None
        if how == "first":
            return good[0]
        counts = Counter(c.key for c in good)
        best = max(counts.values())
        return next(c for c in good if counts[c.key] == best)

    def solve(self, view, hint=None, seed=0):
        led = Ledger(self.tier)
        files, ranked, units = self.localize(view, hint)
        tree = "\n".join(sorted(view.files))
        led.call(est_tokens(view.issue) + est_tokens(tree), 30)
        led.call(est_tokens(view.issue) + sum(est_tokens(skeleton(p, view.files[p])) for p in files), 40)
        led.call(est_tokens(view.issue) + sum(est_tokens(units[k][2]) for k in ranked), 40)
        pool = self.pool(view, ranked, units)
        draws = self.sample(view, pool, seed)
        ctx = est_tokens(view.issue) + sum(est_tokens(units[k][2]) for k in ranked)
        led.call(ctx, sum(est_tokens(c.new_src[:200]) for c in draws))
        repro = S.parse_repro(view.issue)
        if repro and self.select != "none":
            led.call(est_tokens(view.issue), est_tokens(repro_script(repro)))
        valid = self.validate(view, draws) if self.select != "none" else {c.key: (True, True) for c in draws}
        pick = self.choose(draws, valid)
        log = ["%d files, %d functions, pool %d, %d samples, %d distinct" % (
            len(files), len(ranked), len(pool), len(draws), len({c.key for c in draws}))]
        if pick is None:
            log.append("nothing validated; submitted nothing")
            return led.result("", log)
        patch = S.make_diff({pick.path: view.files[pick.path]}, {pick.path: pick.new_src})
        log.append("submit %s edit in %s" % (pick.fam, pick.name))
        return led.result(patch, log)


def selection_study(tasks, n_samples=8, seeds=3, skew=2.0, hidden_cache=None):
    """Draw n_samples candidates per task and seed. Pick one with each strategy. Grade the pick with the hidden
    tests. Returns {strategy: [per-task resolved share over seeds]} in task order."""
    strategies = ["greedy", "random", "visible-tests first", "visible-tests vote", "repro+tests first", "repro+tests vote", "oracle"]
    out = {s: [] for s in strategies}
    cache = hidden_cache if hidden_cache is not None else {}

    def grade(t, c):
        if c is None:
            return 0
        k = (t.id, c.key)
        if k not in cache:
            patch = S.make_diff({c.path: t.base[c.path]}, {c.path: c.new_src})
            cache[k] = int(S.evaluate(t, patch).resolved)
        return cache[k]

    for t in tasks:
        view = t.view()
        pipe = AgentlessPipeline("sel", n_samples=n_samples, skew=skew)
        top, ranked, units = pipe.localize(view, t.fn)
        pool = pipe.pool(view, ranked, units)
        acc = {s: [] for s in strategies}
        for seed in range(seeds):
            draws = pipe.sample(view, pool, seed)
            valid = pipe.validate(view, draws)
            vis = {k: (v[0], True) for k, v in valid.items()}
            rng = rng_for("rand-pick", t.id, seed)
            acc["greedy"].append(grade(t, draws[0] if draws else None))
            acc["random"].append(grade(t, rng.choice(draws) if draws else None))
            acc["visible-tests first"].append(grade(t, pipe.choose(draws, vis, "first")))
            acc["visible-tests vote"].append(grade(t, pipe.choose(draws, vis, "vote")))
            acc["repro+tests first"].append(grade(t, pipe.choose(draws, valid, "first")))
            acc["repro+tests vote"].append(grade(t, pipe.choose(draws, valid, "vote")))
            acc["oracle"].append(int(any(grade(t, c) for c in draws)))
        for s in strategies:
            out[s].append(mean(acc[s]))
    return out


# ---------------------------------------------------------------- Part 5: repo map
def symbol_index(files):
    """{path: [(kind, name, signature, line, parent)]} for classes, functions and methods of every parseable .py file."""
    out = {}
    for p in sorted(files):
        if not p.endswith(".py"):
            continue
        try:
            tree = ast.parse(files[p])
        except SyntaxError:
            continue
        syms = []
        for n in tree.body:
            if isinstance(n, ast.FunctionDef):
                syms.append(("def", n.name, "def %s(%s)" % (n.name, ast.unparse(n.args)), n.lineno, None))
            elif isinstance(n, ast.ClassDef):
                syms.append(("class", n.name, "class %s" % n.name, n.lineno, None))
                for m in n.body:
                    if isinstance(m, ast.FunctionDef):
                        syms.append(("def", m.name, "def %s(%s)" % (m.name, ast.unparse(m.args)), m.lineno, n.name))
        out[p] = syms
    return out


def reference_counts(files):
    """{path: Counter of identifiers used}: every name, attribute and imported name in the file."""
    out = {}
    for p in sorted(files):
        if not p.endswith(".py"):
            continue
        try:
            tree = ast.parse(files[p])
        except SyntaxError:
            continue
        c = Counter()
        for n in ast.walk(tree):
            if isinstance(n, ast.Name):
                c[n.id] += 1
            elif isinstance(n, ast.Attribute):
                c[n.attr] += 1
            elif isinstance(n, ast.ImportFrom):
                for a in n.names:
                    c[a.name] += 1
        out[p] = c
    return out


def file_graph(files):
    """Edges {(a, b): weight}: file a uses symbols that file b defines (and a does not define itself)."""
    idx, refs = symbol_index(files), reference_counts(files)
    defined = {p: {s[1] for s in syms} for p, syms in idx.items()}
    edges = {}
    for a in sorted(refs):
        for b in sorted(idx):
            if a == b:
                continue
            w = sum(refs[a][name] for name in defined[b] if name not in defined[a])
            if w:
                edges[(a, b)] = w
    return edges


def pagerank(nodes, edges, personal=None, d=0.85, iters=60):
    """Plain power iteration. Rank flows along edges a -> b in proportion to weight. `personal` biases the restart."""
    nodes = list(nodes)
    n = len(nodes)
    if n == 0:
        return {}
    pers = {v: (personal or {}).get(v, 0.0) for v in nodes}
    tot = sum(pers.values())
    pers = {v: (pers[v] / tot if tot else 1.0 / n) for v in nodes}
    out_w = Counter()
    for (a, b), w in edges.items():
        out_w[a] += w
    r = {v: 1.0 / n for v in nodes}
    for _ in range(iters):
        dangling = sum(r[v] for v in nodes if out_w[v] == 0)
        new = {v: (1 - d) * pers[v] + d * dangling * pers[v] for v in nodes}
        for (a, b), w in sorted(edges.items()):
            new[b] += d * r[a] * w / out_w[a]
        r = new
    return r


def repo_map(files, budget=300, focus=(), boost=5.0):
    """A compact map of the repository: files in rank order, each with its most referenced symbols, cut to a token
    budget (est_tokens). Files in `focus` get a larger restart weight, the way chat files do in Aider (the
    ranking here is my own simplified version, not Aider's). Returns (text, info)."""
    idx = symbol_index(files)
    edges = file_graph(files)
    rank = pagerank(sorted(idx), edges, personal={p: (boost if p in focus else 1.0) for p in idx})
    refs = reference_counts(files)
    uses = {}
    for p, syms in idx.items():
        for s in syms:
            uses[(p, s[1])] = sum(refs[q][s[1]] for q in refs if q != p)
    order = sorted(idx, key=lambda p: (-rank[p], p))
    chosen = {}

    def render():
        lines = []
        for p in order:
            if p not in chosen:
                continue
            lines.append(p + ":")
            for s in sorted(chosen[p], key=lambda s: s[3]):
                lines.append(("    " if s[4] else "  ") + s[2])
        return "\n".join(lines)

    for p in order:
        cand = sorted(idx[p], key=lambda s: (-uses[(p, s[1])], s[3]))
        for s in cand:
            chosen.setdefault(p, [])
            if s[4] and not any(x[1] == s[4] and x[0] == "class" for x in chosen[p]):
                head = next((x for x in idx[p] if x[0] == "class" and x[1] == s[4]), None)
                trial = chosen[p] + ([head] if head else []) + [s]
            else:
                trial = chosen[p] + [s]
            saved = chosen[p]
            chosen[p] = trial
            if est_tokens(render()) > budget:
                chosen[p] = saved
                break
        if not chosen.get(p):
            chosen.pop(p, None)
    text = render()
    total = sum(len(v) for v in idx.values())
    shown = sum(len(v) for v in chosen.values())
    return text, {"files": len(chosen), "files_total": len(idx), "symbols": shown, "symbols_total": total,
                  "tokens": est_tokens(text), "full_tokens": sum(est_tokens(files[p]) for p in idx), "rank": rank}


def synthetic_repo(n_modules=12, per_module=3, seed=0):
    """A made-up package where early modules are used by many later ones (hubs). Every function is valid python."""
    rng = rng_for("synthetic", n_modules, per_module, seed)
    files = {"pkg/__init__.py": ""}
    for i in range(n_modules):
        imports, body = [], []
        for j in range(per_module):
            calls = []
            if i:
                for _ in range(rng.choice([1, 2])):
                    k = min(i - 1, int(i * rng.random() ** 3))
                    f = "f%d_%d" % (k, rng.randrange(per_module))
                    line = "from pkg.m%02d import %s" % (k, f)
                    if line not in imports:
                        imports.append(line)
                    calls.append(f)
            expr = " + ".join("%s(x)" % c for c in calls) if calls else "x"
            body.append("def f%d_%d(x):\n    return %s + %d\n" % (i, j, expr, j))
        files["pkg/m%02d.py" % i] = "\n".join(imports) + ("\n\n\n" if imports else "") + "\n\n".join(body)
    return files


def localize_study(tasks, mask_names=False):
    """File-level top-1 and function-level top-3 hit rates, comparing an index of full function bodies with an index
    of signatures only (what a tight map would show). One merged repo holds all five toy packages. With
    mask_names the issue text has the buggy function's name replaced by the word 'it' (a harder, behaviour-only issue)."""
    merged = {}
    for t in tasks:
        for p, src in t.base.items():
            merged.setdefault(p, src)
    acc = Counter()
    for t in tasks:
        issue = re.sub(r"\b%s\b" % re.escape(t.fn), "it", t.issue) if mask_names else t.issue
        view = S.TaskView(t.id, issue, merged)
        probe = S.ScriptedAgent("probe", S.STD_OPS)
        units = probe._units(view)
        q = S.content_tokens(issue)
        by_file = {}
        for (p, n), (a, b, text) in units.items():
            by_file.setdefault(p, []).append((n, text))
        for mode in ("bodies", "signatures"):
            def doc(text):
                return S.content_tokens(text if mode == "bodies" else text.split("\n")[0])
            fdocs = {p: [w for n, tx in v for w in doc(tx)] for p, v in by_file.items()}
            top_file = S.bm25_rank(q, fdocs)[0][1]
            acc[mode + " file@1"] += top_file == t.module
            udocs = {k: doc(v[2]) + 3 * [x for x in S.tokens(k[1]) if x not in S.STOP] for k, v in units.items()}
            top3 = [k for _, k in S.bm25_rank(q, udocs)][:3]
            acc[mode + " function@3"] += any(k[1].split(".")[-1] == t.fn for k in top3)
    return {k: v / len(tasks) for k, v in acc.items()}


# ---------------------------------------------------------------- Part 6: long-running harness
def project_files(repo):
    """One package with every planted bug of that repo at once, plus the basic tests (which still pass) and one
    regression test per feature (which fail for the bugs). Returns (files, features)."""
    r = S.REPOS[repo]
    specs = [s for s in S.SPECS if s["repo"] == repo]
    src = r["src"]
    for s in specs:
        try:
            src = S._subs(src, s["bug"])
        except ValueError:
            pass
    files = S.repo_files(repo, [s["fn"] for s in specs])
    files[r["module"]] = src
    tests = files[r["test"]] + "".join("\n\ndef test_%s_regress():\n    %s\n" % (s["fn"], S.REGRESS[s["fn"]]) for s in specs)
    files[r["test"]] = tests
    res = S.run_tests(files)
    feats = []
    for s in specs:
        key = r["test"] + "::test_%s_regress" % s["fn"]
        feats.append({"id": s["fn"], "issue": s["issue"], "test": key, "passes": res.get(key) == "PASS"})
    return files, feats


class HarnessConfig:
    """Which pieces of the harness are switched on, and the made-up limits of the toy world."""

    def __init__(self, feature_list=True, progress=True, commits=True, session_context=400000, crash=0.3,
                 max_sessions=8, max_attempts=2):
        self.feature_list, self.progress, self.commits = feature_list, progress, commits
        self.session_context, self.crash, self.max_sessions, self.max_attempts = session_context, crash, max_sessions, max_attempts


def _visible_pass(files, feats):
    res = S.run_tests(files)
    hidden = {f["test"] for f in feats}
    return all(s == "PASS" for t, s in res.items() if t not in hidden) and bool(res)


def run_long(repo, cfg, seed=0, agent=None):
    """Work through a project in several sessions. A session ends when its context window (session_context,
    made-up tokens) is used up, or when the toy 'crash' cuts it off in the middle of an edit, which leaves a
    half-written file in the working tree. What survives between sessions depends on cfg."""
    agent = agent or AciAgent("harness-agent", tier="mid")
    files, feats = project_files(repo)
    pin, pout = S.TIERS[agent.tier]
    commits, progress, attempts = [dict(files)], [], Counter()
    tokens_in = tokens_out = 0
    log, sessions, declared_done = [], 0, False
    for sess in range(cfg.max_sessions):
        sessions += 1
        if cfg.progress:
            notes = "\n".join(progress) + "\n" + "\n".join("%s %s" % (f["id"], f["passes"]) for f in feats)
            orient = est_tokens(notes) * SCALE
        else:
            orient = sum(est_tokens(v) for v in files.values()) * SCALE
        used = orient
        tokens_in += orient
        if cfg.commits:
            files = dict(commits[-1])
        if cfg.feature_list:
            todo = [f for f in feats if not f["passes"] and (not cfg.progress or attempts[f["id"]] < cfg.max_attempts)]
        else:
            if _visible_pass(files, feats):
                declared_done = True
                log.append("session %d: visible tests pass, declared done" % sess)
                break
            log.append("session %d: visible tests fail and nothing says why; stopped" % sess)
            break
        if not todo:
            log.append("session %d: nothing left to try" % sess)
            break
        for f in todo:
            if used >= cfg.session_context:
                break
            res = agent.solve(S.TaskView(f["id"], f["issue"], files))
            attempts[f["id"]] += 1
            used += res.peak_context
            tokens_in += res.tokens_in
            tokens_out += res.tokens_out
            if rng_for("crash", repo, seed, sess, f["id"], attempts[f["id"]]).random() < cfg.crash:
                if res.patch:
                    mod = S.REPOS[repo]["module"]
                    try:
                        new, _ = S.apply_patch(files, res.patch)
                        files = dict(files)
                        files[mod] = new[mod][:len(new[mod]) // 2]
                    except S.PatchError:
                        pass
                log.append("session %d: cut off while working on %s" % (sess, f["id"]))
                break
            if res.patch:
                try:
                    files, _ = S.apply_patch(files, res.patch)
                except S.PatchError:
                    pass
            if S.run_tests(files).get(f["test"]) == "PASS":
                f["passes"] = True
                if cfg.commits:
                    commits.append(dict(files))
                if cfg.progress:
                    progress.append("done " + f["id"])
            else:
                if cfg.commits:
                    files = dict(commits[-1])
                if cfg.progress:
                    progress.append("failed " + f["id"])
    final = S.run_tests(files)
    ok = sum(1 for f in feats if final.get(f["test"]) == "PASS")
    start_pass = sum(1 for f in project_files(repo)[1] if f["passes"])
    return {"repo": repo, "features": len(feats), "passing": ok, "started_passing": start_pass, "sessions": sessions,
            "tokens": tokens_in + tokens_out, "cost": (tokens_in * pin + tokens_out * pout) / 1e6,
            "healthy": all(s == "PASS" for t, s in final.items() if t not in {f["test"] for f in feats}),
            "declared_done": declared_done, "log": log}


# ---------------------------------------------------------------- Part 7: small helpers
def four_tuple(task):
    """The four parts of a benchmark item (request, environment, stopping criteria, scorer) for one task."""
    return {"request": task.issue.split("\n")[0],
            "environment": "repo at the base commit, %d files" % len(task.base),
            "stopping": "the agent submits a patch, or runs out of turns or budget",
            "scorer": "%d FAIL_TO_PASS and %d PASS_TO_PASS tests must all pass" % (len(task.f2p), len(task.p2p))}


def parse_agents_md(text):
    """Read an AGENTS.md style file: '## Heading' sections with '- bullet' lines. Returns {heading: [bullets]}, plus
    the backticked commands found anywhere under the key '_commands'."""
    out, cur = {}, None
    for ln in text.split("\n"):
        if ln.startswith("#"):
            cur = ln.lstrip("#").strip()
            out.setdefault(cur, [])
        elif ln.strip().startswith("- ") and cur is not None:
            out[cur].append(ln.strip()[2:].strip())
    out["_commands"] = re.findall(r"`([^`]+)`", text)
    return out


# ---------------------------------------------------------------- Part 8: demo
def _run_agent(agent, tasks):
    out = []
    for t in tasks:
        res = agent.solve(t.view(), hint=t.fn)
        out.append((S.evaluate(t, res.patch).resolved, res.turns, S.cost_of(agent, res)))
    return out


def _row(label, runs):
    k, n = sum(r[0] for r in runs), len(runs)
    lo, hi = S.wilson(k, n)
    cost = mean(r[2] for r in runs)
    cps = "%.3f" % (sum(r[2] for r in runs) / k) if k else "n/a"
    return "%-26s %2d/%d (95%% CI %2.0f to %2.0f%%) %6.1f %9.3f %9s" % (label, k, n, 100 * lo, 100 * hi, mean(r[1] for r in runs), cost, cps)


def demo():
    tasks = S.curated_tasks()
    syn, _ = S.synthesize_tasks(limit=40)
    pool = tasks + syn
    hdr = "%-26s %-26s %6s %9s %9s" % ("system", "resolved (Wilson CI)", "turns", "$/task", "$/solved")

    print("== 1. One task through the interface (%s)" % tasks[2].id)
    t = tasks[2]
    ws = Workspace(t.base, window=8, search_cap=3, lint="syntax", deny=("tests/",))
    a, b, _ = S.code_units(t.module, t.base[t.module])["startswith_any"]
    fix = "    return any(s.startswith(p) for p in prefixes)"
    for cmd, args in [("search_dir", ("prefixes",)), ("open", (t.module, a)), ("python", ("startswith_any('foobar', ['foo'])",)),
                      ("edit", (a + 1, b, "    return (")), ("edit", (a + 1, b, fix)),
                      ("python", ("startswith_any('foobar', ['foo'])",)), ("search_dir", ("def",))]:
        obs = ws.act(cmd, *args)
        print("> %s %s\n%s" % (cmd, ", ".join(repr(x) for x in args), "\n".join("    " + x for x in obs.split("\n")[:11])))
    ws.open(t.test_path)
    print("> edit on a test file ->", ws.act("edit", 1, 1, "pass"))
    print("turns %d, input tokens %d, output tokens %d (unscaled), stats %s" % (ws.ledger.turns, ws.ledger.tin, ws.ledger.tout, dict(ws.stats)))

    print("\n== 2. The four parts of a task (request, environment, stopping criteria, scorer)")
    for k, v in four_tuple(t).items():
        print("  %-12s %s" % (k, v))

    print("\n== 3. Interface ablations on %d tasks (20 curated + 40 synthesized). Slip chance 0.4 is made up." % len(pool))
    print(hdr)
    for label, ag in [("lint syntax, no slips", AciAgent("a1")),
                      ("lint none, slips", AciAgent("a2", lint="none", slip=0.4)),
                      ("lint syntax, slips", AciAgent("a3", lint="syntax", slip=0.4)),
                      ("lint names, slips", AciAgent("a4", lint="names", slip=0.4))]:
        print(_row(label, _run_agent(ag, pool)))
    print("viewer window (lines) vs cost on the 20 curated tasks:")
    for w in (5, 15, 100):
        rs = _run_agent(AciAgent("w", window=w), tasks)
        print("  window %3d: resolved %d/20, $/task %.3f" % (w, sum(r[0] for r in rs), mean(r[2] for r in rs)))

    print("\n== 4. Agent loop vs fixed pipeline (same %d tasks)" % len(pool))
    print(hdr + "   (turns = model calls for the pipeline)")
    print(_row("agent loop (ACI)", _run_agent(AciAgent("loop"), pool)))
    for sel in ("none", "first", "vote"):
        print(_row("pipeline, select=" + sel, _run_agent(AgentlessPipeline("p-" + sel, select=sel), pool)))
    print("more samples (select=vote):")
    for n in (1, 4, 8, 16, 32):
        rs = _run_agent(AgentlessPipeline("n%d" % n, n_samples=n), pool)
        print("  n=%2d: resolved %d/%d, $/task %.3f" % (n, sum(r[0] for r in rs), len(rs), mean(r[2] for r in rs)))

    print("\n== 5. Reproduction scripts")
    rq = repro_quality(tasks)
    print("generated test fails before and passes after the gold patch: %s" % dict(Counter(rq.values())))
    repro = S.parse_repro(t.issue)
    print(repro_script(repro, "from textkit.text import *"))
    print("script on the buggy code: %s; on the gold patch: %s" % (
        run_repro(t.base, t.module, repro), run_repro(S.apply_patch(t.base, t.gold_patch)[0], t.module, repro)))

    print("== 6. Choosing among sampled patches (curated 20, 5 seeds)")
    for skew, n in ((2.0, 8), (1.0, 16)):
        st = selection_study(tasks, n_samples=n, seeds=5, skew=skew)
        print("n=%d samples, skew %.1f (1.0 is uniform; larger favours early candidates)" % (n, skew))
        for k, v in st.items():
            lo, hi = S.bootstrap_ci(v)
            print("  %-20s %5.1f of 20 resolved (task bootstrap %2.0f to %2.0f%%)" % (k, sum(v), 100 * lo, 100 * hi))

    print("\n== 7. Repo map")
    syn_repo = synthetic_repo()
    for b in (40, 120, 400):
        text, info = repo_map(syn_repo, budget=b)
        print("  budget %3d: %2d/%2d files, %2d/%2d symbols, %3d of %3d tokens" % (
            b, info["files"], info["files_total"], info["symbols"], info["symbols_total"], info["tokens"], info["full_tokens"]))
    _, info = repo_map(syn_repo, budget=400)
    top = sorted(info["rank"].items(), key=lambda x: -x[1])[:3]
    print("  most central files:", ", ".join("%s %.2f" % kv for kv in top))
    merged = {}
    for x in tasks:
        merged.update(x.base)
    text, _ = repo_map(merged, budget=120, focus=(t.module,))
    print("  map with focus on %s (budget 120):\n%s" % (t.module, "\n".join("    " + x for x in text.split("\n"))))
    for masked in (False, True):
        print("  localization, issue names the function = %s: %s" % (not masked, {k: round(v, 2) for k, v in localize_study(tasks, masked).items()}))

    print("\n== 8. Long-running harness (5 packages x 5 seeds, crash chance 0.3 per attempt, all made up)")
    cfgs = [("bare (no list, notes or commits)", dict(feature_list=False, progress=False, commits=False)),
            ("feature list only", dict(feature_list=True, progress=False, commits=False)),
            ("list + progress notes", dict(feature_list=True, progress=True, commits=False)),
            ("list + notes + commits", dict(feature_list=True, progress=True, commits=True))]
    print("%-34s %8s %9s %9s %8s" % ("setup", "passing", "sessions", "$/run", "healthy"))
    for label, kw in cfgs:
        rows = [run_long(r, HarnessConfig(**kw), seed=s) for r in S.REPOS for s in range(5)]
        print("%-34s %3d/%3d %9.1f %9.2f %5d/25" % (label, sum(r["passing"] for r in rows), sum(r["features"] for r in rows),
                                                    mean(r["sessions"] for r in rows), mean(r["cost"] for r in rows),
                                                    sum(r["healthy"] for r in rows)))
    print("full setup as the crash chance grows:")
    for c in (0.0, 0.3, 0.6):
        rows = [run_long(r, HarnessConfig(crash=c), seed=s) for r in S.REPOS for s in range(5)]
        print("  crash %.1f: passing %d/%d, sessions %.1f" % (c, sum(r["passing"] for r in rows), sum(r["features"] for r in rows),
                                                          mean(r["sessions"] for r in rows)))

    print("\n== 9. Cost and turns")
    led = Ledger("mid")
    cum = []
    for i in range(1, 31):
        led.turn(60, 250)
        cum.append(led.tin)
    print("  input tokens after 10, 20, 30 turns: %d, %d, %d (unscaled); the 30 turn run costs %.1f times the 10 turn run" % (
        cum[9], cum[19], cum[29], cum[29] / cum[9]))
    for cf in (0.0, 0.5, 0.9):
        print("  30 turns, %2.0f%% of input billed as cached at 0.1x: $%.2f" % (100 * cf, led.cost(cached_frac=cf)))

    print("\n== 10. An AGENTS.md file")
    md = "# AGENTS.md\n## Setup\n- Install: `pip install -e .`\n- Test: `python -m unittest`\n## Style\n- Keep functions short\n"
    p = parse_agents_md(md)
    print("  sections: %s; commands: %s" % ([k for k in p if k != "_commands"], p["_commands"]))


if __name__ == "__main__":
    demo()

Run the demo

$ python coding_agents_toolkit.py
== 1. One task through the interface (textkit-3)
> search_dir 'prefixes'
    Found 2 match(es) for "prefixes":
    textkit/text.py:12: def startswith_any(s, prefixes):
    textkit/text.py:13: return any(s.endswith(p) for p in prefixes)
> open 'textkit/text.py', 12
    [File: textkit/text.py (18 lines total)]
    (7 more lines above)
    8:         return s
    9:     return s[: n - 3] + "..."
    10: 
    11: 
    12: def startswith_any(s, prefixes):
    13:     return any(s.endswith(p) for p in prefixes)
    14: 
    15: 
    (3 more lines below)
> python "startswith_any('foobar', ['foo'])"
    => False
> edit 13, 13, '    return ('
    Your edit was not applied (SyntaxError: '(' was never closed (line 13)).
    This is how it would have looked:
    11: 
    12: def startswith_any(s, prefixes):
    13:     return (
    14: 
    15: 
    Fix the edit and try again.
> edit 13, 13, '    return any(s.startswith(p) for p in prefixes)'
    File updated.
    [File: textkit/text.py (18 lines total)]
    (9 more lines above)
    10: 
    11: 
    12: def startswith_any(s, prefixes):
    13:     return any(s.startswith(p) for p in prefixes)
    14: 
    15: 
    16: def is_palindrome(s):
    17:     chars = [c.lower() for c in s if c.isalnum()]
> python "startswith_any('foobar', ['foo'])"
    => True
> search_dir 'def'
    More than 3 matches for "def". Please narrow your search.
> edit on a test file -> Edit blocked: tests/test_text.py matches a protected path.
turns 8, input tokens 6900, output tokens 295 (unscaled), stats {'rejected': 1, 'edits': 1, 'blocked': 1}

== 2. The four parts of a task (request, environment, stopping criteria, scorer)
  request      startswith_any() looks at the end of the string
  environment  repo at the base commit, 3 files
  stopping     the agent submits a patch, or runs out of turns or budget
  scorer       1 FAIL_TO_PASS and 4 PASS_TO_PASS tests must all pass

== 3. Interface ablations on 60 tasks (20 curated + 40 synthesized). Slip chance 0.4 is made up.
system                     resolved (Wilson CI)        turns    $/task  $/solved
lint syntax, no slips      35/60 (95% CI 46 to 70%)   15.7     1.102     1.890
lint none, slips           20/60 (95% CI 23 to 46%)   20.8     1.659     4.977
lint syntax, slips         25/60 (95% CI 30 to 54%)   19.9     1.625     3.899
lint names, slips          35/60 (95% CI 46 to 70%)   17.2     1.342     2.301
viewer window (lines) vs cost on the 20 curated tasks:
  window   5: resolved 14/20, $/task 0.630
  window  15: resolved 14/20, $/task 0.690
  window 100: resolved 14/20, $/task 0.730

== 4. Agent loop vs fixed pipeline (same 60 tasks)
system                     resolved (Wilson CI)        turns    $/task  $/solved   (turns = model calls for the pipeline)
agent loop (ACI)           35/60 (95% CI 46 to 70%)   15.7     1.102     1.890
pipeline, select=none      20/60 (95% CI 23 to 46%)    4.0     0.097     0.290
pipeline, select=first     27/60 (95% CI 33 to 58%)    5.0     0.107     0.239
pipeline, select=vote      27/60 (95% CI 33 to 58%)    5.0     0.107     0.239
more samples (select=vote):
  n= 1: resolved 20/60, $/task 0.051
  n= 4: resolved 22/60, $/task 0.075
  n= 8: resolved 27/60, $/task 0.107
  n=16: resolved 33/60, $/task 0.171
  n=32: resolved 36/60, $/task 0.299

== 5. Reproduction scripts
generated test fails before and passes after the gold patch: {'no repro': 2, 'valid': 18}
from textkit.text import *


def main():
    try:
        got = startswith_any('foobar', ['foo'])
    except Exception:
        print("Other issues")
        return
    if got == True:
        print("Issue resolved")
    else:
        print("Issue reproduced")


main()

script on the buggy code: Issue reproduced; on the gold patch: Issue resolved
== 6. Choosing among sampled patches (curated 20, 5 seeds)
n=8 samples, skew 2.0 (1.0 is uniform; larger favours early candidates)
  greedy                 6.0 of 20 resolved (task bootstrap 10 to 55%)
  random                 4.2 of 20 resolved (task bootstrap 10 to 36%)
  visible-tests first    8.4 of 20 resolved (task bootstrap 24 to 62%)
  visible-tests vote     7.4 of 20 resolved (task bootstrap 20 to 56%)
  repro+tests first     11.2 of 20 resolved (task bootstrap 39 to 73%)
  repro+tests vote      11.2 of 20 resolved (task bootstrap 39 to 73%)
  oracle                11.2 of 20 resolved (task bootstrap 39 to 73%)
n=16 samples, skew 1.0 (1.0 is uniform; larger favours early candidates)
  greedy                 6.0 of 20 resolved (task bootstrap 10 to 55%)
  random                 2.0 of 20 resolved (task bootstrap  4 to 17%)
  visible-tests first    7.6 of 20 resolved (task bootstrap 18 to 61%)
  visible-tests vote     5.0 of 20 resolved (task bootstrap 13 to 38%)
  repro+tests first     12.6 of 20 resolved (task bootstrap 45 to 81%)
  repro+tests vote      12.0 of 20 resolved (task bootstrap 42 to 79%)
  oracle                12.6 of 20 resolved (task bootstrap 45 to 81%)

== 7. Repo map
  budget  40:  3/13 files,  9/36 symbols,  40 of 610 tokens
  budget 120:  9/13 files, 26/36 symbols, 117 of 610 tokens
  budget 400: 12/13 files, 36/36 symbols, 163 of 610 tokens
  most central files: pkg/m00.py 0.39, pkg/m02.py 0.10, pkg/m01.py 0.08
  map with focus on textkit/text.py (budget 120):
    textkit/text.py:
      def slugify(s)
      def truncate(s, n)
      def startswith_any(s, prefixes)
      def is_palindrome(s)
    cartkit/cart.py:
      class Cart
        def __init__(self, discount=0.0)
        def add(self, name, price, qty=1)
        def subtotal(self)
        def total(self)
        def discount_amount(self)
        def can_checkout(self)
    configkit/config.py:
      def parse_kv(line)
      def get_path(d, path, default=None)
      def merge(a, b)
      def to_bool(s)
    listkit/lists.py:
      def chunk(xs, n)
  localization, issue names the function = True: {'bodies file@1': 0.95, 'bodies function@3': 0.95, 'signatures file@1': 0.95, 'signatures function@3': 0.95}
  localization, issue names the function = False: {'bodies file@1': 0.55, 'bodies function@3': 0.45, 'signatures file@1': 0.45, 'signatures function@3': 0.35}

== 8. Long-running harness (5 packages x 5 seeds, crash chance 0.3 per attempt, all made up)
setup                               passing  sessions     $/run  healthy
bare (no list, notes or commits)     0/100       1.0      0.01    25/25
feature list only                   14/100       7.5      6.44     4/25
list + progress notes               14/100       4.3      3.17     4/25
list + notes + commits              73/100       3.8      3.73    25/25
full setup as the crash chance grows:
  crash 0.0: passing 80/100, sessions 2.6
  crash 0.3: passing 73/100, sessions 3.8
  crash 0.6: passing 47/100, sessions 5.7

== 9. Cost and turns
  input tokens after 10, 20, 30 turns: 19950, 70900, 152850 (unscaled); the 30 turn run costs 7.7 times the 10 turn run
  30 turns,  0% of input billed as cached at 0.1x: $6.40
  30 turns, 50% of input billed as cached at 0.1x: $3.65
  30 turns, 90% of input billed as cached at 0.1x: $1.45

== 10. An AGENTS.md file
  sections: ['AGENTS.md', 'Setup', 'Style']; commands: ['pip install -e .', 'python -m unittest']

Exercises

1. Walk a task by hand. Create a Workspace from a toy task's files and solve it by calling act yourself. How many turns did you use compared with AciAgent, and where did it spend turns that you did not [1]?
2. A third linter. Add a mode that also rejects an edit that removes a function that the tests import. Which of the demo's four settings does it change, and what does it cost in turns when an edit is refused [1]?
3. Break the cap. Run the agent with search_cap=1000 and with search_cap=3 and compare turns and tokens. Is there a cap that is best on all 60 tasks, and does it match the paper's finding about iterative search [1]?
4. The viewer window. Sweep the window from 5 to 100 lines on the 40 synthesized tasks, not only the 20 curated ones. Does the cost curve have a minimum? Why might it be flat in the demo [1]?
5. Enforce and instruct. The protected-path rule blocks edits to tests. Write the same rule as an instruction in the issue text instead, and write an agent that sometimes ignores it. Count how often a test file is edited with each. Which row of the table in 2.5 does this correspond to [3]?
6. Sampling and selection. Change the sampler's skew and the number of samples, and find a setting where voting beats "first passing". What does the sampler have to look like for that to happen, and is it a believable property of a model [6]?
7. A bad reproduction test. Write an issue whose example is ambiguous so that repro_quality marks the generated test as invalid. What does a pipeline do when it has only a bad reproduction test [6]?
8. The repo map budget. Run repo_map on one of the toy packages with budgets of 30, 60 and 120 and with and without a focus file. Which symbols appear first, and do they match the ones that a bug fix would need [9]?
9. Hubs. Use synthetic_repo with 30 modules. How do the most central files change, and does the budget needed to show every file grow linearly with the number of files?
10. Harness pieces. Switch on the harness pieces one at a time, in every order, and fill a table of features passing and sessions used. Which piece matters most when the crash chance is 0, and which when it is 0.6 [4]?
11. A cache price. Using the ledger, find the cached fraction at which a 30-turn run costs the same as a 15-turn run with no caching. Then explain, in your own words, why a real provider's cache rules may make this number wrong [3].
12. An AGENTS.md check. Parse the repository's own README or your own AGENTS.md with parse_agents_md. Which commands does it find, and which instructions in it could a deny rule enforce instead [10]?

The tests: test_coding_agents_toolkit.py

The 148 tests check the ledger (the quadratic growth of input tokens and the cache price); the linter (syntax and name errors, the three modes); the workspace (the window and its default of 100 lines, scrolling, going to a line, the search cap and its boundary, the edit that is refused and how the file is shown, the protected-path rule, reverting, the diff, and the output tokens charged per action); the scripted agent (a slip with and without a linter, the turn limit, a vague task); the reproduction scripts (the three labels, a script with set-up lines, the quality check); the pipeline (the candidate pool and its offsets, the sample draw, validation, the three selection modes, and token and turn counts pinned to exact values); the selection study; the symbol index, the reference counts, the file graph and the ranking (a value pinned on a two-node case, and weights); the repo map (the budget, the class header of a method, the uses counted only from other files, the focus boost); the synthetic repository and the localization study; the long-running harness (each piece switched off in turn, a crash, the cost with the tier price, the attempt limit); the 4-tuple; and the AGENTS.md parser.

Mutation check. This was a light, sampled run, not an exhaustive one (the toolkit has 364 places where such a change can be made, and I sampled 70 and then 110 of them at random, with a fixed seed of 0). The first sample of 70 was caught 44 times. For the 26 misses I looked at each one and added tests for the real gaps (the pagerank weights, the token counts for the pipeline and the agent, the default viewer window, the symbol use counts, the cost of the long-running harness, the attempt limit, and the exact rejected-edit message). The second sample of 110 was caught 90 times, and its tests changed while it ran, so its count is a snapshot. I then re-ran the survivors from both samples against the final tests, each at every place on its line where the change applies. That gave 49 distinct changes: 42 were caught and 7 survived. The 7 are on 5 lines. They are a word in a description string (the stopping-criteria text, twice), the floor of 1 for the line count of an empty file, the index inserted into the module path at import, the start offset of the viewer when no line is given, and two changes to the slip test (the and/or and the strict comparison), which I judge to be equivalent when the slip rate is zero or to differ only on an event of probability zero. I did not pin these, and that is my judgement, not a proof. That makes 148 tests in all. I did not run the whole sample again after the last tests were added, apart from the re-run of the survivors.

$ python -m unittest -v test_coding_agents_toolkit
test_agent_never_changes_tests (test_coding_agents_toolkit.TestAciAgent.test_agent_never_changes_tests) ... ok
test_cost_through_swe_bench_helper (test_coding_agents_toolkit.TestAciAgent.test_cost_through_swe_bench_helper) ... ok
test_curated_rate_and_pareto (test_coding_agents_toolkit.TestAciAgent.test_curated_rate_and_pareto) ... ok
test_deterministic (test_coding_agents_toolkit.TestAciAgent.test_deterministic) ... ok
test_names_lint_recovers_name_slips (test_coding_agents_toolkit.TestAciAgent.test_names_lint_recovers_name_slips) ... ok
test_slip_with_lint_costs_turns_not_results (test_coding_agents_toolkit.TestAciAgent.test_slip_with_lint_costs_turns_not_results) ... ok
test_slip_without_lint_hurts (test_coding_agents_toolkit.TestAciAgent.test_slip_without_lint_hurts) ... ok
test_solves_easy_task (test_coding_agents_toolkit.TestAciAgent.test_solves_easy_task) ... ok
test_turn_budget (test_coding_agents_toolkit.TestAciAgent.test_turn_budget) ... ok
test_vague_task_has_no_example_to_check (test_coding_agents_toolkit.TestAciAgent.test_vague_task_has_no_example_to_check) ... ok
test_window_changes_cost_not_outcome (test_coding_agents_toolkit.TestAciAgent.test_window_changes_cost_not_outcome) ... ok
test_without_tests_still_submits (test_coding_agents_toolkit.TestAciAgent.test_without_tests_still_submits) ... ok
test_deny_rule (test_coding_agents_toolkit.TestEdit.test_deny_rule) ... ok
test_edit_can_grow_and_shrink (test_coding_agents_toolkit.TestEdit.test_edit_can_grow_and_shrink) ... ok
test_edit_errors (test_coding_agents_toolkit.TestEdit.test_edit_errors) ... ok
test_edit_rejects_syntax_error_and_leaves_file (test_coding_agents_toolkit.TestEdit.test_edit_rejects_syntax_error_and_leaves_file) ... ok
test_edit_replaces_range (test_coding_agents_toolkit.TestEdit.test_edit_replaces_range) ... ok
test_lint_none_lets_bad_edit_in (test_coding_agents_toolkit.TestEdit.test_lint_none_lets_bad_edit_in) ... ok
test_lint_syntax_misses_names_but_names_catches (test_coding_agents_toolkit.TestEdit.test_lint_syntax_misses_names_but_names_catches) ... ok
test_non_python_file_not_linted (test_coding_agents_toolkit.TestEdit.test_non_python_file_not_linted) ... ok
test_view_after_edit_shows_new_text (test_coding_agents_toolkit.TestEdit.test_view_after_edit_shows_new_text) ... ok
test_attempt_cap_gives_up_on_unfixable_feature (test_coding_agents_toolkit.TestHarness.test_attempt_cap_gives_up_on_unfixable_feature) ... ok
test_bare_run_declares_done (test_coding_agents_toolkit.TestHarness.test_bare_run_declares_done) ... ok
test_config_defaults (test_coding_agents_toolkit.TestHarness.test_config_defaults) ... ok
test_crashes_without_commits_leave_damage (test_coding_agents_toolkit.TestHarness.test_crashes_without_commits_leave_damage) ... ok
test_custom_agent_is_used (test_coding_agents_toolkit.TestHarness.test_custom_agent_is_used) ... ok
test_full_harness_survives_crashes (test_coding_agents_toolkit.TestHarness.test_full_harness_survives_crashes) ... ok
test_full_harness_without_crashes_fixes_most (test_coding_agents_toolkit.TestHarness.test_full_harness_without_crashes_fixes_most) ... ok
test_max_sessions_cap (test_coding_agents_toolkit.TestHarness.test_max_sessions_cap) ... ok
test_progress_notes_reduce_sessions (test_coding_agents_toolkit.TestHarness.test_progress_notes_reduce_sessions) ... ok
test_project_features_pass_when_source_is_fixed (test_coding_agents_toolkit.TestHarness.test_project_features_pass_when_source_is_fixed) ... ok
test_project_files (test_coding_agents_toolkit.TestHarness.test_project_files) ... ok
test_run_is_deterministic (test_coding_agents_toolkit.TestHarness.test_run_is_deterministic) ... ok
test_session_context_limits_attempts (test_coding_agents_toolkit.TestHarness.test_session_context_limits_attempts) ... ok
test_visible_pass_fails_on_broken_source (test_coding_agents_toolkit.TestHarness.test_visible_pass_fails_on_broken_source) ... ok
test_four_tuple (test_coding_agents_toolkit.TestHelpers.test_four_tuple) ... ok
test_parse_agents_md (test_coding_agents_toolkit.TestHelpers.test_parse_agents_md) ... ok
test_call_has_no_history (test_coding_agents_toolkit.TestLedger.test_call_has_no_history) ... ok
test_cost_uses_tier_and_scale (test_coding_agents_toolkit.TestLedger.test_cost_uses_tier_and_scale) ... ok
test_cost_with_cache (test_coding_agents_toolkit.TestLedger.test_cost_with_cache) ... ok
test_input_grows_faster_than_turns (test_coding_agents_toolkit.TestLedger.test_input_grows_faster_than_turns) ... ok
test_result_carries_counts (test_coding_agents_toolkit.TestLedger.test_result_carries_counts) ... ok
test_turn_accumulates_history (test_coding_agents_toolkit.TestLedger.test_turn_accumulates_history) ... ok
test_lines_helpers (test_coding_agents_toolkit.TestLint.test_lines_helpers) ... ok
test_modes (test_coding_agents_toolkit.TestLint.test_modes) ... ok
test_syntax_message_has_line (test_coding_agents_toolkit.TestLint.test_syntax_message_has_line) ... ok
test_undefined_names_scoping_is_coarse_but_sane (test_coding_agents_toolkit.TestLint.test_undefined_names_scoping_is_coarse_but_sane) ... ok
test_undefined_names_sorted_by_line (test_coding_agents_toolkit.TestLint.test_undefined_names_sorted_by_line) ... ok
test_demo_runs_and_is_clean (test_coding_agents_toolkit.TestOnSynthesized.test_demo_runs_and_is_clean) ... ok
test_loop_and_pipeline_on_synthesized (test_coding_agents_toolkit.TestOnSynthesized.test_loop_and_pipeline_on_synthesized) ... ok
test_source_is_clean (test_coding_agents_toolkit.TestOnSynthesized.test_source_is_clean) ... ok
test_act_charges_exact_output_tokens (test_coding_agents_toolkit.TestPinned.test_act_charges_exact_output_tokens) ... ok
test_agent_token_counts_are_pinned (test_coding_agents_toolkit.TestPinned.test_agent_token_counts_are_pinned) ... ok
test_attempt_limit_is_exact (test_coding_agents_toolkit.TestPinned.test_attempt_limit_is_exact) ... ok
test_default_window_is_one_hundred_lines (test_coding_agents_toolkit.TestPinned.test_default_window_is_one_hundred_lines) ... ok
test_documented_defaults (test_coding_agents_toolkit.TestPinned.test_documented_defaults) ... ok
test_find_file_cap_is_inclusive (test_coding_agents_toolkit.TestPinned.test_find_file_cap_is_inclusive) ... ok
test_goto_first_line_is_allowed (test_coding_agents_toolkit.TestPinned.test_goto_first_line_is_allowed) ... ok
test_harness_cost_uses_the_tier_price (test_coding_agents_toolkit.TestPinned.test_harness_cost_uses_the_tier_price) ... ok
test_harness_without_notes_reads_the_whole_repo_each_session (test_coding_agents_toolkit.TestPinned.test_harness_without_notes_reads_the_whole_repo_each_session) ... ok
test_localization_study_is_pinned (test_coding_agents_toolkit.TestPinned.test_localization_study_is_pinned) ... ok
test_method_header_belongs_to_its_own_class (test_coding_agents_toolkit.TestPinned.test_method_header_belongs_to_its_own_class) ... ok
test_pagerank_divides_by_total_weight (test_coding_agents_toolkit.TestPinned.test_pagerank_divides_by_total_weight) ... ok
test_pagerank_single_node (test_coding_agents_toolkit.TestPinned.test_pagerank_single_node) ... ok
test_pipeline_token_counts_are_pinned (test_coding_agents_toolkit.TestPinned.test_pipeline_token_counts_are_pinned) ... ok
test_pool_candidates_change_only_the_unit (test_coding_agents_toolkit.TestPinned.test_pool_candidates_change_only_the_unit) ... ok
test_rejected_edit_message_is_pinned (test_coding_agents_toolkit.TestPinned.test_rejected_edit_message_is_pinned) ... ok
test_row_format (test_coding_agents_toolkit.TestPinned.test_row_format) ... ok
test_row_interval_is_in_percent (test_coding_agents_toolkit.TestPinned.test_row_interval_is_in_percent) ... ok
test_run_repro_uses_setup (test_coding_agents_toolkit.TestPinned.test_run_repro_uses_setup) ... ok
test_symbol_use_counts_only_other_files (test_coding_agents_toolkit.TestPinned.test_symbol_use_counts_only_other_files) ... ok
test_synthetic_repo_defaults_and_values (test_coding_agents_toolkit.TestPinned.test_synthetic_repo_defaults_and_values) ... ok
test_undefined_names_tie_break_by_name (test_coding_agents_toolkit.TestPinned.test_undefined_names_tie_break_by_name) ... ok
test_window_larger_than_the_file_starts_at_line_one (test_coding_agents_toolkit.TestPinned.test_window_larger_than_the_file_starts_at_line_one) ... ok
test_cost_is_far_below_agent_loop (test_coding_agents_toolkit.TestPipelineEndToEnd.test_cost_is_far_below_agent_loop) ... ok
test_counts (test_coding_agents_toolkit.TestPipelineEndToEnd.test_counts) ... ok
test_log_mentions_counts (test_coding_agents_toolkit.TestPipelineEndToEnd.test_log_mentions_counts) ... ok
test_more_samples_help (test_coding_agents_toolkit.TestPipelineEndToEnd.test_more_samples_help) ... ok
test_nothing_validates_submits_nothing (test_coding_agents_toolkit.TestPipelineEndToEnd.test_nothing_validates_submits_nothing) ... ok
test_patch_only_touches_source (test_coding_agents_toolkit.TestPipelineEndToEnd.test_patch_only_touches_source) ... ok
test_turns_are_model_calls (test_coding_agents_toolkit.TestPipelineEndToEnd.test_turns_are_model_calls) ... ok
test_validation_beats_no_validation (test_coding_agents_toolkit.TestPipelineEndToEnd.test_validation_beats_no_validation) ... ok
test_choose_modes (test_coding_agents_toolkit.TestPipelinePieces.test_choose_modes) ... ok
test_choose_requires_both_checks (test_coding_agents_toolkit.TestPipelinePieces.test_choose_requires_both_checks) ... ok
test_localize_finds_the_function (test_coding_agents_toolkit.TestPipelinePieces.test_localize_finds_the_function) ... ok
test_localize_skips_tests (test_coding_agents_toolkit.TestPipelinePieces.test_localize_skips_tests) ... ok
test_normalized_key_ignores_format_and_comments (test_coding_agents_toolkit.TestPipelinePieces.test_normalized_key_ignores_format_and_comments) ... ok
test_pool_candidates_differ_from_base (test_coding_agents_toolkit.TestPipelinePieces.test_pool_candidates_differ_from_base) ... ok
test_sample_greedy_first_and_deterministic (test_coding_agents_toolkit.TestPipelinePieces.test_sample_greedy_first_and_deterministic) ... ok
test_sample_skew_favours_early (test_coding_agents_toolkit.TestPipelinePieces.test_sample_skew_favours_early) ... ok
test_skeleton (test_coding_agents_toolkit.TestPipelinePieces.test_skeleton) ... ok
test_validate_dedupes (test_coding_agents_toolkit.TestPipelinePieces.test_validate_dedupes) ... ok
test_validate_flags_bad_candidates (test_coding_agents_toolkit.TestPipelinePieces.test_validate_flags_bad_candidates) ... ok
test_validate_without_repro_passes_repro_check (test_coding_agents_toolkit.TestPipelinePieces.test_validate_without_repro_passes_repro_check) ... ok
test_vote_tie_goes_to_earliest (test_coding_agents_toolkit.TestPipelinePieces.test_vote_tie_goes_to_earliest) ... ok
test_file_graph_edges (test_coding_agents_toolkit.TestRepoMap.test_file_graph_edges) ... ok
test_file_graph_ignores_own_definitions (test_coding_agents_toolkit.TestRepoMap.test_file_graph_ignores_own_definitions) ... ok
test_focus_moves_a_file_up (test_coding_agents_toolkit.TestRepoMap.test_focus_moves_a_file_up) ... ok
test_hub_listed_first (test_coding_agents_toolkit.TestRepoMap.test_hub_listed_first) ... ok
test_localize_study_bodies_help_when_names_are_missing (test_coding_agents_toolkit.TestRepoMap.test_localize_study_bodies_help_when_names_are_missing) ... ok
test_map_grows_with_budget (test_coding_agents_toolkit.TestRepoMap.test_map_grows_with_budget) ... ok
test_map_is_smaller_than_the_code (test_coding_agents_toolkit.TestRepoMap.test_map_is_smaller_than_the_code) ... ok
test_map_lines_nest_methods_under_class (test_coding_agents_toolkit.TestRepoMap.test_map_lines_nest_methods_under_class) ... ok
test_map_respects_budget (test_coding_agents_toolkit.TestRepoMap.test_map_respects_budget) ... ok
test_method_pulls_in_its_class_header (test_coding_agents_toolkit.TestRepoMap.test_method_pulls_in_its_class_header) ... ok
test_pagerank_personalization (test_coding_agents_toolkit.TestRepoMap.test_pagerank_personalization) ... ok
test_pagerank_sums_to_one_and_favours_hub (test_coding_agents_toolkit.TestRepoMap.test_pagerank_sums_to_one_and_favours_hub) ... ok
test_pagerank_weights_matter (test_coding_agents_toolkit.TestRepoMap.test_pagerank_weights_matter) ... ok
test_reference_counts (test_coding_agents_toolkit.TestRepoMap.test_reference_counts) ... ok
test_symbol_index (test_coding_agents_toolkit.TestRepoMap.test_symbol_index) ... ok
test_symbol_index_skips_bad_and_non_py (test_coding_agents_toolkit.TestRepoMap.test_symbol_index_skips_bad_and_non_py) ... ok
test_synthetic_repo_is_valid_and_deterministic (test_coding_agents_toolkit.TestRepoMap.test_synthetic_repo_is_valid_and_deterministic) ... ok
test_zero_budget (test_coding_agents_toolkit.TestRepoMap.test_zero_budget) ... ok
test_add_repro_test_appends_failing_test (test_coding_agents_toolkit.TestRepro.test_add_repro_test_appends_failing_test) ... ok
test_add_repro_test_keeps_other_files (test_coding_agents_toolkit.TestRepro.test_add_repro_test_keeps_other_files) ... ok
test_repro_quality (test_coding_agents_toolkit.TestRepro.test_repro_quality) ... ok
test_repro_quality_detects_a_test_that_passes_before (test_coding_agents_toolkit.TestRepro.test_repro_quality_detects_a_test_that_passes_before) ... ok
test_repro_quality_detects_wrong_expectation (test_coding_agents_toolkit.TestRepro.test_repro_quality_detects_wrong_expectation) ... ok
test_run_repro_other_issues (test_coding_agents_toolkit.TestRepro.test_run_repro_other_issues) ... ok
test_script_is_valid_python_with_three_labels (test_coding_agents_toolkit.TestRepro.test_script_is_valid_python_with_three_labels) ... ok
test_script_runs_and_matches_run_repro (test_coding_agents_toolkit.TestRepro.test_script_runs_and_matches_run_repro) ... ok
test_script_with_setup_lines (test_coding_agents_toolkit.TestRepro.test_script_with_setup_lines) ... ok
test_act_charges_ledger_and_logs (test_coding_agents_toolkit.TestRunAndSubmit.test_act_charges_ledger_and_logs) ... ok
test_act_rejects_unknown_command (test_coding_agents_toolkit.TestRunAndSubmit.test_act_rejects_unknown_command) ... ok
test_python_needs_open_python_file (test_coding_agents_toolkit.TestRunAndSubmit.test_python_needs_open_python_file) ... ok
test_revert_and_diff (test_coding_agents_toolkit.TestRunAndSubmit.test_revert_and_diff) ... ok
test_revert_removes_new_file (test_coding_agents_toolkit.TestRunAndSubmit.test_revert_removes_new_file) ... ok
test_run_tests_summary (test_coding_agents_toolkit.TestRunAndSubmit.test_run_tests_summary) ... ok
test_find_file (test_coding_agents_toolkit.TestSearch.test_find_file) ... ok
test_search_cap_suppresses_listing (test_coding_agents_toolkit.TestSearch.test_search_cap_suppresses_listing) ... ok
test_search_dir_hits_have_path_and_line (test_coding_agents_toolkit.TestSearch.test_search_dir_hits_have_path_and_line) ... ok
test_search_file (test_coding_agents_toolkit.TestSearch.test_search_file) ... ok
test_search_none (test_coding_agents_toolkit.TestSearch.test_search_none) ... ok
test_cache_is_reused (test_coding_agents_toolkit.TestSelectionStudy.test_cache_is_reused) ... ok
test_oracle_is_upper_bound (test_coding_agents_toolkit.TestSelectionStudy.test_oracle_is_upper_bound) ... ok
test_shape (test_coding_agents_toolkit.TestSelectionStudy.test_shape) ... ok
test_tests_beat_random (test_coding_agents_toolkit.TestSelectionStudy.test_tests_beat_random) ... ok
test_name_slip_falls_back_to_paren (test_coding_agents_toolkit.TestSlip.test_name_slip_falls_back_to_paren) ... ok
test_name_slip_parses_but_is_undefined (test_coding_agents_toolkit.TestSlip.test_name_slip_parses_but_is_undefined) ... ok
test_name_slip_skips_self (test_coding_agents_toolkit.TestSlip.test_name_slip_skips_self) ... ok
test_paren_slip_is_syntax_error (test_coding_agents_toolkit.TestSlip.test_paren_slip_is_syntax_error) ... ok
test_goto_scroll (test_coding_agents_toolkit.TestViewer.test_goto_scroll) ... ok
test_open_centers_on_line (test_coding_agents_toolkit.TestViewer.test_open_centers_on_line) ... ok
test_open_clamps_at_end (test_coding_agents_toolkit.TestViewer.test_open_clamps_at_end) ... ok
test_open_missing (test_coding_agents_toolkit.TestViewer.test_open_missing) ... ok
test_open_shows_header_and_numbers (test_coding_agents_toolkit.TestViewer.test_open_shows_header_and_numbers) ... ok
test_scroll_up_stops_at_top (test_coding_agents_toolkit.TestViewer.test_scroll_up_stops_at_top) ... ok
test_window_is_a_cap (test_coding_agents_toolkit.TestViewer.test_window_is_a_cap) ... ok

----------------------------------------------------------------------
Ran 148 tests in 16.490s

OK

5. Discussion questions

Use these in a study group or a class. Try to answer with an example and a source. Give each about 5 to 10 minutes.

1. In a benchmark the scorer is hidden and the agent cannot see it. In everyday use, nothing plays that role unless you supply it. List four things that can play it, and say which of them the agent can edit or fool [3].
2. SWE-agent's four principles are simple actions, compact actions, concise feedback and guardrails. Pick one command in the lab's Workspace and say which principles it follows and which it breaks [1].
3. Iterative search did worse than no search at all in the paper's ablation. Name another tool for which "more information per call" could be worse than "less", and say how you would test it [1].
4. A leaderboard row bundles a model, a scaffold, a budget and a prompt. If you could publish only three extra numbers next to every row, which would you choose, and why [2] [17]?
5. The OpenHands table compares its agent with SWE-agent, but the models and the prompts differ. What would a fair comparison need, and what would it cost to run [2]?
6. Claude Code separates instructions that the model may ignore from rules that the program enforces. Choose three rules for a project of your own, and say for each whether it belongs in CLAUDE.md (or AGENTS.md) or in a deny rule or hook [3] [10].
7. Agentless matches an agent loop on Lite at a lower cost, and its authors also find that some Lite tasks leak the answer in the issue text. How much of "no agent needed" could be a property of the benchmark and not of the method [6]?
8. The Aider repo map spends a token budget on the most important symbols. What does "important" mean in a graph of file dependencies, and what kinds of code does that ranking hide [9]?
9. Voting among passing patches assumes that right answers are repeated more often than wrong ones. Give a case where that is false, and a check you could run before you trust a vote [6].
10. The long-running post reports no numbers. Design a small experiment, with a made-up project, that would show whether a progress file, a feature list and git commits each help. What would you hold fixed [4]?
11. An agent loop's input cost grows with the square of its turns. Describe three ways to reduce it, and what each gives up [1] [3] [20].
12. METR's 2025 result was a 19% increase in task time, and its 2026 update has -18% for returning developers with data that METR calls unreliable. A colleague says "AI slows developers down". How do you answer in two sentences [12] [13]?
13. Several sources here are written by the people who built the tool, the benchmark or the model, and one of the course's instructors is the first author of SWE-agent. How should a student weigh a builder's account of its own work, and what independent evidence is in this pack [1] [12] [18] [19]?

6. Quiz (10 questions)

Pick one answer for each question, then press Check answers. Check answers 0 of 10 answered.

1. Compare an agent used for everyday work with an agent run on a benchmark. Which part of the 4-tuple is usually missing in everyday use unless you add it [3]?
   • The request, because the agent has to work out for itself what you want from the files that it reads
   • The stopping criteria, because the agent always keeps working until you tell it to stop
   • The environment, because the agent runs on your own machine and has no fixed set of tools to use there, unlike in a benchmark
   • The scorer, because nothing hidden judges the result unless you add tests, a typecheck, a hook or a review
2. In SWE-agent, what happens when an edit would leave a Python file with a syntax error [1]?
   • The edit is discarded, and the agent is shown the error together with how the file would have looked
   • The edit is applied anyway, and the error only shows up the next time that the agent runs the tests or the code
   • The edit is applied, and the interface then reverts the whole file to the base commit at once
   • The agent is stopped, and the instance is counted as unresolved because the edit was invalid
3. In SWE-agent's ablations on SWE-bench Lite, how did the iterative search tool compare with giving the agent no search commands [1]?
   • It was better, because showing one result at a time keeps the context short and the agent focused on the task
   • It was the same, because the agent could always fall back on bash commands such as grep and find to search
   • It was worse (12.0% against 15.7%), because the agent could page through results and use up its budget
   • It was better only on large repositories, where the agent has too many files to read by itself
4. Which three parts make up the design of OpenHands, as its paper describes it [2]?
   • A planner, a critic and a memory store, which pass messages between several agents in a fixed order each time
   • An agent that maps state to action, an event stream of actions and observations, and a Docker runtime
   • A retriever, a code editor and a test runner, each of which is written specially for one benchmark task
   • A browser, a terminal and a file viewer, with no common interface between the agent and the three
5. The Claude Code best-practices page separates two kinds of instruction. Which of these is enforced by the program every time, whatever the model decides [3]?
   • A line in CLAUDE.md that says never to edit the generated client directory of the project
   • A request in the prompt to keep working until all of the tests pass and the lint is clean in the project
   • A skill that describes how the team likes its commit messages to be written and formatted in this repository
   • A deny rule or a hook in the settings, which the program applies even if the model would not choose to
6. What does Agentless do instead of running an agent loop [6]?
   • It runs fixed phases: localize, sample repair patches, and validate them with tests before choosing one
   • It lets the model call tools freely at each step, but it stops the run after a fixed number of turns, which is the usual limit
   • It fine-tunes a model on the repository first, so that no search or edit commands are needed later on, because it has learned the code
   • It asks a person to localize the bug and then lets the model write the patch without any checks
7. The Agentless paper's table gives SWE-agent with GPT-4 a cost of $2.51, and the OpenHands paper's table gives $1.67. What follows [6] [2]?
   • One of the two papers must have made an arithmetic error, which should be corrected before either is cited
   • The OpenHands number is the true cost, because that paper is newer and uses a more recent version of the agent
   • Costs should be compared only within one table with one accounting, and the cause of the gap is unknown
   • SWE-agent must have been run twice, and the average of the two numbers is the best estimate to quote
8. Which pair of failures does the long-running harness post describe for an agent working across context windows [4]?
   • Refusing to run any code, and writing so many tests that the first session ends before any work is done
   • Trying to do too much at once, and declaring the job done early after it saw that progress had been made
   • Forgetting the prompt after the first turn, and ignoring every file that the user has attached to the task
   • Making each commit too small, and spending most of the budget on reading the git history in each session
9. In the lab's cost accounting (a 600-token start and 310 tokens added per turn), how does the total input of a 30-turn run compare with a 10-turn run?
   • About 7.7 times as much, because each turn re-reads the whole history, so input grows with turns squared
   • About three times as much, because the run is three times as long and each turn costs about the same
   • About nine times as much, because the cost of a run always grows with the square of its length in turns exactly
   • About the same, because the model caches the history and so the later turns cost almost nothing
10. Which statement about METR's February 2026 update is supported by the post [13]?
    • It confirms the 19% slowdown, because the new study found the same result with a larger group of developers
    • It withdraws the 2025 study because the sample of 16 developers was found to be chosen at random
    • It shows that AI tools now make every developer faster, because all of the new estimates are below zero
    • It calls its new data an unreliable signal because of selection effects, so the 2025 figure is out of date

Answer key

7. Graded assignment and rubric

Title: Build a small coding agent in a new toy domain, compare it with a pipeline, and report it honestly.
Time: about 3 hours. Total: 100 points. Work: solo.

The task

Extend the lab code and write a short memo (two pages at most). Everything is a toy: no real model, no real repository that you do not own, no Docker and no network. Pick your own domain, such as a unit converter, a date parser or a small inventory library, with its own package, tests and at least 8 tasks written in the format of the Week 9a toolkit. Every price, probability and limit is yours and must be labelled as made up.

Part A. The interface (25 points)

• Add one new command to Workspace (for example a command that lists the functions in a file, or one that runs a single named test) and write tests for it, including a case that should fail.
• Compare two settings of one interface choice (the viewer window, the search cap or the linter mode) on your tasks. Report resolved counts with Wilson intervals, turns, and dollars per task and per solved task.
• Say which of SWE-agent's four principles your command follows, and which it strains [1].

Part B. Loop against pipeline (25 points)

• Run an AciAgent and an AgentlessPipeline on your tasks with the same accounting. Report the table of 2.6 for your domain.
• Add one reproduction test by hand for a task whose issue has no runnable example, and show the effect on the pipeline's selection.
• State whether voting helped on your tasks, and give one reason in your sampler's design for why it did or did not (2.8).

Part C. Context or harness (25 points)

Do one of the two.

• Repo map. Build the map for your package at three budgets, show which symbols appear first, and compare localization by bodies with localization by signatures on your tasks, with the issue names masked. Say how many tasks would have to differ for the comparison to mean anything (2.13).
• Harness. Build a project with at least 4 planted bugs, run the four harness setups of 2.9 at two crash chances, and report features passing, sessions and cost. Say which setup would fail first as the project grows, and why.

Part D. The memo (25 points)

• Write a memo of two pages at most that answers: what does your toy measure, which sources in this pack does each design choice follow, what are three ways it could mislead a reader, and what would you need to see before you believed a real-world claim based on a toy like it. Name every author or vendor overlap in the sources you cite.
• State what in your results is built into the toy by construction.
• State three limits of your tasks and three limits of your memo.

What to hand in

• Your coding_agents_toolkit.py (or your own file names), your package and tasks, and your tests. All tests must pass.
• A memo in PDF or Markdown with the tables from Parts A to C and the memo from Part D.
• A sources list for any reading you cite.

Rubric

Criterion	Full marks	Partial	Points
A. The interface	A new command with tests including a failing case, two settings compared with intervals, turns and cost per solved task, and the principles discussed.	No failing case, rates without intervals, or no cost per solved task.	25
B. Loop against pipeline	Both systems on the same tasks with one accounting, a hand-made reproduction test with its effect shown, and a reasoned account of voting.	Different accounting for the two systems, or no explanation of the voting result.	25
C. Context or harness	One of the two done fully, with numbers, intervals or run counts, and a reasoned answer to the closing question.	Numbers without a comparison, or a claim that goes beyond the data.	25
D. The memo	All four questions answered, overlaps named, built-in effects stated, and three limits of each kind.	A memo that repeats the lab, or missing limits.	25
Total			100

Grade bands: 90 to 100 excellent. 75 to 89 good. 60 to 74 needs work. Under 60: redo and resubmit. Plain writing is rewarded: short sentences, no filler. Label your own arithmetic as your own.

8. Glossary

Term	Meaning
ACI (agent-computer interface)	The commands and feedback through which an agent uses a computer. SWE-agent's design has a linter, a windowed viewer and capped search [1] (2.3).
Agent loop	The cycle in which the model reads, chooses an action, and sees the result (2.1).
Agentless	A pipeline of localization, repair and patch validation with no agent loop [6] (2.6).
AGENTS.md	A file convention for giving instructions to coding agents, described as "a README for agents" [10] (2.5).
Checkpoint	A saved state that you can return to. Claude Code snapshots files before edits, and the lab's harness uses commits [20] (2.5, 2.9).
Clean state	Code in the condition that would be appropriate to merge: no major bugs, orderly and documented [4] (2.9).
CLAUDE.md	A file of project instructions that Claude Code loads each session. It is an instruction that the model may ignore [3] (2.5).
CodeAct	The idea that an agent's actions should be executable code [8] (2.4).
Compaction	Summarizing a conversation when the context window fills, which can lose early instructions [20] (2.5).
Context policy	The rule that decides what the model sees on each turn (2.2, 2.7).
Deny rule	A rule in the settings that the program applies every time, whatever the model decided [3] (2.5).
Event stream	OpenHands' chronological record of actions and observations [2] (2.4).
Feature list	A file of the features that a project needs, each marked as passing or failing, that the agent updates one item at a time [4] (2.9).
Hook	A command that the program runs at a set point, such as before a tool call or when the agent tries to stop [3] (2.5).
Linter-gated edit	An edit command that checks the result and discards the edit if the check fails [1] (2.3).
Oracle (selector)	The best possible choice among candidates: the ceiling for any selector (2.8).
Prompt caching	A lower price for a repeated prefix of a prompt (2.10).
Protected path	A file that the interface refuses to edit, such as a test file (2.3).
Reproduction test	A script made from an issue that fails on the buggy code and passes on a fix [6] (2.8).
Repo map	A compact, ranked list of the most important symbols in a repository, fitted to a token budget [9] (2.7).
Scaffold	Everything around the model: interface, loop, context policy, environment, verification, memory and budget (2.2).
Selector	A rule that picks one patch from many candidates, such as first passing or a vote (2.8).
Session	One run of an agent in one context window. A new session starts with no memory of the last [4] [20] (2.9).
Skeleton	The headers of the classes and functions in a file, without their bodies [6] (2.6).
Stop hook	A hook that runs when the agent tries to stop and can send it back to work [3] (2.5).
Subagent	An agent that works in its own context window and returns a summary [20] (2.5).
The 4-tuple	Request, environment, stopping criteria and scorer: the four parts of a task (Week 7b) (2.1).
Worktree	A separate checkout of a repository so that parallel sessions do not collide [3] (2.10).

9. References

Entries marked (required) or (additional reading) are the course's readings. Entries marked (added) are my own additions. The course site and the slide folder are listed without a mark. Each entry says how much of the source I read. A source by the tool's own authors, or by a company that sells it, is a claim by an interested party, and I say so where it matters.

• [1] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS 2024, as the course lists it. arXiv:2405.15793 (submitted 6 May 2024; version 3, 11 November 2024). John Yang, an instructor of the course, is the first author, and the Week 9a speakers are co-authors. https://arxiv.org/abs/2405.15793 (I read the abstract, the design, results and ablation sections and the tables, and skimmed the appendices; the authors' own numbers, for models of 2024) (required)
• [2] Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., Tran, H. H., Li, F., Ma, R., Zheng, M., Qian, B., Shao, Y., Muennighoff, N., Zhang, Y., Hui, B., Lin, J., and others. "OpenHands: An Open Platform for AI Software Developers as Generalist Agents." ICLR 2025. arXiv:2407.16741 (submitted 23 July 2024; version 3, 18 April 2025). The course schedule gives the title without "as Generalist Agents". The authors include people at Carnegie Mellon University and All Hands AI, which maintains the platform. https://arxiv.org/abs/2407.16741 (I read the abstract, the architecture sections and the SWE-bench Lite results, and did not read the appendices closely; the authors' own claims) (required)
• [3] Anthropic. "Best practices for Claude Code" (Claude Code documentation). The course schedule lists "Claude Code: Best Practices for Agentic Coding" (2025); the 2025 article is not retrievable at its old address, which now shows this page. Vendor documentation, undated. https://code.claude.com/docs/en/best-practices (I read the sections on the two kinds of instruction, verification, the instruction file and unattended runs, and skimmed the rest, on 29 October 2026; the page is a later revision than the 2025 article, and I did not run the tool) (additional reading)
• [4] Young, J. "Effective harnesses for long-running agents." Anthropic engineering blog, 26 November 2025. The schedule gives the author as "Young"; the post is by Justin Young. Vendor-authored. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents (I read the whole post; it describes what the author's team tried and reports no numbers) (additional reading)
• [5] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. arXiv:2310.06770. The SWE-agent authors are co-authors. https://arxiv.org/abs/2310.06770 (reused from Week 9a, where I read the abstract, the main text and Appendix A; the authors' own claims) (added)
• [6] Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. "Agentless: Demystifying LLM-based Software Engineering Agents." arXiv:2407.01489 (submitted 1 July 2024; version 2, 29 October 2024). https://arxiv.org/abs/2407.01489 (I read the abstract and the sections that I cite, not every section; the authors' own claims and costs, which do not line up with other papers' tables) (added)
• [7] Zhang, Y., Ruan, H., Fan, Z., and Roychoudhury, A. "AutoCodeRover: Autonomous Program Improvement." ISSTA 2024. arXiv:2404.05427 (submitted 8 April 2024; version 3, 25 July 2024). https://arxiv.org/abs/2404.05427 (I read the abstract and the sections that I cite, not every section, and the results table; the authors' own claims) (added)
• [8] Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. "Executable Code Actions Elicit Better LLM Agents" (CodeAct). ICML 2024. arXiv:2402.01030 (submitted 1 February 2024; version 4, 7 June 2024). The first and last of several authors also appear on the OpenHands paper. https://arxiv.org/abs/2402.01030 (reused from Week 4a; I read the abstract and the introduction; the authors' own claims) (added)
• [9] Aider. "Repository map" (documentation page). https://aider.chat/docs/repomap.html (I read the page and not the code, so I know the idea but not the details of its ranking; the project's own description) (added)
• [10] AGENTS.md. "A simple, open format for guiding coding agents" (home page). https://agents.md/ (I read the home page, including its list of supporting tools and its claim of use by over 60k open-source projects, which I did not check; the project's own page) (added)
• [11] OpenAI. "Codex CLI" (repository README on GitHub). Vendor-authored. https://github.com/openai/codex (I read the front page and not the code, and did not run it; it says that Codex CLI is "a coding agent from OpenAI that runs locally on your computer") (added)
• [12] Becker, J., Rush, N., Barnes, E., and Rein, D. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." METR. arXiv:2507.09089 (submitted 12 July 2025; version 2, 25 July 2025). The study used Cursor Pro with Claude 3.5 and 3.7 Sonnet as the AI tools. https://arxiv.org/abs/2507.09089 (I read the abstract, the design and the discussion of bias, not every section; independent of the tool vendors; superseded as a statement about current tools by [13]) (added)
• [13] METR. "We are Changing our Developer Productivity Experiment Design." Blog post, 24 February 2026. https://metr.org/blog/2026-02-24-uplift-update/ (I read the post; METR describes its own second study and says that its data is an unreliable signal because of selection effects) (added)
• [14] Miserendino, S., Wang, M., Patwardhan, T., and Heidecke, J. "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?" arXiv:2502.12115 (submitted 17 February 2025; version 4, 29 May 2025). The authors are at OpenAI, which builds the kind of model that it tests. https://arxiv.org/abs/2502.12115 (I read the abstract and the main results; the authors' own claims, for models of early 2025) (added)
• [15] Pan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., and Zhang, Y. "Training Software Engineering Agents and Verifiers with SWE-Gym." ICML 2025. arXiv:2412.21139 (submitted 30 December 2024; version 2, 6 June 2025). https://arxiv.org/abs/2412.21139 (I read the abstract and the data sections, not the verifier experiments; the authors' own claims) (added)
• [16] Merrill, M. A., Shaw, A. G., Carlini, N., Li, B., Raj, H., Bercovich, I., Shi, L., Shin, J. Y., Walshe, T., Buchanan, E. K., and others. "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces." arXiv:2601.11868 (submitted 17 January 2026). https://arxiv.org/abs/2601.11868 (reused from Week 6a; I read the abstract and the task construction; the authors' own claims) (added)
• [17] mini-SWE-agent (SWE-agent project), repository README on GitHub. https://github.com/SWE-agent/mini-swe-agent (I read the front page, not the code, and did not run it; the project's own claims, including a score above 74% on Verified that I did not check) (added)
• [18] Cihan, U., Haratian, V., Icoz, A., Gul, M. K., Devran, O., Bayendur, E. F., Ucar, B. M., and Tuzun, E. "Automated Code Review In Practice." ICSE-SEIP 2025. arXiv:2412.18531 (submitted 24 December 2024; version 2, 28 December 2024). https://arxiv.org/abs/2412.18531 (I read the abstract only; the authors' own claims about one industrial setting) (added)
• [19] Watanabe, M., Li, H., Kashiwa, Y., Reid, B., Iida, H., and Hassan, A. E. "On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub." arXiv:2509.14745 (submitted 18 September 2025; version 3, 9 February 2026). https://arxiv.org/abs/2509.14745 (I read the abstract only; the authors' own claims about 567 pull requests) (added)
• [20] Anthropic. "How Claude Code works" (Claude Code documentation). Vendor documentation, undated. https://code.claude.com/docs/en/how-claude-code-works (I read the whole page on 29 October 2026; I did not run the tool) (added)
• [21] Stanford CS329Z: Engineering AI Agents, course site and schedule (Diyi Yang, Michael Ryan, John Yang). https://cs329z.stanford.edu/ (the course's own page, read on 29 October 2026; it lists "Coding & Software Agents" on Monday, November 30, with two readings and two additional readings, "Proactive Agents" on Wednesday, December 2, and a finals week with a demo day from December 7 to 11)
• [22] CS329Z slide folder and the addresses lecture18.pdf to lecture21.pdf, each tried once on 29 October 2026 at about 9:00 PM Pacific time. https://web.stanford.edu/class/cs329z/slides/lecture18.pdf (the four addresses returned "not found" (HTTP 404), and the folder index listed lecture01.pdf to lecture11.pdf only, so I did not see the slides)