TwinEdgeAI

↑ Course home

← Previous: Week 8b, Agent Safety & Guardrails ·  Next: Week 10, Coding & Software Agents →

Engineering AI Agents · Week 9a

SWE-bench (Guest Lecture)

How a real GitHub issue becomes a graded task, what "resolved" means, how the harness runs it, how the benchmark grew into a family (Lite, Verified, Multimodal, Live, Pro), where it breaks (flawed tasks, memorization, saturation), and a lab that builds a small version of the whole pipeline from scratch.

Author: Grok Bot Version: 1.0, October 29, 2026 Study time: about 17 hours

The slides were not availableThis guest lecture is on Monday, November 16, and I wrote this pack on Thursday, October 29, 2026. I checked the course's slide folder once, at 8:36 PM Pacific time. Its index listed lecture01.pdf to lecture11.pdf, and the two addresses I tried for the weeks after that, lecture16.pdf and lecture17.pdf, each returned "not found" (HTTP 404) [3]. So I have not seen the slides. This pack is built from the schedule's one-sentence description of the lecture, the one required reading, and the public sources that I list. I do not know what the guest speakers will say. Nothing in this pack is a quotation from them or a summary of their talk, and where I describe what a source says, the source is named.

A date and order to checkThe schedule that I read on October 29, 2026 lists this lecture on Monday, November 16, as the first lecture of Week 9. The Safety lecture of the previous pack (Week 8b) is on Wednesday, November 18, two days later [2]. I numbered the files in the order that I wrote them, and the "Previous" link above means the previous page of this pack, not the previous date. If you read in calendar order, read this page before Week 8b.

CreditThis coursework is adapted from, and inspired by, Stanford CS329Z: Engineering AI Agents (Diyi Yang, Michael Ryan, John Yang), Week 9, "Guest Lecture: SWE-bench" (Monday, November 16). The schedule lists the speakers as Carlos Jimenez and Alexander Wettig and describes them as "(Anthropic), creators of SWE-bench, on the evolution and future of evaluations for coding capabilities and beyond". It lists one reading: Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (ICLR 2024) [2]. The topic and the reading come from the course's public schedule. Everything else here is my own work: the notes, the choice of added sources, the lab, the quiz and the assignment. This is not an official Stanford document.

1. Week overview and learning objectives

The last three weeks asked how to grade an agent (Week 8a) and how to keep one safe (Week 8b). This lecture looks at one benchmark that became the standard test for coding agents, from the people who made it. The schedule's description says that the speakers will talk about "the evolution and future of evaluations for coding capabilities and beyond" [2]. I read that as a story with four parts, and the notes follow it. The first is the design: why a real issue plus a set of tests makes a good task, what "resolved" means, and how the harness runs it (sections 2.1 to 2.5). The second is the family: the subsets and variants that followed, and the agents and training data built around them (2.6, 2.8 and 2.9). The third is the cracks: tasks that are unfair, models that may have seen the answer, and scores that rise until the test stops telling models apart (2.7 and 2.10). The fourth is how to report a number honestly (2.11 and 2.12). I cannot say what the speakers will emphasize, and the schedule's phrase "and beyond" tells me only that the talk goes past this list.

The lab uses the standard library only. It does not call any model, any API, Docker or the network. It builds a mini SWE-bench: 20 hand-written toy tasks in five tiny repositories, each with an issue text, a gold patch, a test patch, and lists of FAIL_TO_PASS and PASS_TO_PASS tests that the code works out by running the tests before and after the gold patch. It has a unified diff parser and applier, an in-process test runner, nine scripted "agents" that are programs and not language models, a resolved-rate scorer with confidence intervals, filters in the style of Verified and Lite, contamination probes, a leaderboard with cost, a result cache, and a small SWE-smith-style task generator. Every task, agent, price and number in the lab is made up by me. The lab is a model of how the pipeline works. It tells you nothing about how well any real model does.

Learning objectives

After this week you should be able to:

1. Say what a SWE-bench task is made of (issue text, repository at a base commit, gold patch, test patch, FAIL_TO_PASS and PASS_TO_PASS tests), and what the model sees and does not see. [2.2]
2. State the grading rule exactly, and explain why a patch that fixes the bug can still be marked wrong. [2.2]
3. Describe the three construction stages and name one reason that the final set is about 2.5% of the pull requests that were scraped. [2.3]
4. Explain why the first results were so low and what the oracle retrieval setting measured. [2.4]
5. Describe the layers of the Docker harness, the cache levels and the result cache pitfall, and say why a harness is part of the benchmark's definition. [2.5]
6. Tell Lite, Verified, Multimodal, Live and Pro apart by how each one chooses its tasks. [2.6]
7. Explain the two ways that a test can be unfair (too narrow, too wide), and why an underspecified issue is a third problem. [2.7]
8. Name the design ideas of SWE-agent's agent-computer interface, and say why the scaffold is part of a score. [2.8]
9. Describe how SWE-smith makes tasks without mining pull requests, and what it must check. [2.9]
10. Explain three probes for memorization, and why a gap between two benchmarks is not by itself proof. [2.10]
11. Put an interval on a resolved rate, report cost beside it, and say when two systems cannot be told apart. [2.11, 2.12]
12. Build, break and audit a small benchmark yourself. [lab]

How to use this pack

Time	What to do
3.5 hours	Read the notes (section 2). Skim the figures first.
3.5 hours	Read the required paper with the reading guide questions. Skim the added sources that match your interest, starting with the Verified post.
5 hours	Do the lab and at least five exercises.
1.5 hours	Discussion and the quiz.
3 hours	The graded assignment.

2. Lecture notes

The slides were not available (see the top of the page), so these notes follow the schedule's topic sentence [2], the required paper [1] and the public sources that I added. The section titles are mine. I do not know which of these points the speakers will cover.

2.1 Why issues, and why tests

Before SWE-bench, a common way to test code generation was to give a model a short function description and check a few examples. The SWE-bench paper argues that this is a poor picture of software work, and builds a test from the other end. Each task is a real problem that real people reported on GitHub, in a repository with thousands of files, where a real fix was written and merged [1]. The paper lists four properties that it claims for the design: the tasks are real software engineering work, they can be added continually ("a continual supply of new task instances"), the inputs are long (issues average 195 words, and codebases have many thousands of files), and the evaluation is robust, because each task has at least one test that fails before the fix and passes after, and 40% of tasks have at least two [1]. Those are the authors' claims about their own benchmark.

The key design choice is grading by tests, not by comparing text. A fix can be written in many ways, so the benchmark does not ask whether the model's patch looks like the real one. It asks whether the code behaves correctly when the repository's own tests are run. That makes the benchmark cheap to extend and fair to different styles of fix. It also moves the whole question of fairness into the tests: if a test checks something that the issue never asked for, a correct patch fails, and if the tests are too weak, a wrong patch passes. Sections 2.7 and 2.10 are about those two risks, and the lab lets you measure them.

2.2 The task and the grading rule

The model is given an issue text and a complete codebase, and must edit the codebase to resolve the issue. The edit is returned as a patch file, which lists the lines to change [1]. To score a proposed solution, the harness applies the patch with Unix's patch program, then runs the unit and system tests that belong to the task. If the patch applies and all of those tests pass, the issue counts as resolved. The metric is the percentage of task instances that are resolved [1].

Appendix A.4 spells the rule out. The two lists of tests are called FAIL_TO_PASS (tests that fail before the real fix and pass after it) and PASS_TO_PASS (tests that pass both before and after, which guard against a fix that breaks something else). A task is solved if all tests in both lists pass. A test that is missing from the results, or has any status other than pass, counts as a fail [1]. If the patch does not apply, the harness tries once to repair it, by removing unnecessary context lines and recalculating the header values. If the repaired patch still fails, or the test command fails, the prediction gets a score of 0 [1].

Three details are worth keeping in mind. First, the model's input is the issue and the repository, nothing else. It does not see the tests or the real patch. The issue text that the benchmark uses is an aggregate of the related issues' titles and descriptions and the comments written before the pull request's first commit, which the authors did to avoid leaking the solution [1]. Second, the rule is all or nothing. A patch that makes the new test pass but breaks one old test is scored as unresolved, the same as a patch that does nothing. Third, how well a patch is formatted matters. Because a patch that does not apply scores zero, a model that writes a correct change with a wrong hunk header loses the task, which is why the harness has a repair step, and why the lab has a status for it (a patch with wrong hunk counts is "invalid_patch" until the counts are repaired).

2.3 How the tasks were built

The paper builds the benchmark in three stages. Stage I collects pull requests from 12 popular open-source Python repositories, about 90,000 in total. The authors chose popular repositories because they tend to be better maintained, to have clear contributor guidelines and to have better test coverage. Stage II keeps the merged pull requests that resolve a GitHub issue and change test files, which suggests that the author added tests for the problem. Stage III runs the pull request's tests before and after the rest of its changes, and keeps only the candidates that have at least one fail-to-pass test and that install and run without error [1]. The result is the 2,294 task instances of the original set. As my own arithmetic, that is 2,294 / 90,000, or about 2.5% of the pull requests that were scraped. A candidate drops out at each stage by the rules above: no linked issue, no test change, no failing-then-passing test, or no environment that installs.

The appendix adds two useful details. The code change of each pull request is split into a test patch (changes to test files) and a gold patch (the rest), and the tests for the task are all the tests from the files that the test patch edits [1]. And the authors exclude task instances whose tests call a function or class that the gold patch itself introduces. Their reason is that naming such things is "typically an arbitrary process and usually not explicitly specified in the problem statement", so a task like that "may be an impossible task even for human developers" [1]. Keep that rule in mind. It is the same worry that the later audit of Verified found still present in some tasks (2.7), and the lab's filter checks for it directly.

My own note on what this design does and does not cover. The selection favors well-tested, popular libraries, and a task exists only if someone wrote a test with the fix. That is a narrow slice of software work. The Multimodal paper makes the related point that the original benchmark uses only Python repositories and mostly text [5]. The benchmark measures resolving issues of this kind, which is a useful thing, and it is not the same as "software engineering ability" in general.

2.4 What the first results said

The original paper reported low numbers. With a BM25 retriever that picked which files to show the model, the best model, Claude 2, resolved 1.96% of the issues, and the authors wrote that models "can resolve only the simplest issues" [1]. The paper also ran an oracle retrieval setting, in which the model is shown the files that the real patch edited. That is not a fair task, since it tells the model where to look, and the authors use it for analysis. With it, Claude 2 resolved 4.8% [1]. The retriever mattered: for the 27,000-token context limit, BM25 retrieved a superset of the oracle files in about 40% of instances, and in almost half of the instances it retrieved none of them [1]. The authors also fine-tuned CodeLlama models on pull requests from repositories that are disjoint from the test set, which they did "to eliminate the risk of data contamination", and called them SWE-Llama [1].

The picture changed fast. SWE-agent, which let a model run commands in a loop instead of reading files that a retriever chose, reported 12.47% of the full test set with GPT-4 Turbo, against a previous best of 3.8% for a non-interactive system [4]. OpenAI's Verified post reports that its best scaffold with GPT-4o resolved 33.2% on Verified, against 16% on the original set [7]. By February 2026, the same company reports that progress on Verified had gone from 74.9% to 80.9% in the last six months [8]. Do not draw one curve through these numbers. They come from different subsets, different scaffolds, different models and different groups, and the harness and the tasks were repaired in between. They show the speed of change, not a clean trend.

2.5 The harness and Docker evaluation design

Running a repository's tests needs the right Python version, the right dependencies and the right commit. If those drift, a task that passed yesterday fails today for reasons that have nothing to do with the model. The project's answer is a Docker harness. The documentation says that it uses containers "to create reproducible environments for evaluating model-generated patches", and that it has three layers [13].

The layers make the work reusable. Base images hold the language and tooling, environment images hold the dependencies of one repository and version, and instance images add the problem-specific setup, so that a build does not repeat shared work [13]. The --cache_level setting chooses what stays on disk between runs: none and base need about 120 GB during a run, the default env level about 100 GB, and instance about 2,000 GB, for the fastest reruns [13]. The project's news list records that the benchmark was "Docker-ized" in June 2024 "for easier evaluation" [14]. The documentation also describes a way to check the setup, which is to run the harness on the gold patches: a sensible check, since I would read a gold patch that fails as a sign of a broken environment (that reading is mine) [13].

Why does this belong in a lecture about a benchmark? Because the harness is part of the benchmark's definition. Two groups that run the same patches through different environments can get different scores. OpenAI's Verified post lists environment problems as the third kind of flaw that its annotators found, next to the issue text and the tests, and says that it developed a containerized harness with the SWE-bench authors to make evaluation more reliable [7]. The later SWE-bench-Live paper gives every task its own Docker image and automates the environment setup with an agent-driven pipeline that it calls RepoLaunch [11]. SWE-smith takes the opposite storage trade: tasks from one repository share one environment, which the paper says reduces storage compared with an image per task instance [6]. A lab built without Docker, like mine, gives up exactly this realism. My runner executes code in-process, with a step limit, and is not a sandbox: it must never be used with code that you do not trust.

2.6 Lite, Verified, Multimodal and the rest of the family

The benchmark did not stay one benchmark. Over three years the project and other groups released subsets and relatives, and each one was a response to a specific problem with the one before. Figure 4 lays the dates out, and the table that follows says how each member picks its tasks.

Member	Size	How it picks its tasks, and what it was for
SWE-bench (original)	2,294	Merged pull requests with tests, from 12 Python repositories, kept if a test goes from fail to pass and the environment installs [1]. The project's leaderboards call this the "Full" set [14].
Lite	300	A subset "for less costly evaluation". Instances were removed if they had images, external links, commit hashes or references to other issues or pull requests; had fewer than 40 words in the problem statement; edited more than one file; had a gold patch with more than three hunks; created or removed files; or had tests that check error messages. Then 300 test instances and 23 development instances were sampled from what remained. It covers 11 of the 12 repositories. The page recommends that systems report Lite "in lieu of" the full set "when compute efficiency is a concern" [15].
Verified	500	A subset that human annotators checked for underspecified issues and unfair tests, made with OpenAI, August 2024 (details in 2.7) [7]. The project site lists it as "a human-filtered subset" and uses it as the default leaderboard view [14].
Multimodal	617 (abstract)	Visual, user-facing JavaScript software: issues with at least one image in the problem statement or the tests, from 17 JavaScript libraries (web interface design, diagramming, data visualization, syntax highlighting, interactive mapping). The paper reports that SWE-agent resolves 12% against 6% for the next best system [5]. See the note on the count below.
Multilingual	300	Tasks from 42 repositories across 9 programming languages. I know it only from the project site's one-line description [14].
SWE-bench-Live	1,319 at first release	Issues created since 2024, from 93 repositories, each with its own Docker image, built by an automated pipeline so that new tasks can be added continually. The authors are not the SWE-bench authors. The authors report "a substantial performance gap compared to static benchmarks like SWE-bench" [11].
SWE-Bench Pro	1,865	41 repositories, long tasks that "may require hours to days" for an engineer, human-verified and given extra context. Split into a public set (11 repositories, 731 tasks), a held-out set (12 repositories, 858 tasks) and a commercial set of 276 tasks from 18 proprietary startup repositories, which are private [12]. The three counts add up to 1,865 (my arithmetic).

Notice what each change is a reaction to. Lite is a reaction to cost: it shrinks the set and keeps the simplest shape of task. Verified is a reaction to unfair tasks. Multimodal is a reaction to narrow coverage (one language, text only). Live and Pro are reactions to staleness and contamination: Live by using new issues and Pro by using licenses and private sets, as 2.10 explains. OpenAI's post makes the same contrast between two of them: Lite subsampled the original set to make it easier, whereas Verified tries to remove tasks that cannot be solved [7]. Each reaction fixes one thing and leaves the others. My lab shows one case of this directly: the Lite-style shape filter keeps every flawed task that I planted, because the shape of a task says nothing about whether its tests are fair (lab demo section 4).

2.7 Flawed tasks: what Verified fixed and what it left

Three problems make a task unfair even when the harness works. The issue is underspecified, so a careful engineer could not tell what to build. The tests are too narrow, so they reject a valid fix because it differs from the original (for example, the tests import a helper function that the original patch added and the issue never names). Or the tests are too wide, so they check behavior that the issue never asked for (for example, the original pull request fixed three issues and the issue text describes one). OpenAI's Verified post names the first two, calls the first "underspecified" and the second "unfair unit tests", and lists a third, trouble with the environment [7].

The Verified process was a human one. In August 2024, OpenAI reports, 93 Python developers annotated 1,699 random samples of the benchmark. Each sample was labeled three times on a scale of 0 to 3 for each problem, the final label was the highest of the three, and a label of 2 or 3 counts as severe. Of the samples, 38.3% were flagged as underspecified and 61.1% as having unit tests that might reject valid fixes, and 68.3% were filtered out overall. The 500 samples of Verified were chosen from the ones that remained: as many of the harder ones (estimated at one hour or more) as possible, and the rest at random [7]. As my own arithmetic, 1,699 x (1 - 0.683) is about 539, so 500 is a part of what was left. The post calls its own filter "likely to be overzealous" on purpose, to be confident in what stays [7].

Verified had a second life as a target. In February 2026, OpenAI published "Why we no longer evaluate SWE-bench Verified" and gave two reasons [8]. The first is that the benchmark still has bad tasks, even after the filter. The authors audited 138 problems that a model, o3, did not consistently solve over 64 runs. Each was reviewed by at least six engineers, and 59.4% were found to have material issues: 35.5% of the 138 had tests that are too narrow and 18.8% had tests that are too wide, with 5.1% in other categories. The post's examples are a pylint task whose tests import a function that the issue never names, and a sympy task in which the pull request fixed three issues and the description covers one [8]. Two cautions. The 138 are not a random sample: they are the problems that a particular model could not solve, so the share cannot be read as the share of all of Verified. And the audit was done by a co-creator of the benchmark, which also uses it. The second reason is contamination (2.10).

The same company applied the same kind of audit to Pro. In July 2026, OpenAI reports that the frontier pass rate on the public Pro split (731 tasks) had gone from 23.3% to 80.3% in eight months, that an automatic pipeline flagged 200 tasks (27.4%) as broken and a human campaign with five engineers per task flagged 249 (34.1%), and that it estimates about 30% of Pro tasks are broken, because of overly strict tests, underspecified prompts, low-coverage tests or misleading prompts. It states that it retracts its earlier recommendation to adopt Pro [9]. Read this carefully. It is a vendor audit of a benchmark it had recommended in its February post, and I read the post but did not re-check the tasks. What I take from it is the pattern: a benchmark is recommended, the scores climb, someone audits the tasks, and a large share turn out to be flawed. The pattern, not the exact percentage, is the lesson, and the lab builds the audit step in miniature (demo section 4).

2.8 Agents on SWE-bench: SWE-agent and the interface

SWE-agent's claim is that the interface between a model and a computer matters as much as the model. The authors argue that language-model agents are a new kind of user with their own needs, and they design an agent-computer interface (ACI) for them. Their four design principles are that actions should be simple and easy to understand, compact and efficient, that environment feedback should be informative but concise, and that guardrails help an agent recover from mistakes [4]. The system turns these into specific parts:

• Search commands (find_file, search_file, search_dir) that return a summary, and at most 50 results; if a search returns more, it asks the agent to narrow the query instead of listing them all [4].
• A file viewer that shows a window of at most 100 lines at a time [4].
• An edit command with a built-in code linter, so that a syntax mistake is shown to the agent immediately, with a snippet of the file from before and after the change [4].
• A cost budget: the authors set a limit of $4 per instance, and if a run went over, its existing edits were submitted automatically [4].

The paper's ablations support the claim in a way that you can check in the lab. A version without the linter did worse than one with it, and a version that made the agent search iteratively through many results did worse than a version with no extra search tools at all, because it wasted budget [4]. With GPT-4 Turbo, SWE-agent resolved 12.47% (286 of 2,294) of the full test set and 18.00% (54 of 300) of Lite [4]. Those are the numbers of 2024. The lab's scripted agents imitate the shape of these findings: one agent has "sloppy" formatting that breaks its patches, one has guardrails that repair them, and the results show how much a format rule alone can change a score (demo section 3).

A later project went the other way, toward less scaffold. mini-SWE-agent is described in its README as about 100 lines of Python for the agent class, with no tools other than bash, a completely linear history, and each action executed with subprocess.run as an independent command [16]. The README's header says that it scores above 74% on SWE-bench Verified, and the project's news list for July 2025 says 65% [14] [16]. I did not check which model or date goes with each figure, so do not compare them. The project's "Bash Only" leaderboard runs every model in the same mini-SWE-agent environment, and the site says that this is the default Verified view [14]. The point is an old one in evaluation: if the scaffold changes the score, a number that names only a model is incomplete, and a leaderboard that fixes the scaffold answers a different question from one that lets each team bring its own.

2.9 Making training tasks: SWE-smith

Mining pull requests is slow. The SWE-smith paper says that existing training sets are small, "at most 1,000s of training instances from 11 or fewer GitHub repositories", that curation needs hundreds of hours of human work, and that companion environments take several terabytes of storage [6]. Its answer is to invent the bugs. Given any Python codebase, the pipeline builds an execution environment, then automatically synthesizes hundreds to thousands of task instances that break existing tests in the codebase. The authors report 50,000 instances from 128 repositories [6].

The steps, from the paper's body, are these. First, set up a repository: the authors verify the installation and testing instructions by hand, check that more than 80% of the existing tests pass, and build one Docker image for the repository, which all of its tasks share [6]. Second, break it, in four ways: ask a language model to introduce an error into a function or to rewrite a function from its header and docstring, apply random transformations to the function's abstract syntax tree (remove a conditional or loop, change an operator, and others), undo a real pull request ("PR Mirror"), or combine bugs [6]. Third, keep a candidate only if it breaks at least one existing test, which is the same fail-then-pass check that Stage III of the original benchmark uses, turned around. Fourth, have a language model write an issue text for the bug, which the authors report approximates real issue text effectively [6]. Fifth, use the tasks for training: they ran SWE-agent with Claude 3.7 Sonnet as the expert model for at most 75 steps and a $2.00 cost limit, kept the trajectories that resolved their instances (5,016 of them), and fine-tuned Qwen 2.5 Coder Instruct (32B) on them. The result, SWE-agent-LM-32B, reaches 40.2% Pass@1 on Verified, which the authors call state of the art among open-source models at the time [6]. The authors give the cost of building the dataset as $1,360, of which $1,000 went to generating bugs, $160 to installing repositories with SWE-agent and $200 to generating issue text for 10,000 bugs [6]. Note that the issue text was not generated for all 50,000.

2.10 Contamination and benchmark decay

A model has seen a benchmark task if the task, or its answer, was in the model's training data. Real GitHub issues and their fixes are public, so a model trained on a large crawl of GitHub may have read the pull request that is its test answer. The original paper's answer was to note that its collection process could keep supplying tasks from after a model's training date [1]. The OpenAI Verified post lists pretraining contamination as a likely limitation of any static dataset [7]. The more interesting question is how you would tell. The sources use three kinds of evidence, and none is conclusive alone.

• Ask the model something it could only know if it remembered. The "SWE-Bench Illusion" paper gives models only an issue description, with no repository, and asks which file contains the bug. The authors report that models reach up to 76% accuracy on Verified instances, and up to 53% on tasks from repositories that are not in SWE-bench [10]. In a second probe the model gets the current file and the issue and must reproduce the real fix; the verbatim match (consecutive 5-grams) is up to 35% on Verified and Full, against up to 18% on tasks from other benchmarks [10]. The authors conclude that gains on Verified "may be partially driven by memorization rather than genuine problem-solving" [10]. Note the hedges in the abstract: "up to", "may be", "possible". In the body the authors say that some of the drop is "partly compounded by confounding variables", such as much shorter issue descriptions in one outside benchmark, and they repeat the file path test on issues that name no file or import, where the pattern persists [10]. Even so, nobody outside the labs can see a model's training data, so the evidence is indirect.
• Ask the model to leak. OpenAI's red-team setup had GPT-5 probe three other models (GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview) over 15 turns, and the post reports that all three produced task-specific details on tasks that included the examples it names. It also says that no model could produce a complete verbatim gold patch on the public Pro split at that time [8]. This is a vendor's own test, and I read it from the post and did not repeat it.
• Compare old tasks with new ones. SWE-bench-Live uses only issues created since 2024, and reports a substantial gap with static benchmarks [11]. In one comparison that the paper presents, OpenHands with Claude 3.7 Sonnet scored 43.2% on a re-run of Verified and 19.25% on Live, which is about 2.2 times higher on the older benchmark (my arithmetic). The Live authors say that this "raises concerns about potential overfitting to SWE-bench", and also report that within Live the resolve rate shows no clear link with how recent a task is, so newer issues are not inherently harder for current agents [11]. Neither paper can see the training data, and the two benchmarks also differ in repositories and construction, so I treat the gap as a clue and not a verdict (that caution is mine).

Pro and Live also try to prevent the problem. Pro's public set uses only repositories with strong copyleft (GPL) licenses, on the reasoning that such licenses make it unlikely that the code is in commercial training sets, and it keeps a held-out set and a commercial set of proprietary repositories private, to test for overfitting later [12]. That reasoning is the authors' argument and I know of no test of it in the sources I read. And the February 2026 audit found that a benchmark can be clean of leaks and still full of flawed tasks (2.7). Contamination and task quality are two separate problems, and fixing one does not fix the other.

My lab builds the three kinds of probe on a toy, and it is important to say what it does and does not show. I built a memorizer to recall: it keeps a fingerprint of the buggy code of each task that it has "seen" and returns the stored fix. So its high score on seen tasks is true by construction. What the lab can show is what the diagnostics look like when memorization is present: the memorizer is perfect on seen tasks (27 of 27) and falls to 42% on the 33 unseen tasks, and an honest scripted agent is at 56% and 42% (demo section 6). Rewording the issue text does not stop the recall, because the recall is keyed on the code, while renaming the functions does (15 of 27). And a duplicate check found 3 unseen tasks whose issue text is identical to a seen one, a leak path that a date cutoff does not catch. All of these numbers are made up and the intervals are wide. A real model is not built like this, and a real probe has to deal with a model that half remembers.

2.11 Reporting a result: intervals, cost and scaffold

A resolved rate looks like a precise number and is not. Each task is one yes or no, so a rate is an average of a few hundred coin flips with different odds. There are three habits that make a report honest.

1. Put an interval on the rate, and say how it was computed. The lab uses the Wilson interval for a rate and a bootstrap over tasks for a difference. With 20 tasks, 14 resolved is 70% with a 95% interval of about 48% to 85%, which is too wide to rank two neighbors (demo section 3). With 500 tasks the standard error of a 70% rate is about 2 points (my arithmetic, 2.12), so two systems 2 points apart are not clearly different. Because systems are scored on the same tasks, a paired comparison is sharper than comparing two separate intervals.
2. Report cost beside the rate. The project's leaderboard site has views of resolved rate against cost, against cost limit and against step limit, and a flag for "Run performed or directly checked by the SWE-bench team" [14]. SWE-agent reported a per-instance budget [4]. In the lab, two agents tie at 14 of 20 and one costs 1.6 times as much (made-up prices), which a rate-only leaderboard hides (Figure 9).
3. Name everything that is not the model. Which subset, which harness version, which scaffold, how many attempts, who ran it, and what the step and cost limits were. The Verified post's 33.2% against 16% is a statement about a scaffold as much as a model [7], and the Bash Only leaderboard exists to hold the scaffold fixed [14].

A last point on filters and leaderboards. In the lab, removing the seven flawed tasks (Verified-style) leaves 13 tasks, on which the two best agents solve all 13 and the rest move down together. A leaderboard with a ceiling like that stops separating the top. That is the same shape as OpenAI's account of Verified scores climbing from 74.9% to 80.9% while a share of the remaining tasks were flawed [8]: once most of the clean tasks are solved, the leftover failures are mostly the broken ones, and the benchmark stops telling you about the model.

2.12 Worked arithmetic

Everything in this table is my own arithmetic from numbers in the cited sources or from the lab, and the lab numbers are made up.

Question	Working
What share of the scraped pull requests became tasks? [1]	2,294 / 90,000 = 0.0255, about 2.5%.
How big is Lite compared with the full set? [15]	300 / 2,294 = 0.131, about 13%.
How many Verified-annotated samples remained after filtering? [7]	1,699 x (1 - 0.683) = 1,699 x 0.317 = 538.6, about 539. The 500 of Verified are a part of that.
How many of the 138 audited problems were narrow, wide and other? [8]	138 x 0.594 = 82 with a material issue. 138 x 0.355 = 49 narrow, 138 x 0.188 = 26 wide, 138 x 0.051 = 7 other. 49 + 26 + 7 = 82.
Do Pro's three parts add up? [12]	731 + 858 + 276 = 1,865.
How far apart are the two Illusion probes' groups? [10]	File path: 76 - 53 = 23 points. Five-gram match: 35 - 18 = 17 points (rounded from 34.9 and 18.2).
Verified against Live, one setup [11]	43.2 / 19.25 = 2.24, about 2.2 times; 43.2 - 19.25 = 23.95 points.
The standard error of a resolved rate, 500 tasks, rate 0.70	sqrt(0.7 x 0.3 / 500) = sqrt(0.00042) = 0.0205, about 2 points. A 95% interval is about plus or minus 4 points (1.96 x 0.0205 = 0.040), as a rough normal approximation.
The same, 20 tasks, rate 0.70 (lab)	sqrt(0.7 x 0.3 / 20) = 0.102, so plus or minus 0.20 by the rough rule. The lab's Wilson interval is 48% to 85%, which is not symmetric and is more reliable at small n.
Cost per solved task, lab, full-repertoire (made-up price)	$0.2643 per task x 20 tasks / 14 solved = $0.378 per solved task (the demo prints 0.378).
The same, repro-std	$0.0417 x 20 / 10 = $0.083.
SWE-agent resolve rate [4]	286 / 2,294 = 0.1247; 54 / 300 = 0.18.
Pro public-split change in eight months [9]	80.3 - 23.3 = 57 percentage points.

2.13 Week 9a in one page

1. Task: issue text plus repository in, patch out. Grade: apply the test patch and the model patch, run FAIL_TO_PASS and PASS_TO_PASS, resolved only if all pass [1].
2. Construction: about 90,000 pull requests from 12 Python repositories, three filters, 2,294 tasks [1].
3. Harness: Docker layers (base, environment, instance), cache levels, and a result cache that ignores the patch. The harness is part of the definition [13].
4. Family: Lite for cost, Verified for fairness, Multimodal for coverage, Live and Pro for freshness and contamination [7] [5] [11] [12] [15].
5. Task quality: underspecified issues, tests too narrow, tests too wide. A shape filter does not catch them; human review or an alternative-fix check can [7] [8].
6. Agents: the interface and scaffold change the score. SWE-agent's linter, window and capped search; mini-SWE-agent's bash only [4] [16].
7. Training data: SWE-smith breaks working code, keeps the candidates that break a test, and writes the issue [6].
8. Contamination: probes with file paths and verbatim fixes, red-team leaks, and old against new tasks. A gap is a clue, not a verdict [10] [11] [8].
9. Reporting: intervals, paired comparisons, cost, scaffold, harness version, who ran it.
10. Humility: I have not seen the slides, and the vendor audits are the vendor's own.

3. Reading guide

The course lists one reading for this lecture [2]. For it you get: why it matters, the key ideas, and questions to think about as you read. Read the questions first. The table at the end points to the sources that I added for the notes. For those I read the abstract and the sections that I cite, and not the whole paper, and the references say which. Several authors are close to the subject: the lecture's two speakers are authors of the required paper, and the course's instructors John Yang and Diyi Yang are authors of papers in my table [1] [4] [5] [6].

The sources that I added

Source	Ref	What I used it for, and what I read
OpenAI, "Introducing SWE-bench Verified" (2024)	[7]	The annotation study behind Verified. Read the whole post. Vendor-authored, with two SWE-bench authors in the author list.
OpenAI, "Why we no longer evaluate SWE-bench Verified" (2026)	[8]	The audit of 138 problems and the contamination red-team. Read the whole post. Vendor-authored.
OpenAI, "Separating signal from noise in coding evaluations" (2026)	[9]	The audit of SWE-Bench Pro and the retraction. Read the whole post. Vendor-authored.
SWE-agent (Yang et al.), NeurIPS 2024 as the course lists it	[4]	The agent-computer interface. Abstract and the sections on the interface, the results and the ablations. The course lists it as a reading for a later lecture, on coding agents.
SWE-bench Multimodal (Yang et al.)	[5]	The visual JavaScript benchmark. Abstract and the dataset description. Note the count discrepancy in 2.6.
SWE-smith (Yang et al.)	[6]	Synthesizing training tasks. Abstract and the method and cost sections.
The SWE-Bench Illusion (Liang et al.)	[10]	The file path and function reproduction probes. Abstract and the experiments I cite. The authors are not SWE-bench authors.
SWE-bench Goes Live! (Zhang et al.)	[11]	Fresh issues and automated environments. Abstract and the sections on construction and results.
SWE-Bench Pro (Deng et al.)	[12]	Licenses, held-out and commercial sets. Abstract and the dataset section.
The SWE-bench harness documentation	[13]	Layers, cache levels, the result cache. I read the page and did not run the harness.
The SWE-bench project site (swebench.com): news, benchmarks, leaderboards	[14]	The family, sizes, dates and the Bash Only leaderboard. Read the home page. The site is the project's own, and it lists OpenAI and Anthropic among its supporters.
The SWE-bench Lite page	[15]	Lite's selection criteria. Read the whole page.
mini-SWE-agent README	[16]	A bash-only agent. Read the top of the README; I did not run it.

4. Hands-on lab: a mini SWE-bench with no Docker

Time: about 5 hours. Goal: build and test a small toolkit that does what this week's notes describe, end to end: a unified diff parser and applier that can validate and repair a patch; five tiny in-memory repositories and 20 hand-written tasks, each with an issue text, a base commit, a gold patch, a test patch and lists of FAIL_TO_PASS and PASS_TO_PASS tests that the code works out by running the tests before and after the gold patch; an in-process test runner with a step limit; the grading rule; a result cache; nine scripted agents that localize with BM25 and try edits; a resolved-rate scorer with Wilson and bootstrap intervals; a leaderboard with cost; Verified-style and Lite-style task filters; contamination probes (file path, prefix completion, reword, rename, date split); and a SWE-smith-style task generator. It uses the standard library only, with no model, no API, no Docker and no network.

What you will learn

• How a unified diff is parsed and applied, why a wrong hunk count or a shifted context line makes a patch fail, and what a repair step can and cannot rescue.
• How FAIL_TO_PASS and PASS_TO_PASS lists are derived by running tests before and after a patch, and how to grade with the all-or-nothing rule.
• How a result cache that ignores the patch gives wrong answers.
• How a resolved rate gets an interval, how two agents are compared on the same tasks, and how cost and a cost cap change a leaderboard.
• How a static check (names the tests use but the issue never mentions) and a dynamic check (a valid alternative fix fails the tests) find flawed tasks, and why a Lite-style shape filter does not.
• How to probe for memorization with file paths, prefixes, rewording, renaming and a date split, and what each probe can and cannot show.
• How a SWE-smith-style generator makes tasks by breaking working code, and what it must validate.

Map of the code

Part	Main names in the code	What it does, and what it stands for
0	mean, rng_for, wilson, bootstrap_ci, tokens, jaccard, est_tokens	Shared helpers. rng_for makes a random generator from a string of labels, so that every agent sees the same draw for the same task (common random numbers). Wilson and bootstrap intervals.
1	parse_diff, apply_patch, make_diff, fix_hunk_counts, diff_stats, PatchError	The unified diff: parse, apply with a small offset search, build from two versions of the files, and repair wrong hunk counts. This is the patch apply and validate step [1].
2	REPOS, SPECS, REGRESS	Five toy packages (text, list, math, cart and config helpers) and 20 task specifications with a bug, an issue text and a planted flaw label.
3	Sandbox, run_tests, eval_snippet, StepLimit	The in-process test runner. Modules are loaded from memory, a trace function counts steps, and a runaway loop ends as a timeout. Not a sandbox.
4	Task, build_task, validate_task, curated_tasks	A task with its FAIL_TO_PASS and PASS_TO_PASS lists derived by running the tests, and Stage III style checks [1].
5	evaluate, EvalResult, EvalCache	The grading rule, with statuses (empty, invalid_patch, apply_fail, unresolved, resolved), and a result cache with a content key and a flawed key that ignores the patch [13].
6	ScriptedAgent, NullAgent, bm25_rank, code_units, roster	Nine scripted agents, from a null agent to one that is told the right function. BM25 localization [1], a fixed edit repertoire, and checks in the style of SWE-agent's loop [4].
7	run_benchmark, resolved_rate, leaderboard, paired_gap, cost_of	The scorer and leaderboard, with intervals, cost per task, cost per solved task and the cost and rate frontier.
8	annotate, issue_severity, unmentioned_symbols, alt_solution_fails, verified_filter, lite_filter, filter_scores	Task-quality filters: simulated annotators with a maximum rule, a static and a dynamic check for over-specific tests, and the Lite-style shape rules [7] [15].
9	file_path_probe, prefix_probe, reword_task, rename_task, date_split, MemorizerAgent, contamination_study	Contamination probes in the spirit of [10] and [8], on a pool of 60 tasks with a made-up cutoff, and a memorizer that recalls by construction.
10	synth_repo, synthesize_tasks	A SWE-smith style generator: mutate, keep only if an existing test breaks, validate, write the issue text [6].
11	demo	The seven-section demo.

Setup

1. You need Python 3.10 or later. There is nothing to install. I ran it on Python 3.13 only.
2. Save the code below as swe_bench_toolkit.py. Or download it: swe_bench_toolkit.py.
3. Save the tests as test_swe_bench_toolkit.py in the same folder. Or download: test_swe_bench_toolkit.py. The test code is not printed on this page, to keep the page shorter. It is in the download, and the names of all the tests are listed after the exercises.
4. Run python swe_bench_toolkit.py for the seven-section demo, and python -m unittest test_swe_bench_toolkit for the tests. Both are deterministic. The demo and the tests each take a few seconds on my machine.

The code: swe_bench_toolkit.py

"""swe_bench_toolkit.py: a toy lab for Week 9a, SWE-bench (guest lecture by C. Jimenez and A. Wettig).

Everything here is made up and runs offline with the standard library. There is no language
model and no Docker. The "repos" are five tiny Python packages that live in memory, the "issues"
are written by hand, and the "agents" are scripted programs that edit source text. A task is
resolved when its FAIL_TO_PASS and PASS_TO_PASS tests all pass after a patch is applied, which is
the rule that the SWE-bench sources describe. Every number the demo prints is about this toy
and about no real model or benchmark. The in-process test runner executes the code that it is
given, so it is NOT a sandbox: do not feed it patches that you did not write.

Parts: 0 helpers, 1 unified diff (parse, apply, make, fix counts), 2 toy repos and task specs,
3 in-process test runner, 4 task build and validation, 5 evaluation harness, 6 scripted agents,
7 scorer, cost and leaderboard, 8 task-quality filters, 9 contamination probes, 10 task
synthesis, 11 demo.
"""
import ast
import builtins
import copy
import difflib
import hashlib
import math
import random
import re
import sys
import types
from collections import Counter, defaultdict


# ---------------------------------------------------------------- Part 0: helpers
def mean(xs):
    xs = list(xs)
    return sum(xs) / len(xs) if xs else 0.0


def rng_for(*parts):
    """A random generator whose stream depends only on its name, never on call order."""
    return random.Random("|".join(str(p) for p in parts))


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    r = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, (c - r) / d), min(1.0, (c + r) / d))


def bootstrap_ci(values, n_boot=400, seed=0, alpha=0.05):
    """Percentile bootstrap interval for the mean of per-task values (resampling tasks)."""
    values = list(values)
    if not values:
        return (0.0, 1.0)
    rng = rng_for("boot", seed, len(values))
    ms = sorted(mean(rng.choice(values) for _ in values) for _ in range(n_boot))
    return (ms[int(alpha / 2 * n_boot)], ms[min(n_boot - 1, int((1 - alpha / 2) * n_boot))])


STOP = set("the a an of to in is it and or for on with when that this be as are was at by from but not "
           "can i if so do does should would returns return get gets there than then its has have".split())


def tokens(text):
    """Lower-case word tokens; identifiers are split on underscores and camel case."""
    out = []
    for w in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", text):
        parts = re.sub(r"([a-z])([A-Z])", r"\1_\2", w).lower().split("_")
        out.extend(p for p in parts if p)
        if "_" in w:
            out.append(w.lower())
    return out


def content_tokens(text):
    return [t for t in tokens(text) if t not in STOP and len(t) > 1]


def jaccard(a, b):
    a, b = set(a), set(b)
    return len(a & b) / len(a | b) if (a | b) else 1.0


def est_tokens(text):
    """A rough count of model tokens, four characters each. My own rule of thumb, not a tokenizer."""
    return max(1, len(text) // 4)


# ---------------------------------------------------------------- Part 1: unified diff
class PatchError(Exception):
    """A patch that cannot be parsed or applied. The text starts with the reason."""


class Hunk:
    def __init__(self, old_start, old_len, new_start, new_len, lines):
        self.old_start, self.old_len, self.new_start, self.new_len = old_start, old_len, new_start, new_len
        self.lines = lines  # list of (tag, text) with tag in " ", "-", "+"

    def old_lines(self):
        return [t for tag, t in self.lines if tag in " -"]

    def new_lines(self):
        return [t for tag, t in self.lines if tag in " +"]


class FilePatch:
    def __init__(self, old_path, new_path, hunks):
        self.old_path, self.new_path, self.hunks = old_path, new_path, hunks

    @property
    def path(self):
        return self.new_path if self.new_path != "/dev/null" else self.old_path

    @property
    def is_new(self):
        return self.old_path == "/dev/null"

    @property
    def is_delete(self):
        return self.new_path == "/dev/null"


HUNK_RE = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")


def _strip_prefix(p):
    p = p.split("\t")[0].strip()
    if p == "/dev/null":
        return p
    return p[2:] if p[:2] in ("a/", "b/") else p


def parse_diff(text, strict_counts=True):
    """Parse a unified diff into FilePatch objects. With strict_counts a hunk is read until its header
    counts are met and then must end (as the patch program requires); without it the hunk is whatever
    run of diff lines follows. Raises PatchError."""
    lines = text.split("\n")
    if lines and lines[-1] == "":
        lines.pop()
    out, i = [], 0
    while i < len(lines):
        if not lines[i].startswith("--- "):
            i += 1  # "diff --git", "index ..." and prose lines are skipped
            continue
        if i + 1 >= len(lines) or not lines[i + 1].startswith("+++ "):
            raise PatchError("malformed header: '---' line without '+++' line")
        old_p, new_p = _strip_prefix(lines[i][4:]), _strip_prefix(lines[i + 1][4:])
        i += 2
        hunks = []
        while i < len(lines) and lines[i].startswith("@@"):
            m = HUNK_RE.match(lines[i])
            if not m:
                raise PatchError("malformed hunk header: %r" % lines[i][:40])
            os_, ol, ns, nl = int(m.group(1)), m.group(2), int(m.group(3)), m.group(4)
            ol = 1 if ol is None else int(ol)
            nl = 1 if nl is None else int(nl)
            i += 1
            body, so, sn = [], 0, 0
            while i < len(lines):
                ln = lines[i]
                if strict_counts and so >= ol and sn >= nl:
                    break
                if ln.startswith("\\"):
                    i += 1
                    continue
                tag = ln[:1] if ln else " "
                if tag not in (" ", "-", "+") or (ln.startswith(("--- ", "+++ ")) and
                                                   i + 1 < len(lines) and lines[i + 1].startswith(("+++ ", "@@"))):
                    break
                body.append((tag, ln[1:]))
                so += tag in " -"
                sn += tag in " +"
                i += 1
            h = Hunk(os_, ol, ns, nl, body)
            if strict_counts and (so != ol or sn != nl):
                raise PatchError("hunk header counts %d,%d but body has %d,%d lines" % (ol, nl, so, sn))
            hunks.append(h)
            if strict_counts and i < len(lines) and lines[i][:1] in (" ", "-", "+") \
                    and not lines[i].startswith(("--- ", "+++ ")):
                raise PatchError("hunk body is longer than its header says")
        if not hunks:
            raise PatchError("file header without hunks: %s" % new_p)
        out.append(FilePatch(old_p, new_p, hunks))
    if not out:
        raise PatchError("no file patches found")
    return out


def _split(text):
    return text.split("\n") if text else []


def _join(lines):
    return "\n".join(lines) + ("\n" if lines else "")


def apply_patch(files, diff, strict_counts=True, max_offset=8):
    """Apply a unified diff to a dict of {path: text}. Returns (new_files, report). Context lines must
    match exactly; a hunk may sit up to max_offset lines away from where its header says (as the patch
    program allows). Nothing is changed if any hunk fails. Raises PatchError."""
    fps = parse_diff(diff, strict_counts)
    new = dict(files)
    report = {"files": [], "offsets": 0, "hunks": 0}
    for fp in fps:
        if fp.is_new:
            if fp.path in new:
                raise PatchError("file already exists: " + fp.path)
            new[fp.path] = _join([t for h in fp.hunks for t in h.new_lines()])
            report["files"].append(fp.path)
            report["hunks"] += len(fp.hunks)
            continue
        if fp.old_path not in new:
            raise PatchError("file not found: " + fp.old_path)
        src = _split(new[fp.old_path].rstrip("\n")) if new[fp.old_path] else []
        shift = 0
        for h in fp.hunks:
            old = h.old_lines()
            want = h.old_start - 1 + shift if old else h.old_start + shift
            placed = None
            for off in sorted(range(-max_offset, max_offset + 1), key=lambda d: (abs(d), d)):
                s = want + off
                if 0 <= s <= len(src) and src[s:s + len(old)] == old:
                    placed = s
                    if off:
                        report["offsets"] += 1
                    break
            if placed is None:
                raise PatchError("context does not match in %s near line %d" % (fp.old_path, h.old_start))
            src[placed:placed + len(old)] = h.new_lines()
            shift += len(h.new_lines()) - len(old)
            report["hunks"] += 1
        if fp.is_delete:
            del new[fp.old_path]
        else:
            new[fp.new_path] = _join(src)
        report["files"].append(fp.path)
    return new, report


def make_diff(old_files, new_files, context=3):
    """A unified diff that turns old_files into new_files (sorted by path)."""
    out = []
    for p in sorted(set(old_files) | set(new_files)):
        a, b = old_files.get(p), new_files.get(p)
        if a == b:
            continue
        al = a.split("\n") if a else []
        bl = b.split("\n") if b else []
        if a is not None and a.endswith("\n"):
            al = al[:-1]
        if b is not None and b.endswith("\n"):
            bl = bl[:-1]
        fa = "a/" + p if a is not None else "/dev/null"
        fb = "b/" + p if b is not None else "/dev/null"
        body = list(difflib.unified_diff(al, bl, fa, fb, lineterm="", n=context))
        out.extend(body)
    return "\n".join(out) + "\n" if out else ""


def fix_hunk_counts(diff):
    """Recompute the line counts in each hunk header from the hunk body (the kind of repair that a harness
    can make for a model that miscounts). Leaves the start lines alone."""
    lines = diff.rstrip("\n").split("\n")
    out, i = [], 0
    while i < len(lines):
        m = HUNK_RE.match(lines[i])
        if not m:
            out.append(lines[i])
            i += 1
            continue
        j = i + 1
        while j < len(lines) and lines[j][:1] in (" ", "-", "+", "\\") and not lines[j].startswith(("--- ", "+++ ")):
            j += 1
        body = lines[i + 1:j]
        ol = sum(1 for b in body if b[:1] in (" ", "-"))
        nl = sum(1 for b in body if b[:1] in (" ", "+"))
        out.append("@@ -%s,%d +%s,%d @@%s" % (m.group(1), ol, m.group(3), nl, lines[i][m.end():]))
        out.extend(body)
        i = j
    return "\n".join(out) + "\n"


def diff_stats(diff):
    """Files touched, lines added and removed, hunks."""
    try:
        fps = parse_diff(diff, strict_counts=False)
    except PatchError:
        return {"files": 0, "added": 0, "removed": 0, "hunks": 0}
    add = sum(1 for fp in fps for h in fp.hunks for t, _ in h.lines if t == "+")
    rem = sum(1 for fp in fps for h in fp.hunks for t, _ in h.lines if t == "-")
    return {"files": len(fps), "added": add, "removed": rem, "hunks": sum(len(fp.hunks) for fp in fps)}


# ---------------------------------------------------------------- Part 2: toy repos and task specs
_TEXT = '''def slugify(s):
    words = s.lower().split()
    return "-".join(words)


def truncate(s, n):
    if len(s) <= n:
        return s
    return s[: n - 3] + "..."


def startswith_any(s, prefixes):
    return any(s.startswith(p) for p in prefixes)


def is_palindrome(s):
    chars = [c.lower() for c in s if c.isalnum()]
    return chars == chars[::-1]
'''
_LISTS = '''def chunk(xs, n):
    return [xs[i : i + n] for i in range(0, len(xs), n)]


def flatten(xs):
    out = []
    for x in xs:
        if isinstance(x, list):
            out.extend(x)
        else:
            out.append(x)
    return out


def rotate(xs, k):
    if not xs:
        return xs
    k = k % len(xs)
    return xs[k:] + xs[:k]


def window_sums(xs, k):
    return [sum(xs[i : i + k]) for i in range(len(xs) - k + 1)]
'''
_STATS = '''def spread(xs):
    return max(xs) - min(xs)


def median(xs):
    s = sorted(xs)
    n = len(s)
    mid = n // 2
    if n % 2:
        return s[mid]
    return (s[mid - 1] + s[mid]) / 2


def mean(xs):
    if not xs:
        return 0.0
    return sum(xs) / len(xs)


def gcd(a, b):
    while b:
        a, b = b, a % b
    return abs(a)
'''
_CART = '''class Cart:
    def __init__(self, discount=0.0):
        self.items = {}
        self.prices = {}
        self.discount = discount

    def add(self, name, price, qty=1):
        self.prices[name] = price
        if name in self.items:
            self.items[name] += qty
        else:
            self.items[name] = qty

    def subtotal(self):
        return sum(self.prices[n] * q for n, q in self.items.items())

    def total(self):
        return round(self.subtotal() * (1 - self.discount), 2)

    def discount_amount(self):
        return self.subtotal() - self.total()

    def can_checkout(self):
        return bool(self.items) and self.total() > 0
'''
_CONFIG = '''def parse_kv(line):
    key, _, value = line.partition("=")
    return key.strip(), value.strip()


def get_path(d, path, default=None):
    cur = d
    for part in path.split("."):
        if not isinstance(cur, dict) or part not in cur:
            return default
        cur = cur[part]
    return cur


def merge(a, b):
    out = dict(a)
    for k, v in b.items():
        if isinstance(v, dict) and isinstance(out.get(k), dict):
            out[k] = merge(out[k], v)
        else:
            out[k] = v
    return out


def to_bool(s):
    return s.strip().lower() in ("1", "true", "yes")
'''
REPOS = {
    "textkit": {"module": "textkit/text.py", "test": "tests/test_text.py", "src": _TEXT,
                "header": "from textkit.text import slugify, truncate, startswith_any, is_palindrome\n"},
    "listkit": {"module": "listkit/lists.py", "test": "tests/test_lists.py", "src": _LISTS,
                "header": "from listkit.lists import chunk, flatten, rotate, window_sums\n"},
    "mathkit": {"module": "mathkit/stats.py", "test": "tests/test_stats.py", "src": _STATS,
                "header": "from mathkit.stats import spread, median, mean, gcd\n"},
    "cartkit": {"module": "cartkit/cart.py", "test": "tests/test_cart.py", "src": _CART,
                "header": "from cartkit.cart import Cart\n"},
    "configkit": {"module": "configkit/config.py", "test": "tests/test_config.py", "src": _CONFIG,
                  "header": "from configkit.config import parse_kv, get_path, merge, to_bool\n"},
}
# Existing tests: each passes on the buggy code too (they are the PASS_TO_PASS tests).
BASIC = {
    "slugify": 'assert slugify("hello world") == "hello-world"',
    "truncate": 'assert truncate("hi", 5) == "hi"\n    assert truncate("hello world", 8) == "hello..."',
    "startswith_any": 'assert startswith_any("abc", []) is False\n    assert startswith_any("abc", ["abc"]) is True',
    "is_palindrome": 'assert is_palindrome("abba") is True\n    assert is_palindrome("ab") is False',
    "chunk": "assert chunk([1, 2, 3, 4], 2) == [[1, 2], [3, 4]]\n    assert chunk([], 2) == []",
    "flatten": "assert flatten([1, 2]) == [1, 2]\n    assert flatten([]) == []",
    "rotate": "assert rotate([1, 2, 3], 1) == [2, 3, 1]",
    "window_sums": "assert window_sums([], 2) == []\n    assert window_sums([1, 2, 3], 4) == []",
    "spread": "assert spread([2]) == 0",
    "median": "assert median([1, 2, 3]) == 2",
    "mean": "assert mean([1, 2, 3]) == 2.0",
    "gcd": "assert gcd(12, 18) == 6\n    assert gcd(7, 0) == 7",
    "add": 'c = Cart()\n    c.add("a", 2.0)\n    assert c.items == {"a": 1}',
    "total": 'c = Cart()\n    c.add("a", 3.0, 2)\n    assert c.total() == 6.0',
    "can_checkout": 'assert Cart().can_checkout() is False\n    c = Cart()\n    c.add("a", 2.0)\n    assert c.can_checkout() is True',
    "discount_amount": 'c = Cart()\n    c.add("a", 2.0)\n    assert c.discount_amount() == 0.0',
    "parse_kv": 'assert parse_kv("a=1") == ("a", "1")',
    "get_path": 'assert get_path({"a": {"b": 1}}, "a.b") == 1',
    "merge": 'assert merge({"a": 1}, {"b": 2}) == {"a": 1, "b": 2}',
    "to_bool": 'assert to_bool("yes") is True\n    assert to_bool("no") is False',
}
# New tests added by the "pull request" (the FAIL_TO_PASS tests).
REGRESS = {
    "slugify": 'assert slugify("Hello World") == "hello-world"',
    "truncate": 'from textkit.text import ELLIPSIS\n    assert truncate("hello", 5) == "hello"\n    assert truncate("hello world", 8).endswith(ELLIPSIS)',
    "startswith_any": 'assert startswith_any("foobar", ["foo"]) is True',
    "is_palindrome": 'assert is_palindrome("a1b2a") is False',
    "chunk": "assert chunk([1, 2, 3], 2) == [[1, 2], [3]]",
    "flatten": "from listkit.lists import _is_seq\n    assert _is_seq([1]) and not _is_seq(1)\n    assert flatten([1, [2, 3]]) == [1, 2, 3]",
    "rotate": "assert rotate([], 3) == []",
    "window_sums": "assert window_sums([1, 2, 3, 4], 2) == [3, 5, 7]",
    "spread": "from mathkit.stats import _extent\n    assert _extent([1, 4]) == (1, 4)\n    assert spread([1, 4, 2]) == 3",
    "median": "assert median([3, 1, 2]) == 2",
    "mean": "assert mean([]) == 0.0",
    "gcd": "assert gcd(4, -6) == 2",
    "add": 'c = Cart()\n    c.add("a", 2.0)\n    c.add("a", 2.0, 2)\n    assert c.items["a"] == 3',
    "total": 'c = Cart(0.1)\n    c.add("a", 3.33, 3)\n    assert c.total() == 8.99',
    "can_checkout": 'c = Cart()\n    c.add("free", 0.0)\n    assert c.can_checkout() is False',
    "discount_amount": 'c = Cart(0.1)\n    c.add("a", 5.0)\n    assert c.discount_amount() == 0.5',
    "parse_kv": 'assert parse_kv("a=b=c") == ("a", "b=c")',
    "get_path": 'assert get_path({"a": 1}, "a.b", 5) == 5',
    "merge": 'assert merge({"a": {"x": 1}}, {"a": {"y": 2}}) == {"a": {"x": 1, "y": 2}}',
    "to_bool": 'assert to_bool(" yes ") is True',
}
_PARSE_KV = ('    key, _, value = line.partition("=")\n    return key.strip(), value.strip()',
             '    parts = line.split("=")\n    return parts[0].strip(), parts[1].strip()')
_GET_PATH = ('        if not isinstance(cur, dict) or part not in cur:\n            return default\n        cur = cur[part]',
             '        cur = cur[part]')
_MERGE = ('        if isinstance(v, dict) and isinstance(out.get(k), dict):\n            out[k] = merge(out[k], v)\n        else:\n            out[k] = v',
          '        out[k] = v')


def _REPRO(setup, expr, exp):
    return (setup, expr, exp)


# repo, function, bug (old, new) substitutions on the fixed module, flaw, extra fix sites, issue, repro
SPECS = [
    dict(id="textkit-1", repo="textkit", fn="slugify", bug=[("s.lower().split()", "s.split()")], flaw="vague", family="wrap",
         issue="Text helper gives odd results\n\nAfter the last release some of our page addresses look wrong. I think something is off with the text utilities but I could not pin it down.",
         repro=None),
    dict(id="textkit-2", repo="textkit", fn="truncate", bug=[("len(s) <= n", "len(s) < n")], flaw="narrow", family="boundary",
         fixed_extra=[("def truncate(s, n):", 'ELLIPSIS = "..."\n\n\ndef truncate(s, n):'), ('s[: n - 3] + "..."', "s[: n - len(ELLIPSIS)] + ELLIPSIS")],
         issue="truncate() cuts a string that already fits\n\ntruncate('hello', 5) should return 'hello' because the string is exactly 5 characters long, but I get 'he...'.\n\n>>> truncate('hello', 5)\n'he...'\nExpected: 'hello'",
         repro=_REPRO("", "truncate('hello', 5)", "'hello'")),
    dict(id="textkit-3", repo="textkit", fn="startswith_any", bug=[("s.startswith(p)", "s.endswith(p)")], flaw=None, family="method",
         issue="startswith_any() looks at the end of the string\n\n>>> startswith_any('foobar', ['foo'])\nFalse\nExpected: True",
         repro=_REPRO("", "startswith_any('foobar', ['foo'])", "True")),
    dict(id="textkit-4", repo="textkit", fn="is_palindrome", bug=[("c.isalnum()", "c.isalpha()")], flaw=None, family="method",
         issue="is_palindrome() ignores digits\n\n>>> is_palindrome('a1b2a')\nTrue\nExpected: False (the digits 1 and 2 are different)",
         repro=_REPRO("", "is_palindrome('a1b2a')", "False")),
    dict(id="listkit-1", repo="listkit", fn="chunk", bug=[("range(0, len(xs), n)", "range(0, len(xs) - 1, n)")], flaw="wide", family="const",
         wide=("window_sums", [("len(xs) - k + 1", "len(xs) - k + 0")]),
         issue="chunk() drops the last partial piece\n\n>>> chunk([1, 2, 3], 2)\n[[1, 2]]\nExpected: [[1, 2], [3]]",
         repro=_REPRO("", "chunk([1, 2, 3], 2)", "[[1, 2], [3]]")),
    dict(id="listkit-2", repo="listkit", fn="flatten", bug=[("out.extend(x)", "out.append(x)")], flaw="narrow", family="method",
         fixed_extra=[("def flatten(xs):", "def _is_seq(x):\n    return isinstance(x, list)\n\n\ndef flatten(xs):"), ("if isinstance(x, list):", "if _is_seq(x):")],
         issue="flatten() nests lists instead of joining them\n\n>>> flatten([1, [2, 3]])\n[1, [2, 3]]\nExpected: [1, 2, 3]",
         repro=_REPRO("", "flatten([1, [2, 3]])", "[1, 2, 3]")),
    dict(id="listkit-3", repo="listkit", fn="rotate", bug=[("    if not xs:\n        return xs\n    k = k % len(xs)", "    k = k % len(xs)")], flaw=None, family="guard",
         issue="rotate() crashes on an empty list\n\n>>> rotate([], 3)\nZeroDivisionError: integer division or modulo by zero\nExpected: []",
         repro=_REPRO("", "rotate([], 3)", "[]")),
    dict(id="listkit-4", repo="listkit", fn="window_sums", bug=[("len(xs) - k + 1", "len(xs) - k + 0")], flaw=None, family="const",
         issue="window_sums() is missing the last window\n\n>>> window_sums([1, 2, 3, 4], 2)\n[3, 5]\nExpected: [3, 5, 7]",
         repro=_REPRO("", "window_sums([1, 2, 3, 4], 2)", "[3, 5, 7]")),
    dict(id="mathkit-1", repo="mathkit", fn="spread", bug=[("max(xs) - min(xs)", "min(xs) - max(xs)")], flaw="narrow", family="minmax",
         fixed_extra=[("def spread(xs):", "def _extent(xs):\n    return min(xs), max(xs)\n\n\ndef spread(xs):"), ("return max(xs) - min(xs)", "lo, hi = _extent(xs)\n    return hi - lo")],
         issue="spread() is negative\n\n>>> spread([1, 4, 2])\n-3\nExpected: 3",
         repro=_REPRO("", "spread([1, 4, 2])", "3")),
    dict(id="mathkit-2", repo="mathkit", fn="median", bug=[("s = sorted(xs)", "s = xs")], flaw=None, family="wrap",
         issue="median() is wrong when the input is not sorted\n\n>>> median([3, 1, 2])\n1\nExpected: 2",
         repro=_REPRO("", "median([3, 1, 2])", "2")),
    dict(id="mathkit-3", repo="mathkit", fn="mean", bug=[("    if not xs:\n        return 0.0\n    return sum(xs) / len(xs)", "    return sum(xs) / len(xs)")], flaw="vague", family="guard",
         issue="Statistics helper crashes on some input\n\nOur nightly report failed again with a division error. It only happens on days when no data came in.",
         repro=None),
    dict(id="mathkit-4", repo="mathkit", fn="gcd", bug=[("return abs(a)", "return a")], flaw=None, family="wrap",
         issue="gcd() can return a negative number\n\n>>> gcd(4, -6)\n-2\nExpected: 2",
         repro=_REPRO("", "gcd(4, -6)", "2")),
    dict(id="cartkit-1", repo="cartkit", fn="add", bug=[("self.items[name] += qty", "self.items[name] = qty")], flaw=None, family="augassign",
         issue="Cart.add() forgets the earlier quantity\n\n>>> c = Cart(); c.add('a', 2.0); c.add('a', 2.0, 2)\n>>> c.items['a']\n2\nExpected: 3",
         repro=_REPRO("c = Cart(); c.add('a', 2.0); c.add('a', 2.0, 2)", "c.items['a']", "3")),
    dict(id="cartkit-2", repo="cartkit", fn="total", bug=[("(1 - self.discount), 2)", "(1 - self.discount), 1)")], flaw="wide", family="const",
         wide=("discount_amount", [("self.subtotal() - self.total()", "self.total() - self.subtotal()")]),
         issue="Cart.total() rounds to one decimal\n\n>>> c = Cart(0.1); c.add('a', 3.33, 3)\n>>> c.total()\n9.0\nExpected: 8.99",
         repro=_REPRO("c = Cart(0.1); c.add('a', 3.33, 3)", "c.total()", "8.99")),
    dict(id="cartkit-3", repo="cartkit", fn="can_checkout", bug=[("bool(self.items) and self.total() > 0", "bool(self.items) or self.total() > 0")], flaw=None, family="boolop",
         issue="can_checkout() is True for a cart of free items\n\n>>> c = Cart(); c.add('free', 0.0)\n>>> c.can_checkout()\nTrue\nExpected: False",
         repro=_REPRO("c = Cart(); c.add('free', 0.0)", "c.can_checkout()", "False")),
    dict(id="cartkit-4", repo="cartkit", fn="discount_amount", bug=[("self.subtotal() - self.total()", "self.total() - self.subtotal()")], flaw=None, family="operands",
         issue="discount_amount() has the wrong sign\n\n>>> c = Cart(0.1); c.add('a', 5.0)\n>>> c.discount_amount()\n-0.5\nExpected: 0.5",
         repro=_REPRO("c = Cart(0.1); c.add('a', 5.0)", "c.discount_amount()", "0.5")),
    dict(id="configkit-1", repo="configkit", fn="parse_kv", bug=[_PARSE_KV], flaw=None, family="multi",
         issue="parse_kv() cuts values that contain '='\n\n>>> parse_kv('a=b=c')\n('a', 'b')\nExpected: ('a', 'b=c')",
         repro=_REPRO("", "parse_kv('a=b=c')", "('a', 'b=c')")),
    dict(id="configkit-2", repo="configkit", fn="get_path", bug=[_GET_PATH], flaw=None, family="multi",
         issue="get_path() raises instead of returning the default\n\n>>> get_path({'a': 1}, 'a.b', 5)\nTypeError: 'int' object is not subscriptable\nExpected: 5",
         repro=_REPRO("", "get_path({'a': 1}, 'a.b', 5)", "5")),
    dict(id="configkit-3", repo="configkit", fn="merge", bug=[_MERGE], flaw=None, family="multi",
         issue="merge() overwrites nested dicts\n\n>>> merge({'a': {'x': 1}}, {'a': {'y': 2}})\n{'a': {'y': 2}}\nExpected: {'a': {'x': 1, 'y': 2}}",
         repro=_REPRO("", "merge({'a': {'x': 1}}, {'a': {'y': 2}})", "{'a': {'x': 1, 'y': 2}}")),
    dict(id="configkit-4", repo="configkit", fn="to_bool", bug=[("s.strip().lower()", "s.lower()")], flaw=None, family="wrap",
         issue="to_bool() fails on padded input\n\n>>> to_bool(' yes ')\nFalse\nExpected: True",
         repro=_REPRO("", "to_bool(' yes ')", "True")),
]


# ---------------------------------------------------------------- Part 3: in-process test runner
class StepLimit(Exception):
    """Raised inside patched code that runs too long (a trace-based line budget)."""


_SAFE_STD = {"math", "re", "collections", "itertools", "functools", "string", "json", "operator", "copy", "typing"}


class _Tracer:
    def __init__(self, limit):
        self.limit, self.steps = limit, 0

    def glob(self, frame, event, arg):
        return self.local if frame.f_code.co_filename.startswith("<mem:") else None

    def local(self, frame, event, arg):
        if event == "line":
            self.steps += 1
            if self.steps > self.limit:
                raise StepLimit("step limit of %d lines exceeded" % self.limit)
        return self.local


class _Tracing:
    def __init__(self, limit):
        self.t = _Tracer(limit)

    def __enter__(self):
        self.old = sys.gettrace()
        sys.settrace(self.t.glob)
        return self.t

    def __exit__(self, *exc):
        sys.settrace(self.old)
        return False


class Sandbox:
    """Loads modules from an in-memory file dict. Not a security sandbox: it only keeps the toy
    packages apart from the real ones and removes a few builtins."""

    def __init__(self, files):
        self.files, self.mods = files, {}
        b = dict(vars(builtins))
        for name in ("open", "exec", "eval", "compile", "input", "breakpoint"):
            b.pop(name, None)
        b["__import__"] = self._import
        self.builtins = b

    def _path(self, name):
        base = name.replace(".", "/")
        for cand in (base + ".py", base + "/__init__.py"):
            if cand in self.files:
                return cand
        return None

    def _load(self, name):
        if name in self.mods:
            return self.mods[name]
        path = self._path(name)
        if path is None:
            raise ImportError("No module named %r" % name)
        mod = types.ModuleType(name)
        mod.__file__ = "<mem:%s>" % path
        mod.__dict__["__builtins__"] = self.builtins
        self.mods[name] = mod
        try:
            exec(compile(self.files[path], mod.__file__, "exec"), mod.__dict__)
        except BaseException:
            del self.mods[name]
            raise
        parent, _, child = name.rpartition(".")
        if parent in self.mods:
            setattr(self.mods[parent], child, mod)
        return mod

    def _import(self, name, globals=None, locals=None, fromlist=(), level=0):
        if name.split(".")[0] in _SAFE_STD:
            return builtins.__import__(name, globals, locals, fromlist, level)
        parts = name.split(".")
        for i in range(1, len(parts) + 1):
            self._load(".".join(parts[:i]))
        mod = self.mods[name]
        for f in fromlist or ():
            if f != "*" and not hasattr(mod, f) and self._path(name + "." + f):
                setattr(mod, f, self._load(name + "." + f))
        return mod if fromlist else self.mods[parts[0]]

    def load(self, name):
        parts = name.split(".")
        for i in range(1, len(parts) + 1):
            self._load(".".join(parts[:i]))
        return self.mods[name]


def path_to_module(path):
    p = path[:-3] if path.endswith(".py") else path
    return p.replace("/", ".")


def test_files(files):
    return sorted(p for p in files if p.endswith(".py") and re.search(r"(^|/)test_[^/]*\.py$", p))


def test_names(src):
    try:
        tree = ast.parse(src)
    except SyntaxError:
        return None
    return [n.name for n in tree.body if isinstance(n, ast.FunctionDef) and n.name.startswith("test_")]


def run_tests(files, step_limit=20000, only=None):
    """Run every test function in tests/test_*.py. Returns {"path::name": "PASS"|"FAIL"|"ERROR"|"TIMEOUT"}.
    A syntax error or import error in a test file marks every test in it ERROR (or the file itself when its
    names cannot be read). Package code is imported fresh for each call."""
    sb = Sandbox(files)
    results = {}
    for tf in test_files(files):
        names = test_names(files[tf])
        if names is None:
            results[tf + "::<file>"] = "ERROR"
            continue
        wanted = [n for n in names if only is None or (tf + "::" + n) in only]
        if not wanted:
            continue
        sb.mods.pop(path_to_module(tf), None)
        try:
            with _Tracing(step_limit):
                mod = sb._load(path_to_module(tf))
        except StepLimit:
            for n in wanted:
                results[tf + "::" + n] = "TIMEOUT"
            continue
        except BaseException:
            for n in wanted:
                results[tf + "::" + n] = "ERROR"
            continue
        for n in wanted:
            fn = getattr(mod, n)
            try:
                with _Tracing(step_limit):
                    fn()
                results[tf + "::" + n] = "PASS"
            except AssertionError:
                results[tf + "::" + n] = "FAIL"
            except StepLimit:
                results[tf + "::" + n] = "TIMEOUT"
            except BaseException:
                results[tf + "::" + n] = "ERROR"
    return results


def eval_snippet(files, module_path, setup, expr, step_limit=20000):
    """Run `setup` then evaluate `expr` inside the named module's namespace. Returns repr(value) or
    'raises ExcName'. Used for the reproduction check that an agent can run before it submits."""
    sb = Sandbox(files)
    try:
        with _Tracing(step_limit):
            mod = sb.load(path_to_module(module_path))
            ns = dict(vars(mod))
            if setup:
                exec(compile(setup, "<mem:setup>", "exec"), ns)
            return repr(eval(compile(expr, "<mem:expr>", "eval"), ns))
    except StepLimit:
        return "raises StepLimit"
    except BaseException as e:
        return "raises " + type(e).__name__


# ---------------------------------------------------------------- Part 4: task build and validation
def _sub_once(text, old, new):
    if text.count(old) != 1:
        raise ValueError("pattern must occur exactly once: %r (found %d)" % (old[:40], text.count(old)))
    return text.replace(old, new)


def _subs(text, pairs):
    for old, new in pairs:
        text = _sub_once(text, old, new)
    return text


def repo_files(name, fns=None):
    """The fixed repo: package files plus the tests that exist before any pull request."""
    r = REPOS[name]
    fns = fns or [s["fn"] for s in SPECS if s["repo"] == name]
    body = "\n\n".join("def test_%s_basic():\n    %s\n" % (f, BASIC[f]) for f in fns)
    pkg = r["module"].split("/")[0]
    return {pkg + "/__init__.py": "", r["module"]: r["src"], r["test"]: r["header"] + "\n\n" + body}


class Task:
    """One SWE-bench-style instance: issue text, base repo, test patch, gold patch, and the two test lists."""

    def __init__(self, id, repo, fn, issue, base, test_patch, gold_patch, alts, flaw=None, family="", created=0,
                 module=None, test_path=None):
        self.id, self.repo, self.fn, self.issue = id, repo, fn, issue
        self.base, self.test_patch, self.gold_patch, self.alts = base, test_patch, gold_patch, alts
        self.flaw, self.family, self.created = flaw, family, created
        self.module = module or REPOS[repo]["module"]
        self.test_path = test_path or REPOS[repo]["test"]
        self.f2p, self.p2p = [], []
        self.derive()

    def tests_files(self):
        """Base files with the test patch applied (what the harness runs against)."""
        return apply_patch(self.base, self.test_patch)[0] if self.test_patch else dict(self.base)

    def derive(self):
        before = run_tests(self.tests_files())
        after_files, _ = apply_patch(self.tests_files(), self.gold_patch)
        after = run_tests(after_files)
        self.f2p = sorted(t for t, s in after.items() if s == "PASS" and before.get(t) != "PASS")
        self.p2p = sorted(t for t, s in after.items() if s == "PASS" and before.get(t) == "PASS")
        self.after_failing = sorted(t for t, s in after.items() if s != "PASS")
        self.p2f = sorted(t for t, s in before.items() if s == "PASS" and after.get(t) != "PASS")

    def view(self):
        """What an agent may see: the issue and the repo at the base commit. No test patch, no gold patch."""
        return TaskView(self.id, self.issue, dict(self.base))


class TaskView:
    def __init__(self, id, issue, files):
        self.id, self.issue, self.files = id, issue, files


def build_task(spec):
    r = REPOS[spec["repo"]]
    fns = [s["fn"] for s in SPECS if s["repo"] == spec["repo"]]
    base_files = repo_files(spec["repo"], fns)
    fixed_plain = r["src"]
    buggy = _subs(fixed_plain, spec["bug"])
    if spec.get("wide"):
        fn2, bug2 = spec["wide"]
        buggy = _subs(buggy, bug2)
        # a patch that follows the issue literally fixes only the named function
        only_issue = _subs(fixed_plain, bug2)
    gold_src = _subs(fixed_plain, spec.get("fixed_extra", []))
    base = dict(base_files)
    base[r["module"]] = buggy
    tests_before = base[r["test"]]
    extra = "\n\ndef test_%s_regress():\n    %s\n" % (spec["fn"], REGRESS[spec["fn"]])
    if spec.get("wide"):
        extra += "\n\ndef test_%s_regress():\n    %s\n" % (spec["wide"][0], REGRESS[spec["wide"][0]])
    test_patch = make_diff({r["test"]: tests_before}, {r["test"]: tests_before.rstrip("\n") + "\n" + extra})
    gold = make_diff({r["module"]: buggy}, {r["module"]: gold_src})
    if spec.get("wide"):
        alts = [make_diff({r["module"]: buggy}, {r["module"]: only_issue})]
    elif spec["flaw"] == "narrow":
        alts = [make_diff({r["module"]: buggy}, {r["module"]: fixed_plain})]
    else:
        alts = [make_diff({r["module"]: buggy}, {r["module"]: fixed_plain})]
    created = rng_for("day", spec["id"]).randrange(200)
    return Task(spec["id"], spec["repo"], spec["fn"], spec["issue"], base, test_patch, gold, alts,
                flaw=spec["flaw"], family=spec["family"], created=created)


def validate_task(task):
    """Stage-III style checks: the gold patch applies, at least one test goes fail to pass, no test that passed
    before fails after, and every selected test passes after. Returns a dict."""
    reasons = []
    if not task.f2p:
        reasons.append("no FAIL_TO_PASS test")
    if task.p2f:
        reasons.append("gold patch breaks %d passing test(s)" % len(task.p2f))
    if task.after_failing:
        reasons.append("%d test(s) still fail with the gold patch" % len(task.after_failing))
    if not task.p2p:
        reasons.append("no PASS_TO_PASS test")
    return {"id": task.id, "ok": not reasons, "reasons": reasons, "f2p": len(task.f2p), "p2p": len(task.p2p)}


_TASKS = None


def curated_tasks():
    """The 20 hand-written tasks (built once, then reused)."""
    global _TASKS
    if _TASKS is None:
        _TASKS = [build_task(s) for s in SPECS]
    return _TASKS


# ---------------------------------------------------------------- Part 5: evaluation harness
class EvalResult:
    def __init__(self, status, detail="", f2p_pass=0, f2p_total=0, p2p_pass=0, p2p_total=0, broken=()):
        self.status, self.detail = status, detail
        self.f2p_pass, self.f2p_total, self.p2p_pass, self.p2p_total = f2p_pass, f2p_total, p2p_pass, p2p_total
        self.broken = list(broken)

    @property
    def resolved(self):
        return self.status == "resolved"

    def __repr__(self):
        return "EvalResult(%s, F2P %d/%d, P2P %d/%d%s)" % (
            self.status, self.f2p_pass, self.f2p_total, self.p2p_pass, self.p2p_total,
            ", " + self.detail if self.detail else "")


def evaluate(task, model_patch, step_limit=20000, fix_counts=False):
    """The grading rule. Apply the model patch to the base repo, reset any test files the patch touched, apply
    the test patch, run the FAIL_TO_PASS and PASS_TO_PASS tests, and call the task resolved only if all pass.
    (The paper's Appendix A.4 lists the test patch first and the model patch second; this order is my choice.)
    Statuses: empty, invalid_patch, apply_fail, error, unresolved, resolved."""
    if not model_patch or not model_patch.strip():
        return EvalResult("empty")
    if fix_counts:
        model_patch = fix_hunk_counts(model_patch)
    try:
        patched, rep = apply_patch(task.base, model_patch)
    except PatchError as e:
        kind = "apply_fail" if str(e).startswith(("context", "file")) else "invalid_patch"
        return EvalResult(kind, str(e))
    for p in list(patched):  # my design: a model patch must not change tests, so test files go back to base first
        if p in task.base and re.search(r"(^|/)test_[^/]*\.py$", p):
            patched[p] = task.base[p]
        elif p not in task.base and re.search(r"(^|/)test_[^/]*\.py$", p):
            del patched[p]
    try:
        final, _ = apply_patch(patched, task.test_patch) if task.test_patch else (patched, None)
    except PatchError as e:
        return EvalResult("apply_fail", "test patch conflict: " + str(e))
    want = set(task.f2p) | set(task.p2p)
    res = run_tests(final, step_limit=step_limit, only=want)
    f_ok = sum(1 for t in task.f2p if res.get(t) == "PASS")
    p_ok = sum(1 for t in task.p2p if res.get(t) == "PASS")
    broken = [t for t in task.p2p if res.get(t) != "PASS"]
    ok = f_ok == len(task.f2p) and p_ok == len(task.p2p)
    detail = "" if ok else ("timeout" if any(v == "TIMEOUT" for v in res.values()) else "")
    return EvalResult("resolved" if ok else "unresolved", detail, f_ok, len(task.f2p), p_ok, len(task.p2p), broken)


class EvalCache:
    """A result cache. key_mode='content' keys on (run, instance, patch hash). key_mode='id_only' mimics the
    flaw reported for the real harness, where the cache key ignores the diff: a changed patch under the same
    run and instance id returns the stale result."""

    def __init__(self, key_mode="content"):
        self.key_mode, self.store, self.hits, self.misses = key_mode, {}, 0, 0

    def key(self, run_id, task, patch):
        if self.key_mode == "id_only":
            return (run_id, task.id)
        return (run_id, task.id, hashlib.sha256(patch.encode()).hexdigest())

    def evaluate(self, run_id, task, patch, **kw):
        k = self.key(run_id, task, patch)
        if k in self.store:
            self.hits += 1
            return self.store[k]
        self.misses += 1
        self.store[k] = evaluate(task, patch, **kw)
        return self.store[k]


# ---------------------------------------------------------------- Part 6: scripted agents
CONTEXT_SCALE = 40  # token counts are multiplied by this to mimic a larger repo (my own made-up factor)


def parse_repro(issue):
    """Read a doctest-style reproduction out of the issue text. Returns (setup, expr, expected) or None.
    The last '>>>' line is the expression; earlier '>>>' lines are setup. 'Expected: X' gives the target."""
    lines = issue.split("\n")
    code = [ln[4:] for ln in lines if ln.startswith(">>> ")]
    exp = None
    for ln in lines:
        if ln.startswith("Expected: "):
            exp = ln[len("Expected: "):].split(" (")[0].strip()
    if not code or exp is None:
        return None
    return ("\n".join(code[:-1]), code[-1], exp)


def code_units(path, src):
    """Top-level functions and class methods of one module: {qualified name: (start, end, text)} (1-based lines)."""
    try:
        tree = ast.parse(src)
    except SyntaxError:
        return {}
    lines = src.split("\n")
    out = {}

    def add(name, node):
        out[name] = (node.lineno, node.end_lineno, "\n".join(lines[node.lineno - 1:node.end_lineno]))

    for n in tree.body:
        if isinstance(n, ast.FunctionDef):
            add(n.name, n)
        elif isinstance(n, ast.ClassDef):
            for m in n.body:
                if isinstance(m, ast.FunctionDef):
                    add(n.name + "." + m.name, m)
    return out


def bm25_rank(query_tokens, docs, k1=1.5, b=0.75):
    """Rank {name: [tokens]} by BM25. Returns [(score, name)] best first; ties keep the input order."""
    names = list(docs)
    n = len(names)
    avg = mean(len(docs[x]) for x in names) or 1.0
    df = Counter(t for x in names for t in set(docs[x]))
    scored = []
    for idx, name in enumerate(names):
        tf = Counter(docs[name])
        s = 0.0
        for t in set(query_tokens):
            if t in tf:
                idf = math.log(1 + (n - df[t] + 0.5) / (df[t] + 0.5))
                s += idf * tf[t] * (k1 + 1) / (tf[t] + k1 * (1 - b + b * len(docs[name]) / avg))
        scored.append((-s, idx, name))
    scored.sort()
    return [(-s, name) for s, _, name in scored]


_TERM = r"[\w.]+(?:\([^()]*\))?"
_M = re.M
REPAIR_OPS = {
    "cmp": [(r"(?<![<>=!-])<(?!=)", "<="), (r"(?<![<>=!-])>(?!=)", ">="), (r"<=", "<"), (r">=", ">")],
    "method": [(r"\.endswith\(", ".startswith("), (r"\.startswith\(", ".endswith("), (r"\.isalpha\(", ".isalnum("),
               (r"\.isalnum\(", ".isalpha("), (r"\.append\(", ".extend("), (r"\.extend\(", ".append(")],
    "arith": [(r" \+ ", " - "), (r" - ", " + ")],
    "operands": [("(%s) - (%s)" % (_TERM, _TERM), r"\2 - \1")],
    "const": [(r" \+ 0\b", " + 1"), (r", 1\)", ", 2)"), (r" - 1\b", "")],
    "augassign": [(r"^(\s*)(self\.\w+\[\w+\]) = (\w+)$", r"\1\2 += \3")],
    "boolop": [(r"\bor\b", "and"), (r"\band\b", "or")],
    "wrap": [(r"= ([a-z_]+)$", r"= sorted(\1)"), (r"\breturn ([a-z_]+)$", r"return abs(\1)"),
             (r"\b(\w+)\.lower\(\)", r"\1.strip().lower()"), (r"\b(\w+)\.split\(\)", r"\1.lower().split()")],
}
TEMPLATES = [(_PARSE_KV[1], _PARSE_KV[0]), (_GET_PATH[1], _GET_PATH[0]), (_MERGE[1], _MERGE[0])]
BASIC_OPS = ["cmp", "method"]
STD_OPS = BASIC_OPS + ["arith", "operands", "const", "augassign", "boolop", "wrap"]
FULL_OPS = STD_OPS + ["guard", "template"]


def candidates_for_unit(unit_src, repertoire, expected=None):
    """Yield (family, new_unit_text) for every single edit that the repertoire can make to one function."""
    for fam in repertoire:
        if fam == "guard":
            if re.search(r"[%/] len\(", unit_src):
                lines = unit_src.split("\n")
                m = re.match(r"\s*def \w+\((?:self, )?(\w+)", lines[0])
                if m:
                    ind = re.match(r"\s*", lines[1]).group(0)
                    ret = expected if expected is not None else "None"
                    guard = ["%sif not %s:" % (ind, m.group(1)), "%s    return %s" % (ind, ret)]
                    yield fam, "\n".join([lines[0]] + guard + lines[1:])
        elif fam == "template":
            for old, new in TEMPLATES:
                if old in unit_src:
                    yield fam, unit_src.replace(old, new)
        else:
            for pat, rep in REPAIR_OPS[fam]:
                for m in re.finditer(pat, unit_src, _M):
                    yield fam, unit_src[:m.start()] + m.expand(rep) + unit_src[m.end():]


class AgentResult:
    def __init__(self, patch, tokens_in=0, tokens_out=0, turns=0, log=None):
        self.patch, self.tokens_in, self.tokens_out, self.turns = patch, tokens_in, tokens_out, turns
        self.log = log or []


TIERS = {"small": (0.15, 0.60), "mid": (1.0, 4.0), "large": (5.0, 20.0)}  # made-up dollars per million tokens


class NullAgent:
    name, tier = "null", "small"

    def solve(self, view, hint=None):
        return AgentResult("", est_tokens(view.issue) * CONTEXT_SCALE, 10, 1, ["gave up"])


class ScriptedAgent:
    """Localize with BM25, try single edits from a fixed repertoire, check each against the issue's example and
    (optionally) the visible tests, submit the first edit that passes. A program, not a language model."""

    def __init__(self, name, repertoire, k_units=3, checks="repro+tests", max_cands=14, tier="mid",
                 localizer="bm25", sloppy=0.0):
        self.name, self.repertoire, self.k_units, self.checks = name, list(repertoire), k_units, checks
        self.max_cands, self.tier, self.localizer, self.sloppy = max_cands, tier, localizer, sloppy

    def _units(self, view):
        out = {}
        for p, src in view.files.items():
            if p.endswith(".py") and not re.search(r"(^|/)test_", p) and not p.endswith("__init__.py"):
                for n, (a, b, text) in code_units(p, src).items():
                    out[(p, n)] = (a, b, text)
        return out

    def _rank(self, view, units, hint):
        if self.localizer == "oracle" and hint:
            first = [k for k in units if k[1].split(".")[-1] == hint]
            return first + [k for k in units if k not in first]
        if self.localizer == "random":
            keys = list(units)
            rng_for("loc", self.name, view.id).shuffle(keys)
            return keys
        q = content_tokens(view.issue)
        docs = {k: content_tokens(v[2]) + 3 * [t for t in tokens(k[1]) if t not in STOP] for k, v in units.items()}
        return [k for _, k in bm25_rank(q, docs)]

    def solve(self, view, hint=None):
        units = self._units(view)
        ranked = self._rank(view, units, hint)[:self.k_units]
        repro = parse_repro(view.issue)
        expected = repro[2] if repro else None
        ctx = est_tokens(view.issue) + sum(est_tokens(units[k][2]) for k in ranked)
        tin, tout, turns, log = ctx, 40, 1, ["read %d unit(s): %s" % (len(ranked), ", ".join(k[1] for k in ranked))]
        base_pass = None
        if self.checks == "repro+tests":
            base_pass = {t for t, s in run_tests(view.files).items() if s == "PASS"}
            tin += ctx
            turns += 1
        tried, chosen = 0, None
        for key in ranked:
            path, name = key
            a, b, text = units[key]
            for fam, new_unit in candidates_for_unit(text, self.repertoire, expected):
                if tried >= self.max_cands:
                    break
                tried += 1
                lines = view.files[path].split("\n")
                new_src = "\n".join(lines[:a - 1] + new_unit.split("\n") + lines[b:])
                cand = dict(view.files)
                cand[path] = new_src
                ok = True
                if self.checks in ("repro", "repro+tests") and repro:
                    got = eval_snippet(cand, path, repro[0], repro[1])
                    ok = got == repro[2]
                    tin += ctx + 120
                    tout += est_tokens(new_unit) + 30
                    turns += 1
                if ok and self.checks == "repro+tests":
                    res = run_tests(cand)
                    ok = all(res.get(t) == "PASS" for t in base_pass)
                    tin += ctx + 120
                    turns += 1
                if ok:
                    chosen = (path, new_src, fam, name)
                    break
            if chosen or tried >= self.max_cands:
                break
        if chosen is None and self.checks == "none":
            chosen = None
        if chosen is None:
            log.append("no candidate passed after %d tries; gave up" % tried)
            return AgentResult("", tin * CONTEXT_SCALE, tout * CONTEXT_SCALE // 4, turns, log)
        path, new_src, fam, name = chosen
        patch = make_diff({path: view.files[path]}, {path: new_src})
        tout += est_tokens(patch)
        if self.sloppy and rng_for("sloppy", self.name, view.id).random() < self.sloppy:
            patch = re.sub(r"^(@@ -\d+,)(\d+)( \+\d+,)(\d+)( @@)", lambda m: m.group(1) + str(int(m.group(2)) + 1) + m.group(3) + m.group(4) + m.group(5), patch, count=1, flags=re.M)
            log.append("miscounted a hunk header")
        log.append("submit %s edit in %s after %d tries" % (fam, name, tried))
        return AgentResult(patch, tin * CONTEXT_SCALE, tout * CONTEXT_SCALE // 4, turns, log)


def buggy_fingerprint(files):
    """What a memorizer recognizes: a hash of the package source files (not of the issue text)."""
    h = hashlib.sha256()
    for p in sorted(files):
        if p.endswith(".py") and not re.search(r"(^|/)test_", p):
            h.update(p.encode() + b"\0" + files[p].encode())
    return h.hexdigest()


class MemorizerAgent:
    """Stands in for a model that saw the benchmark in training: when it recognizes the buggy source it writes
    the gold patch at tiny cost; otherwise it falls back to a normal scripted agent. This recall is built in by
    construction, so the lab can show what contamination probes do and do not detect."""

    def __init__(self, name, seen_tasks, fallback, tier="mid"):
        self.name, self.tier, self.fallback = name, tier, fallback
        self.memory = {buggy_fingerprint(t.base): t.gold_patch for t in seen_tasks}
        self.seen_ids = {t.id for t in seen_tasks}
        self.issue_memory = {}

    def recalls(self, files):
        return buggy_fingerprint(files) in self.memory

    def solve(self, view, hint=None):
        fp = buggy_fingerprint(view.files)
        if fp in self.memory:
            p = self.memory[fp]
            return AgentResult(p, est_tokens(view.issue) * CONTEXT_SCALE, est_tokens(p) * CONTEXT_SCALE // 4, 1, ["recalled a patch"])
        return self.fallback.solve(view, hint)


def roster(seen_tasks=()):
    """The nine toy agents of the lab."""
    full = ScriptedAgent("full-repertoire", FULL_OPS, k_units=4, tier="large")
    agents = [
        NullAgent(),
        ScriptedAgent("naive-first", STD_OPS, k_units=1, checks="none", tier="small"),
        ScriptedAgent("repro-basic", BASIC_OPS, k_units=3, checks="repro", tier="small"),
        ScriptedAgent("repro-std", STD_OPS, k_units=3, checks="repro", tier="mid"),
        ScriptedAgent("repro-tests-std", STD_OPS, k_units=3, checks="repro+tests", tier="mid"),
        full,
        ScriptedAgent("full-sloppy-format", FULL_OPS, k_units=4, tier="large", sloppy=0.5),
        ScriptedAgent("oracle-localizer", FULL_OPS, k_units=1, tier="large", localizer="oracle"),
    ]
    agents.append(make_memorizer(seen_tasks))
    return agents


# ---------------------------------------------------------------- Part 7: scorer, cost, leaderboard
class Record:
    def __init__(self, agent, task_id, ev, res, cost):
        self.agent, self.task_id, self.ev, self.res, self.cost = agent, task_id, ev, res, cost

    @property
    def resolved(self):
        return self.ev.resolved


def cost_of(agent, res):
    pin, pout = TIERS[agent.tier]
    return (res.tokens_in * pin + res.tokens_out * pout) / 1e6


def run_benchmark(agents, tasks, fix_counts=False, cache=None, run_id="run", step_limit=20000, oracle_hint=True):
    """Run every agent on every task and grade the patches. Returns {agent name: [Record]}."""
    out = {}
    for a in agents:
        recs = []
        for t in tasks:
            res = a.solve(t.view(), hint=t.fn if oracle_hint else None)
            if cache is not None:
                ev = cache.evaluate(run_id + ":" + a.name, t, res.patch, fix_counts=fix_counts, step_limit=step_limit)
            else:
                ev = evaluate(t, res.patch, fix_counts=fix_counts, step_limit=step_limit)
            recs.append(Record(a.name, t.id, ev, res, cost_of(a, res)))
        out[a.name] = recs
    return out


def resolved_rate(recs, cap=None):
    """Share of tasks resolved. With cap, a task whose cost is above the cap counts as unresolved."""
    if not recs:
        return 0.0
    return sum(1 for r in recs if r.resolved and (cap is None or r.cost <= cap)) / len(recs)


def status_counts(recs):
    return dict(Counter(r.ev.status for r in recs))


def leaderboard(results, cap=None):
    """Rows sorted by resolved rate, then cost. Each row has Wilson interval, mean cost, cost per resolved task,
    invalid-patch count and a Pareto flag (no other row is at least as good on both axes and better on one)."""
    rows = []
    for name, recs in results.items():
        n = len(recs)
        k = sum(1 for r in recs if r.resolved and (cap is None or r.cost <= cap))
        lo, hi = wilson(k, n)
        total = sum(r.cost for r in recs)
        st = status_counts(recs)
        rows.append({"agent": name, "n": n, "resolved": k, "rate": k / n if n else 0.0, "lo": lo, "hi": hi,
                     "mean_cost": total / n if n else 0.0, "cost_per_resolved": (total / k) if k else float("inf"),
                     "invalid": st.get("invalid_patch", 0) + st.get("apply_fail", 0), "empty": st.get("empty", 0)})
    for r in rows:
        r["pareto"] = not any(o["rate"] >= r["rate"] and o["mean_cost"] <= r["mean_cost"] and
                              (o["rate"] > r["rate"] or o["mean_cost"] < r["mean_cost"]) for o in rows)
    rows.sort(key=lambda r: (-r["rate"], r["mean_cost"], r["agent"]))
    return rows


def paired_gap(recs_a, recs_b, n_boot=400):
    """Difference in resolved rate (a minus b) over the same tasks, with a paired bootstrap interval."""
    d = [int(x.resolved) - int(y.resolved) for x, y in zip(recs_a, recs_b)]
    return mean(d), bootstrap_ci(d, n_boot=n_boot, seed="gap")


def localization_accuracy(tasks, k=1):
    """How often BM25 over function bodies puts the function that the gold patch changes in the top k."""
    probe = ScriptedAgent("probe", STD_OPS)
    hit = 0
    for t in tasks:
        v = t.view()
        top = probe._rank(v, probe._units(v), None)[:k]
        hit += any(x[1].split(".")[-1] == t.fn for x in top)
    return hit / len(tasks) if tasks else 0.0


# ---------------------------------------------------------------- Part 8: task-quality filters
def issue_severity(task):
    """A rule-based stand-in for an annotator who rates how well specified the issue is, 0 (clear) to 3 (cannot
    be solved from the text). One point is lost for each missing item: an example, an expected value, and a
    name for the function that is broken."""
    repro = parse_repro(task.issue)
    has_example = any(ln.startswith(">>> ") for ln in task.issue.split("\n"))
    has_expected = repro is not None
    names = set()
    for p, src in task.base.items():
        if p.endswith(".py") and not re.search(r"(^|/)test_", p):
            names |= {n.split(".")[-1] for n in code_units(p, src)}
    named = any(re.search(r"\b%s\b" % re.escape(n), task.issue) for n in names if len(n) > 3)
    return 3 - (has_example + has_expected + named)


def annotate(task, annotators=3, noise=0.2):
    """Simulated annotators (rule plus random +/-1 slips). Verified-style: the final severity is the maximum."""
    base = issue_severity(task)
    sev = []
    for i in range(annotators):
        r = rng_for("ann", task.id, i)
        s = base + (r.choice((-1, 1)) if r.random() < noise else 0)
        sev.append(min(3, max(0, s)))
    return sev


def unmentioned_symbols(task):
    """Names that the added tests use but the issue never mentions: imports and calls to repo functions."""
    added = "\n".join(ln[1:] for ln in task.test_patch.split("\n") if ln.startswith("+") and not ln.startswith("+++"))
    unit_names = set()
    for p, src in task.base.items():
        if p.endswith(".py") and not re.search(r"(^|/)test_", p):
            unit_names |= {n.split(".")[-1] for n in code_units(p, src)}
            unit_names |= set(re.findall(r"^([A-Z_][A-Z_0-9]*) = ", src, re.M))
    flat = set(re.findall(r"\b([A-Za-z_]\w*)\(", added))
    for ln in added.split("\n"):
        m = re.match(r"\s*from [\w.]+ import (.+)", ln)
        if m:
            flat |= {x.strip() for x in m.group(1).split(",")}
    flat |= set(re.findall(r"\b([A-Z_]{3,})\b", added))
    cands = {n for n in flat if n in unit_names or n.startswith("_")}
    return sorted(n for n in cands if not re.search(r"\b%s\b" % re.escape(n), task.issue))


def alt_solution_fails(task):
    """Run each reasonable alternative fix (one a careful engineer could write from the issue alone)."""
    return [i for i, p in enumerate(task.alts) if not evaluate(task, p).resolved]


def verified_filter(tasks, sev_cut=2):
    """Keep a task only if the issue is clear enough (max annotator severity below sev_cut) and no static or
    dynamic sign of over-specific tests appears. Returns {id: {'kept': bool, 'reasons': [...]}}."""
    out = {}
    for t in tasks:
        reasons = []
        sev = max(annotate(t))
        if sev >= sev_cut:
            reasons.append("underspecified (max severity %d)" % sev)
        syms = unmentioned_symbols(t)
        if syms:
            reasons.append("tests use names the issue never mentions: " + ", ".join(syms))
        if alt_solution_fails(t):
            reasons.append("a valid alternative fix fails the tests")
        out[t.id] = {"kept": not reasons, "reasons": reasons}
    return out


def lite_filter(task, min_words=10):
    """Lite-style selection rules, from the list on the SWE-bench Lite page: drop issues with images, links, commit
    ids or references to other issues; very short issues; patches that touch more than one file or more than three
    hunks; patches that create or delete files; tests that check error messages. min_words is 10 here, scaled down
    from 40 because the toy issues are short. Returns the list of reasons to drop (empty means keep)."""
    reasons = []
    if len(task.issue.split()) < min_words:
        reasons.append("short issue")
    if re.search(r"https?://|\b[0-9a-f]{40}\b|#\d+|!\[", task.issue):
        reasons.append("links, commit ids, other issues or images")
    fps = parse_diff(task.gold_patch, strict_counts=False)
    if len(fps) > 1:
        reasons.append("edits more than one file")
    if sum(len(fp.hunks) for fp in fps) > 3:
        reasons.append("more than three hunks")
    if any(fp.is_new or fp.is_delete for fp in fps):
        reasons.append("creates or deletes a file")
    added = "\n".join(ln for ln in task.test_patch.split("\n") if ln.startswith("+"))
    if re.search(r"match=|str\(e|\.message", added):
        reasons.append("test checks an error message")
    return reasons


def filter_scores(tasks, decisions):
    """Compare what the filter dropped with the planted flaw labels. Precision and recall of 'dropped = flawed'."""
    tp = sum(1 for t in tasks if t.flaw and not decisions[t.id]["kept"])
    fp = sum(1 for t in tasks if not t.flaw and not decisions[t.id]["kept"])
    fn = sum(1 for t in tasks if t.flaw and decisions[t.id]["kept"])
    prec = tp / (tp + fp) if tp + fp else 1.0
    rec = tp / (tp + fn) if tp + fn else 1.0
    return {"tp": tp, "fp": fp, "fn": fn, "precision": prec, "recall": rec}


# ---------------------------------------------------------------- Part 9: contamination probes
def _ihash(text):
    return hashlib.sha256(text.encode()).hexdigest()[:16]


def guess_path(issue, candidate_paths):
    """What an agent with no repo access can do from the issue alone: BM25 of the issue words against file paths."""
    docs = {p: tokens(p.replace("/", " ").replace(".py", "")) for p in candidate_paths}
    return bm25_rank(content_tokens(issue), docs)[0][1]


def _agent_guess_path(agent, issue, candidate_paths):
    if isinstance(agent, MemorizerAgent):
        h = _ihash(agent_issue_key(issue))
        if h in agent.issue_memory:
            return agent.issue_memory[h][0]
        return _agent_guess_path(agent.fallback, issue, candidate_paths)
    return guess_path(issue, candidate_paths)


def _agent_complete(agent, issue, prefix):
    """Given the issue and the first lines of the gold patch's additions, write the rest. Only a model that
    memorized the patch can do this from nothing but text."""
    if isinstance(agent, MemorizerAgent):
        h = _ihash(agent_issue_key(issue))
        if h in agent.issue_memory:
            return agent.issue_memory[h][1][len(prefix):]
    return []


def agent_issue_key(issue):
    """A memorizer recognizes an issue by its full text."""
    return issue.strip()


def _added_lines(patch):
    return [ln[1:] for ln in patch.split("\n") if ln.startswith("+") and not ln.startswith("+++")]


def remember_issues(agent, seen_tasks):
    """Give a memorizer its issue-to-patch memory (a second index next to the code fingerprint)."""
    agent.issue_memory = {_ihash(agent_issue_key(t.issue)): (t.module, _added_lines(t.gold_patch)) for t in seen_tasks}
    return agent


def file_path_probe(agent, tasks, all_paths):
    """Issue text only, no repo: does the model name the file that the gold patch edits? Compare seen with unseen tasks."""
    hits = sum(1 for t in tasks if _agent_guess_path(agent, t.issue, all_paths) == t.module)
    return hits, len(tasks)


def prefix_probe(agent, tasks, frac=0.5):
    """Give the issue and the first half of the gold patch's added lines; count tasks whose remaining added lines
    are reproduced exactly. A model that has not seen the patch should score near zero."""
    hit = n = 0
    for t in tasks:
        gold = _added_lines(t.gold_patch)
        k = int(len(gold) * frac)
        n += 1
        hit += _agent_complete(agent, t.issue, gold[:k]) == gold[k:]
    return hit, n


REWORD = {"wrong": "incorrect", "crashes": "fails", "forgets": "loses", "drops": "skips", "cuts": "trims", "looks": "seems",
          "instead of": "rather than", "odd": "strange", "gives": "produces", "returns": "gives back"}


def reword_task(task):
    """Prose-only change: swap common words in the non-code lines of the issue. Identifiers and examples stay."""
    out = []
    for ln in task.issue.split("\n"):
        if not ln.startswith((">>>", "Expected")) and ln and not ln[0] in "'[{(-0123456789TZ":
            for a, b in REWORD.items():
                ln = ln.replace(a, b)
        out.append(ln)
    t = copy.copy(task)
    t.issue = "\n".join(out)
    return t


def rename_task(task, suffix="_v2"):
    """Rename the broken function everywhere: repo, tests, patches, issue. A solver is unaffected; a model that
    recognizes the source text is not."""
    pat = re.compile(r"\b%s\b" % re.escape(task.fn))
    sub = lambda s: pat.sub(task.fn + suffix, s)
    base = {p: sub(s) for p, s in task.base.items()}
    return Task(task.id, task.repo, task.fn + suffix, sub(task.issue), base, sub(task.test_patch), sub(task.gold_patch),
                [sub(a) for a in task.alts], flaw=task.flaw, family=task.family, created=task.created,
                module=task.module, test_path=task.test_path)


def date_split(results, tasks, cutoff):
    """Resolved counts before and after a date cutoff (tasks carry a made-up 'created' day, 0 to 199)."""
    by = {t.id: t for t in tasks}
    pre = [r for r in results if by[r.task_id].created < cutoff]
    post = [r for r in results if by[r.task_id].created >= cutoff]
    return (sum(r.resolved for r in pre), len(pre)), (sum(r.resolved for r in post), len(post))


# ---------------------------------------------------------------- Part 10: SWE-smith style task synthesis
INJECT_OPS = [
    ("lt_to_le", r"(?<![<>=!-])<(?!=)", "<="), ("le_to_lt", r"<=", "<"), ("gt_to_ge", r"(?<![-<>=!])>(?!=)", ">="), ("ge_to_gt", r">=", ">"),
    ("plus_to_minus", r" \+ ", " - "), ("minus_to_plus", r" - ", " + "), ("eq_to_ne", r"==", "!="),
    ("and_to_or", r"\band\b", "or"), ("or_to_and", r"\bor\b", "and"),
    ("startswith_to_endswith", r"\.startswith\(", ".endswith("), ("drop_strip", r"\.strip\(\)", ""), ("drop_lower", r"\.lower\(\)", ""),
    ("drop_abs", r"abs\(([^()]*)\)", r"\1"), ("drop_plus_one", r" \+ 1\b", ""), ("zero_to_one", r"\b0\b", "1"),
    ("one_to_two", r"\b1\b", "2"), ("two_to_three", r"\b2\b", "3"), ("drop_not", r"\bnot ", ""), ("in_to_not_in", r" in ", " not in "),
    ("extend_to_append", r"\.extend\(", ".append("), ("isalnum_to_isalpha", r"\.isalnum\(", ".isalpha("),
    ("drop_sorted", r"sorted\((\w+)\)", r"\1"), ("floor_to_true_div", r" // ", " / "), ("mod_to_floor", r" % ", " // "),
]


def synth_repo(name):
    """A repo whose tests all pass: the narrow tasks' fixed code (with helpers) plus all basic and regression tests."""
    r = REPOS[name]
    fns = [s["fn"] for s in SPECS if s["repo"] == name]
    src = r["src"]
    for s in SPECS:
        if s["repo"] == name and s.get("fixed_extra"):
            src = _subs(src, s["fixed_extra"])
    files = repo_files(name, fns)
    body = files[r["test"]] + "".join("\n\ndef test_%s_regress():\n    %s\n" % (f, REGRESS[f]) for f in fns)
    files[r["module"]] = src
    files[r["test"]] = body
    return files


def repro_from_test(files, module, test_src, test_name):
    """Turn the first failing assertion of a test into a doctest-style issue snippet (a stand-in for the
    language-model step that SWE-smith uses to write an issue from a failing test)."""
    try:
        tree = ast.parse(test_src)
    except SyntaxError:
        return None
    fn = next((n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == test_name), None)
    if fn is None:
        return None
    setup = []
    for st in fn.body:
        if isinstance(st, ast.Assert) and isinstance(st.test, ast.Compare) and len(st.test.ops) == 1 \
                and isinstance(st.test.ops[0], (ast.Eq, ast.Is)):
            left, right = ast.unparse(st.test.left), ast.unparse(st.test.comparators[0])
            got = eval_snippet(files, module, "\n".join(setup), left)
            try:
                same = got == repr(eval(right, {}))
            except Exception:
                same = False
            if not same:
                return ("\n".join(setup), left, right, got)
        elif not isinstance(st, ast.Assert):
            setup.append(ast.unparse(st))
    return None


def synthesize_tasks(limit=40, seed="synth", repos=None):
    """Inject one bug at a time into each function of each repo, keep a mutant only if at least one existing test
    fails (and none times out). Returns (tasks, stats)."""
    stats = Counter()
    pool = []
    for name in (repos or sorted(REPOS)):
        files = synth_repo(name)
        r = REPOS[name]
        base_res = run_tests(files)
        assert all(v == "PASS" for v in base_res.values()), "synthetic base repo must pass its own tests"
        src = files[r["module"]]
        for uname, (a, b, text) in code_units(r["module"], src).items():
            for op, pat, rep in INJECT_OPS:
                for m in re.finditer(pat, text, re.M):
                    new_unit = text[:m.start()] + m.expand(rep) + text[m.end():]
                    stats["candidates"] += 1
                    if new_unit == text:
                        stats["no_change"] += 1
                        continue
                    try:
                        ast.parse("class _X:\n" + "\n".join("    " + l for l in new_unit.split("\n")) if "." in uname else new_unit)
                    except SyntaxError:
                        stats["syntax_error"] += 1
                        continue
                    lines = src.split("\n")
                    mutated = "\n".join(lines[:a - 1] + new_unit.split("\n") + lines[b:])
                    mf = dict(files)
                    mf[r["module"]] = mutated
                    res = run_tests(mf, step_limit=4000)
                    if any(v == "TIMEOUT" for v in res.values()):
                        stats["timeout"] += 1
                        continue
                    failing = sorted(t for t, v in res.items() if v != "PASS")
                    if not failing:
                        stats["no_test_fails"] += 1
                        continue
                    stats["kept"] += 1
                    pool.append((name, uname, op, m.start(), mutated, failing, mf))
    rng = rng_for(seed, "pick")
    rng.shuffle(pool)
    tasks = []
    for name, uname, op, pos, mutated, failing, mf in pool:
        if len(tasks) >= limit:
            break
        r = REPOS[name]
        fn_short = uname.split(".")[-1]
        tname = failing[0].split("::")[1]
        rep = repro_from_test(mf, r["module"], mf[r["test"]], tname)
        title = "%s returns a wrong result" % (uname + "()")
        if rep:
            lines = [title, "", "A test that used to pass now fails: %s." % tname, ""]
            lines += [">>> " + l for l in (rep[0].split("\n") if rep[0] else [])] + [">>> " + rep[1], rep[3], "Expected: " + rep[2]]
            issue = "\n".join(lines)
        else:
            issue = "%s\n\nA test that used to pass now fails: %s." % (title, tname)
        fixed = synth_repo(name)
        gold = make_diff({r["module"]: mutated}, {r["module"]: fixed[r["module"]]})
        tid = "synth-%s-%s-%s-%d" % (name, fn_short, op, pos)
        t = Task(tid, name, fn_short, issue, mf, "", gold, [gold], flaw=None, family=op,
                 created=rng_for("day", tid).randrange(200), module=r["module"], test_path=r["test"])
        if not validate_task(t)["ok"]:
            stats["failed_validation"] += 1
            continue
        tasks.append(t)
    stats["tasks"] = len(tasks)
    return tasks, dict(stats)


# ---------------------------------------------------------------- Part 11: study and demo
def make_memorizer(seen_tasks):
    fb = ScriptedAgent("memorizer-fallback", STD_OPS, k_units=3, checks="repro", tier="mid")
    return remember_issues(MemorizerAgent("memorizer", list(seen_tasks), fb), seen_tasks)


def contamination_study(pool, cutoff=100):
    """Everything created before `cutoff` is in the memorizer's training data. Compare it with an honest solver
    on three diagnostics: the date split, the issue-only file-path probe and the prefix probe, then show how
    resolved rate moves when the seen tasks are reworded and renamed."""
    seen = [t for t in pool if t.created < cutoff]
    unseen = [t for t in pool if t.created >= cutoff]
    mem = make_memorizer(seen)
    honest = ScriptedAgent("repro-std", STD_OPS, k_units=3, checks="repro")
    paths = sorted({t.module for t in pool})
    seen_issues = {t.issue for t in seen}
    out = {"n_seen": len(seen), "n_unseen": len(unseen), "cutoff": cutoff,
           "dup_issues": sum(1 for t in unseen if t.issue in seen_issues)}
    for ag in (mem, honest):
        recs = run_benchmark([ag], pool)[ag.name]
        pre, post = date_split(recs, pool, cutoff)
        out[ag.name] = {
            "pre": pre, "post": post,
            "path_seen": file_path_probe(ag, seen, paths), "path_unseen": file_path_probe(ag, unseen, paths),
            "prefix_seen": prefix_probe(ag, seen), "prefix_unseen": prefix_probe(ag, unseen),
        }
        levels = {}
        for lvl, fn in (("original", lambda t: t), ("reworded", reword_task), ("renamed", rename_task)):
            ts = [fn(t) for t in seen]
            r = run_benchmark([ag], ts)[ag.name]
            levels[lvl] = (sum(x.resolved for x in r), len(r))
        out[ag.name]["levels"] = levels
    return out


def _fmt_ci(k, n):
    lo, hi = wilson(k, n)
    return "%d/%d (%.0f%%, 95%% CI %.0f to %.0f%%)" % (k, n, 100 * k / n if n else 0, 100 * lo, 100 * hi)


def demo():
    tasks = curated_tasks()
    print("== 1. Task table (20 hand-written toy tasks; flaw column is planted by me)")
    print("%-12s %-7s %-10s %4s %4s %5s" % ("task", "flaw", "family", "F2P", "P2P", "gold+-"))
    for t in tasks:
        st = diff_stats(t.gold_patch)
        print("%-12s %-7s %-10s %4d %4d %3d/%-2d" % (t.id, t.flaw or "-", t.family, len(t.f2p), len(t.p2p), st["added"], st["removed"]))
    bad = [validate_task(t) for t in tasks if not validate_task(t)["ok"]]
    print("validation: %d of %d tasks pass the Stage III style checks" % (len(tasks) - len(bad), len(tasks)))

    print("\n== 2. Patch validation on one task (%s)" % tasks[3].id)
    t = tasks[3]
    good = t.gold_patch
    miscount = re.sub(r"^(@@ -\d+,)(\d+)", lambda m: m.group(1) + str(int(m.group(2)) + 1), good, count=1, flags=re.M)
    wrong_ctx = good.replace("def is_palindrome", "def is_palindrom")
    tests_edit = make_diff({t.test_path: t.base[t.test_path]}, {t.test_path: t.base[t.test_path] + "\ndef test_is_palindrome_regress():\n    assert True\n"})
    cases = [("gold patch", good, False), ("empty patch", "", False), ("hunk count off by one", miscount, False),
             ("same, with count repair", miscount, True), ("context does not match", wrong_ctx, False),
             ("only edits the test file", tests_edit, False), ("prose, no diff", "I fixed it, trust me.\n", False)]
    for label, p, fx in cases:
        print("  %-26s -> %r" % (label, evaluate(t, p, fix_counts=fx)))

    print("\n== 3. Leaderboard, 20 curated tasks (cost is made up: dollars from token counts times a toy price)")
    seen0 = [t for t in tasks if t.created < 100]
    res = run_benchmark(roster(seen0), tasks)
    print("   (the memorizer has seen the %d tasks created before day 100)" % len(seen0))
    print("%-20s %-28s %9s %9s %5s" % ("agent", "resolved (Wilson CI)", "cost/task", "$/solved", "pareto"))
    for r in leaderboard(res):
        cps = "n/a" if r["cost_per_resolved"] == float("inf") else "%.3f" % r["cost_per_resolved"]
        print("%-20s %-28s %9.4f %9s %5s" % (r["agent"], _fmt_ci(r["resolved"], r["n"]), r["mean_cost"], cps, "yes" if r["pareto"] else ""))
    st = {a: status_counts(rs) for a, rs in res.items()}
    print("status counts (full-sloppy-format):", st["full-sloppy-format"])
    res_fix = run_benchmark([a for a in roster(seen0) if a.name == "full-sloppy-format"], tasks, fix_counts=True)
    print("same agent when the harness repairs hunk counts: %s" % _fmt_ci(sum(r.resolved for r in res_fix["full-sloppy-format"]), len(tasks)))
    print("BM25 localization, top-1 %.0f%%, top-3 %.0f%%" % (100 * localization_accuracy(tasks, 1), 100 * localization_accuracy(tasks, 3)))

    print("\n== 4. Task-quality filters (Verified-style)")
    dec = verified_filter(tasks)
    for t in tasks:
        if not dec[t.id]["kept"]:
            print("  drop %-12s planted=%-6s %s" % (t.id, t.flaw or "-", "; ".join(dec[t.id]["reasons"])))
    fs = filter_scores(tasks, dec)
    print("  vs planted flaws: caught %d, missed %d, wrongly dropped %d" % (fs["tp"], fs["fn"], fs["fp"]))
    kept = [t for t in tasks if dec[t.id]["kept"]]
    res_k = {a: [r for r in rs if r.task_id in {t.id for t in kept}] for a, rs in res.items()}
    print("  leaderboard on the %d kept tasks (resolved before -> after):" % len(kept))
    for r in leaderboard(res_k):
        before = sum(x.resolved for x in res[r["agent"]])
        print("    %-20s %2d/%d -> %2d/%d" % (r["agent"], before, len(tasks), r["resolved"], r["n"]))

    lite_keep = [t for t in tasks if not lite_filter(t)]
    print("  Lite-style shape filter keeps %d of %d, including %d of the %d planted flaws" % (
        len(lite_keep), len(tasks), sum(1 for t in lite_keep if t.flaw), sum(1 for t in tasks if t.flaw)))

    print("\n== 5. Synthesizing tasks (SWE-smith style)")
    synth, ss = synthesize_tasks(limit=40)
    print("  ", ss)
    res_s = run_benchmark(roster(), synth)  # memorizer sees none of these
    print("   resolved on %d synthetic tasks:" % len(synth), ", ".join("%s %d" % (a, sum(r.resolved for r in rs)) for a, rs in res_s.items()))

    print("\n== 6. Contamination diagnostics (memorizer is built to recall; made-up cutoff day 100)")
    pool = tasks + synth
    cs = contamination_study(pool)
    print("   pool %d tasks: %d before the cutoff (seen), %d after" % (len(pool), cs["n_seen"], cs["n_unseen"]))
    print("   %d unseen task(s) have issue text identical to a seen one (same symptom, different bug)" % cs["dup_issues"])
    for name in ("memorizer", "repro-std"):
        c = cs[name]
        print("   %-10s resolved pre %s | post %s" % (name, _fmt_ci(*c["pre"]), _fmt_ci(*c["post"])))
        print("   %-10s path-from-issue seen %d/%d, unseen %d/%d | prefix completion seen %d/%d, unseen %d/%d" % (
            "", c["path_seen"][0], c["path_seen"][1], c["path_unseen"][0], c["path_unseen"][1],
            c["prefix_seen"][0], c["prefix_seen"][1], c["prefix_unseen"][0], c["prefix_unseen"][1]))
        print("   %-10s on seen tasks: original %d/%d, reworded %d/%d, renamed %d/%d" % ("", *c["levels"]["original"], *c["levels"]["reworded"], *c["levels"]["renamed"]))

    print("\n== 7. Result cache keyed without the patch")
    t = tasks[3]
    for mode in ("content", "id_only"):
        cache = EvalCache(mode)
        a = cache.evaluate("r1", t, "", step_limit=20000)
        b = cache.evaluate("r1", t, t.gold_patch, step_limit=20000)
        print("   key=%-8s first (empty patch): %s, second (gold patch): %s, hits %d" % (mode, a.status, b.status, cache.hits))
if __name__ == "__main__":
    demo()

Run the demo

$ python swe_bench_toolkit.py
== 1. Task table (20 hand-written toy tasks; flaw column is planted by me)
task         flaw    family      F2P  P2P gold+-
textkit-1    vague   wrap          1    4   1/1 
textkit-2    narrow  boundary      1    4   5/2 
textkit-3    -       method        1    4   1/1 
textkit-4    -       method        1    4   1/1 
listkit-1    wide    const         2    4   2/2 
listkit-2    narrow  method        1    4   6/2 
listkit-3    -       guard         1    4   2/0 
listkit-4    -       const         1    4   1/1 
mathkit-1    narrow  minmax        1    4   6/1 
mathkit-2    -       wrap          1    4   1/1 
mathkit-3    vague   guard         1    4   2/0 
mathkit-4    -       wrap          1    4   1/1 
cartkit-1    -       augassign     1    4   1/1 
cartkit-2    wide    const         2    4   2/2 
cartkit-3    -       boolop        1    4   1/1 
cartkit-4    -       operands      1    4   1/1 
configkit-1  -       multi         1    4   2/2 
configkit-2  -       multi         1    4   2/0 
configkit-3  -       multi         1    4   4/1 
configkit-4  -       wrap          1    4   1/1 
validation: 20 of 20 tasks pass the Stage III style checks

== 2. Patch validation on one task (textkit-4)
  gold patch                 -> EvalResult(resolved, F2P 1/1, P2P 4/4)
  empty patch                -> EvalResult(empty, F2P 0/0, P2P 0/0)
  hunk count off by one      -> EvalResult(invalid_patch, F2P 0/0, P2P 0/0, hunk header counts 6,5 but body has 5,5 lines)
  same, with count repair    -> EvalResult(resolved, F2P 1/1, P2P 4/4)
  context does not match     -> EvalResult(apply_fail, F2P 0/0, P2P 0/0, context does not match in textkit/text.py near line 14)
  only edits the test file   -> EvalResult(unresolved, F2P 0/1, P2P 4/4)
  prose, no diff             -> EvalResult(invalid_patch, F2P 0/0, P2P 0/0, no file patches found)

== 3. Leaderboard, 20 curated tasks (cost is made up: dollars from token counts times a toy price)
   (the memorizer has seen the 6 tasks created before day 100)
agent                resolved (Wilson CI)         cost/task  $/solved pareto
oracle-localizer     14/20 (70%, 95% CI 48 to 85%)    0.1650     0.236   yes
full-repertoire      14/20 (70%, 95% CI 48 to 85%)    0.2643     0.378      
memorizer            11/20 (55%, 95% CI 34 to 74%)    0.0271     0.049   yes
repro-std            10/20 (50%, 95% CI 30 to 70%)    0.0417     0.083      
repro-tests-std      10/20 (50%, 95% CI 30 to 70%)    0.0550     0.110      
full-sloppy-format   6/20 (30%, 95% CI 15 to 52%)    0.2643     0.881      
naive-first          5/20 (25%, 95% CI 11 to 47%)    0.0009     0.003   yes
repro-basic          2/20 (10%, 95% CI 3 to 30%)     0.0021     0.021      
null                 0/20 (0%, 95% CI 0 to 16%)      0.0002       n/a   yes
status counts (full-sloppy-format): {'invalid_patch': 9, 'unresolved': 5, 'resolved': 6}
same agent when the harness repairs hunk counts: 14/20 (70%, 95% CI 48 to 85%)
BM25 localization, top-1 90%, top-3 100%

== 4. Task-quality filters (Verified-style)
  drop textkit-1    planted=vague  underspecified (max severity 3); tests use names the issue never mentions: slugify
  drop textkit-2    planted=narrow a valid alternative fix fails the tests
  drop listkit-1    planted=wide   tests use names the issue never mentions: window_sums; a valid alternative fix fails the tests
  drop listkit-2    planted=narrow tests use names the issue never mentions: _is_seq; a valid alternative fix fails the tests
  drop mathkit-1    planted=narrow tests use names the issue never mentions: _extent; a valid alternative fix fails the tests
  drop mathkit-3    planted=vague  underspecified (max severity 3); tests use names the issue never mentions: mean
  drop cartkit-2    planted=wide   tests use names the issue never mentions: discount_amount; a valid alternative fix fails the tests
  vs planted flaws: caught 7, missed 0, wrongly dropped 0
  leaderboard on the 13 kept tasks (resolved before -> after):
    oracle-localizer     14/20 -> 13/13
    full-repertoire      14/20 -> 13/13
    memorizer            11/20 -> 10/13
    repro-std            10/20 ->  9/13
    repro-tests-std      10/20 ->  9/13
    full-sloppy-format    6/20 ->  6/13
    naive-first           5/20 ->  4/13
    repro-basic           2/20 ->  2/13
    null                  0/20 ->  0/13
  Lite-style shape filter keeps 20 of 20, including 7 of the 7 planted flaws

== 5. Synthesizing tasks (SWE-smith style)
   {'candidates': 65, 'kept': 43, 'syntax_error': 9, 'no_test_fails': 13, 'failed_validation': 1, 'tasks': 40}
   resolved on 40 synthetic tasks: null 0, naive-first 12, repro-basic 4, repro-std 19, repro-tests-std 19, full-repertoire 21, full-sloppy-format 12, oracle-localizer 21, memorizer 19

== 6. Contamination diagnostics (memorizer is built to recall; made-up cutoff day 100)
   pool 60 tasks: 27 before the cutoff (seen), 33 after
   3 unseen task(s) have issue text identical to a seen one (same symptom, different bug)
   memorizer  resolved pre 27/27 (100%, 95% CI 88 to 100%) | post 14/33 (42%, 95% CI 27 to 59%)
              path-from-issue seen 27/27, unseen 9/33 | prefix completion seen 27/27, unseen 2/33
              on seen tasks: original 27/27, reworded 27/27, renamed 15/27
   repro-std  resolved pre 15/27 (56%, 95% CI 37 to 72%) | post 14/33 (42%, 95% CI 27 to 59%)
              path-from-issue seen 8/27, unseen 7/33 | prefix completion seen 0/27, unseen 0/33
              on seen tasks: original 15/27, reworded 15/27, renamed 15/27

== 7. Result cache keyed without the patch
   key=content  first (empty patch): empty, second (gold patch): resolved, hits 0
   key=id_only  first (empty patch): empty, second (gold patch): empty, hits 1

Exercises

1. Break a patch on purpose. Take the gold patch of one task and (a) change one hunk count, (b) change one context line, (c) shift the whole hunk by five lines. Which statuses do you get with and without fix_counts, and how far does the offset search in apply_patch rescue a shift? Compare with the repair step described in the paper's Appendix A.4 [1].
2. Add a task. Write a new spec (a bug, an issue text, a flaw label), build the task and check that F2P is non-empty and the gold patch breaks no passing test. Then make the issue vague on purpose and see whether issue_severity and the annotators catch it.
3. Evade the static check. Write a narrow-test task whose tests import no name that the issue omits, so that unmentioned_symbols misses it. Does alt_solution_fails still catch it? Now write one that evades both. What would a human reviewer need to see [8]?
4. Make the Lite filter work harder. Combine lite_filter with the Verified-style checks. Compute precision and recall with filter_scores. What does the shape filter give you that the quality filter does not, and the other way round [15] [7]?
5. A cache bug. Run the benchmark with an EvalCache(key_mode="id_only") and the same run id for two different agents. How many results are wrong? Then fix it by changing the key, and write the test that would have caught it [13].
6. A step limit sweep. Write a patch that adds an infinite loop. What status does it get? Then run the leaderboard with step_limit at 500, 2000 and 20000. Do any agent's results change? The project site has a "resolved vs step limit" view [14].
7. A cost cap. Use resolved_rate(recs, cap=...) to count a task as unresolved if its cost is over a cap. Plot the rate against the cap for three agents. Where do the lines cross? How does this relate to SWE-agent's $4 budget [4]?
8. Localization. Run the agent with localizer="bm25", "random" and "oracle". Use localization_accuracy at k = 1 and 3. How much of the score gap is localization, and how does that compare with the paper's oracle finding [1]?
9. A better memorizer, and a better probe. Change the memorizer so that it recalls by the issue text instead of the buggy code. Before you run it, predict what the reword and rename probes will show. Then design a probe that would tell the two memorizers apart [10].
10. Does synthetic rank like curated? Run the nine agents on the 40 synthesized tasks and compare their ranking with the curated one (demo sections 3 and 5). Add one mutation operator to the generator and measure how the yield changes. With nine agents, how much should you trust a rank correlation [6]?
11. A paired comparison. Use paired_gap to compare repro-std and repro-tests-std. Both resolve 10 of 20. Is the gap interval centred on zero because they are the same, or because there are only 20 tasks?

The tests: test_swe_bench_toolkit.py

The 151 tests check the shared helpers (the Wilson interval on a known value, the bootstrap interval pinned for a fixed seed, the order independence of the random generator); the unified diff (parse and apply round trips, the hunk header forms with and without counts, files that are created or deleted, no newline markers, a hunk that is shifted, pure-insertion hunks after a large earlier hunk, the repair of wrong counts, and every kind of malformed patch); the runner (imports, relative imports, a step limit, a test that raises, and a module that is missing); the tasks (each of the 20 must derive non-empty FAIL_TO_PASS and PASS_TO_PASS lists and break nothing with its gold patch); the grading rule and its statuses; the result cache; the agents (BM25 scores on a small case, the edit repertoire, the token counts, the sloppy agent and the oracle localizer); the scorer, the leaderboard and the cost frontier; the filters (the severity rule, the annotator slips, the static and dynamic checks, and the Lite rules); the contamination probes and the memorizer; and the task synthesis. There is also a test that the demo runs and that neither it nor the source has a long dash. The names are listed here, and the code is in the download.

Mutation check. This was a light, sampled run, not an exhaustive one (the toolkit has about 711 places where such a change can be made, and I sampled 70 and then 110 of them at random, with a fixed seed). The first sample of 70 was caught 51 times. For the 19 misses I added tests (the diff parser and applier edge cases, the BM25 scores, the token counts, the step-limit threshold, the bootstrap interval, the annotator slips and the duplicate-issue check). The second sample of 110 was caught 94 times. I then re-ran the 35 survivors listed from the two samples against the final tests, each at every place on its line where the change applies. That gave 49 distinct changes: 39 were caught and 10 survived. Two of the survivors, the sign of the offset for a pure-insertion hunk after an earlier hunk and the percentile index of the bootstrap interval, were real gaps, and I added tests that pin both. The 10 that remain I judge to be harmless or equivalent: a word inside a toy issue text, a constant inside a toy task's code string, the step limit of 20000 against 20001, the guard that stops a hunk body at a file header (it matters only for a deleted line that itself begins with two dashes and a space, and no test patch has one), a clamp at 1.0 against 1.1 on a Wilson bound that never exceeds one, the bootstrap upper index with n_boot plus one inside a min() that already caps it, three off-by-one changes in the module-path loop (the parents are loaded either way), and a fifth candidate unit for the full-repertoire agent, which no toy task needs. That makes 151 tests in all. I did not run the whole mutation sample again after the last two tests were added, apart from the changes that they were written for.

$ python -m unittest -v test_swe_bench_toolkit
test_agent_gives_up_without_a_passing_candidate (test_swe_bench_toolkit.TestAgents.test_agent_gives_up_without_a_passing_candidate) ... ok
test_agents_are_deterministic (test_swe_bench_toolkit.TestAgents.test_agents_are_deterministic) ... ok
test_bm25_localization_accuracy (test_swe_bench_toolkit.TestAgents.test_bm25_localization_accuracy) ... ok
test_bm25_prefers_matching_doc (test_swe_bench_toolkit.TestAgents.test_bm25_prefers_matching_doc) ... ok
test_bm25_scores_are_pinned (test_swe_bench_toolkit.TestAgents.test_bm25_scores_are_pinned) ... ok
test_candidates_each_family (test_swe_bench_toolkit.TestAgents.test_candidates_each_family) ... ok
test_code_units (test_swe_bench_toolkit.TestAgents.test_code_units) ... ok
test_default_agent_reads_three_units (test_swe_bench_toolkit.TestAgents.test_default_agent_reads_three_units) ... ok
test_fingerprint_ignores_tests_and_issue (test_swe_bench_toolkit.TestAgents.test_fingerprint_ignores_tests_and_issue) ... ok
test_guard_candidate (test_swe_bench_toolkit.TestAgents.test_guard_candidate) ... ok
test_max_cands_caps_tries (test_swe_bench_toolkit.TestAgents.test_max_cands_caps_tries) ... ok
test_max_cands_counts_candidates_actually_tried (test_swe_bench_toolkit.TestAgents.test_max_cands_counts_candidates_actually_tried) ... ok
test_memorizer_recalls_only_seen_source (test_swe_bench_toolkit.TestAgents.test_memorizer_recalls_only_seen_source) ... ok
test_null_agent (test_swe_bench_toolkit.TestAgents.test_null_agent) ... ok
test_oracle_localizer_uses_hint (test_swe_bench_toolkit.TestAgents.test_oracle_localizer_uses_hint) ... ok
test_parse_repro_edge_cases (test_swe_bench_toolkit.TestAgents.test_parse_repro_edge_cases) ... ok
test_parse_repro_matches_specs (test_swe_bench_toolkit.TestAgents.test_parse_repro_matches_specs) ... ok
test_repro_check_picks_the_right_edit (test_swe_bench_toolkit.TestAgents.test_repro_check_picks_the_right_edit) ... ok
test_roster (test_swe_bench_toolkit.TestAgents.test_roster) ... ok
test_sloppy_agent_breaks_headers_and_repair_recovers (test_swe_bench_toolkit.TestAgents.test_sloppy_agent_breaks_headers_and_repair_recovers) ... ok
test_sloppy_header_gets_exactly_one_extra_old_line (test_swe_bench_toolkit.TestAgents.test_sloppy_header_gets_exactly_one_extra_old_line) ... ok
test_template_candidate (test_swe_bench_toolkit.TestAgents.test_template_candidate) ... ok
test_tests_check_rejects_edit_that_breaks_visible_test (test_swe_bench_toolkit.TestAgents.test_tests_check_rejects_edit_that_breaks_visible_test) ... ok
test_token_accounting_is_pinned (test_swe_bench_toolkit.TestAgents.test_token_accounting_is_pinned) ... ok
test_demo_runs_and_has_no_dashes (test_swe_bench_toolkit.TestDemo.test_demo_runs_and_has_no_dashes) ... ok
test_source_has_no_dashes (test_swe_bench_toolkit.TestDemo.test_source_has_no_dashes) ... ok
test_blank_lines_survive (test_swe_bench_toolkit.TestDiff.test_blank_lines_survive) ... ok
test_context_mismatch (test_swe_bench_toolkit.TestDiff.test_context_mismatch) ... ok
test_delete_file (test_swe_bench_toolkit.TestDiff.test_delete_file) ... ok
test_diff_stats (test_swe_bench_toolkit.TestDiff.test_diff_stats) ... ok
test_failed_patch_changes_nothing (test_swe_bench_toolkit.TestDiff.test_failed_patch_changes_nothing) ... ok
test_file_patch_flags (test_swe_bench_toolkit.TestDiff.test_file_patch_flags) ... ok
test_fix_hunk_counts_handles_two_hunks (test_swe_bench_toolkit.TestDiff.test_fix_hunk_counts_handles_two_hunks) ... ok
test_fix_hunk_counts_leaves_good_patch_alone (test_swe_bench_toolkit.TestDiff.test_fix_hunk_counts_leaves_good_patch_alone) ... ok
test_git_header_lines_are_skipped (test_swe_bench_toolkit.TestDiff.test_git_header_lines_are_skipped) ... ok
test_header_error_messages (test_swe_bench_toolkit.TestDiff.test_header_error_messages) ... ok
test_hunk_applies_at_offset (test_swe_bench_toolkit.TestDiff.test_hunk_applies_at_offset) ... ok
test_hunk_old_new_lines (test_swe_bench_toolkit.TestDiff.test_hunk_old_new_lines) ... ok
test_identical_trees_give_empty_diff (test_swe_bench_toolkit.TestDiff.test_identical_trees_give_empty_diff) ... ok
test_insert_at_top_and_empty_old_side (test_swe_bench_toolkit.TestDiff.test_insert_at_top_and_empty_old_side) ... ok
test_lenient_parse_and_count_repair (test_swe_bench_toolkit.TestDiff.test_lenient_parse_and_count_repair) ... ok
test_lenient_parse_stops_at_the_next_file_header (test_swe_bench_toolkit.TestDiff.test_lenient_parse_stops_at_the_next_file_header) ... ok
test_missing_file (test_swe_bench_toolkit.TestDiff.test_missing_file) ... ok
test_new_file_must_not_exist (test_swe_bench_toolkit.TestDiff.test_new_file_must_not_exist) ... ok
test_no_newline_marker_ignored (test_swe_bench_toolkit.TestDiff.test_no_newline_marker_ignored) ... ok
test_offset_limit (test_swe_bench_toolkit.TestDiff.test_offset_limit) ... ok
test_parse_errors (test_swe_bench_toolkit.TestDiff.test_parse_errors) ... ok
test_pure_insertion_hunks_follow_earlier_size_changes (test_swe_bench_toolkit.TestDiff.test_pure_insertion_hunks_follow_earlier_size_changes) ... ok
test_round_trip_two_files (test_swe_bench_toolkit.TestDiff.test_round_trip_two_files) ... ok
test_second_hunk_is_placed_after_a_large_first_hunk (test_swe_bench_toolkit.TestDiff.test_second_hunk_is_placed_after_a_large_first_hunk) ... ok
test_single_prose_line_before_header_is_skipped (test_swe_bench_toolkit.TestDiff.test_single_prose_line_before_header_is_skipped) ... ok
test_strict_counts_reject_long_and_short_headers (test_swe_bench_toolkit.TestDiff.test_strict_counts_reject_long_and_short_headers) ... ok
test_annotators_take_max_and_stay_in_range (test_swe_bench_toolkit.TestFilters.test_annotators_take_max_and_stay_in_range) ... ok
test_default_annotator_noise_is_pinned (test_swe_bench_toolkit.TestFilters.test_default_annotator_noise_is_pinned) ... ok
test_filter_raises_resolved_rate_of_full_agent (test_swe_bench_toolkit.TestFilters.test_filter_raises_resolved_rate_of_full_agent) ... ok
test_filter_score_arithmetic (test_swe_bench_toolkit.TestFilters.test_filter_score_arithmetic) ... ok
test_lite_filter_keeps_the_curated_tasks_including_the_flawed_ones (test_swe_bench_toolkit.TestFilters.test_lite_filter_keeps_the_curated_tasks_including_the_flawed_ones) ... ok
test_lite_filter_rules (test_swe_bench_toolkit.TestFilters.test_lite_filter_rules) ... ok
test_sev_cut_changes_outcome (test_swe_bench_toolkit.TestFilters.test_sev_cut_changes_outcome) ... ok
test_severity (test_swe_bench_toolkit.TestFilters.test_severity) ... ok
test_static_check_misses_the_constant_but_the_dynamic_check_catches_it (test_swe_bench_toolkit.TestFilters.test_static_check_misses_the_constant_but_the_dynamic_check_catches_it) ... ok
test_unmentioned_symbols (test_swe_bench_toolkit.TestFilters.test_unmentioned_symbols) ... ok
test_unmentioned_symbols_with_top_level_import (test_swe_bench_toolkit.TestFilters.test_unmentioned_symbols_with_top_level_import) ... ok
test_verified_filter_matches_planted_flaws (test_swe_bench_toolkit.TestFilters.test_verified_filter_matches_planted_flaws) ... ok
test_cache_modes (test_swe_bench_toolkit.TestHarness.test_cache_modes) ... ok
test_editing_tests_does_not_help (test_swe_bench_toolkit.TestHarness.test_editing_tests_does_not_help) ... ok
test_eval_result_defaults (test_swe_bench_toolkit.TestHarness.test_eval_result_defaults) ... ok
test_infinite_loop_patch_times_out (test_swe_bench_toolkit.TestHarness.test_infinite_loop_patch_times_out) ... ok
test_invalid_versus_apply_fail (test_swe_bench_toolkit.TestHarness.test_invalid_versus_apply_fail) ... ok
test_miscounted_header_needs_repair (test_swe_bench_toolkit.TestHarness.test_miscounted_header_needs_repair) ... ok
test_new_test_file_is_deleted_before_grading (test_swe_bench_toolkit.TestHarness.test_new_test_file_is_deleted_before_grading) ... ok
test_repr (test_swe_bench_toolkit.TestHarness.test_repr) ... ok
test_resolution_needs_every_f2p_test (test_swe_bench_toolkit.TestHarness.test_resolution_needs_every_f2p_test) ... ok
test_resolution_needs_every_p2p_test (test_swe_bench_toolkit.TestHarness.test_resolution_needs_every_p2p_test) ... ok
test_whitespace_only_patch_is_empty (test_swe_bench_toolkit.TestHarness.test_whitespace_only_patch_is_empty) ... ok
test_bootstrap_ci (test_swe_bench_toolkit.TestHelpers.test_bootstrap_ci) ... ok
test_content_tokens_drop_stopwords_and_single_letters (test_swe_bench_toolkit.TestHelpers.test_content_tokens_drop_stopwords_and_single_letters) ... ok
test_est_tokens (test_swe_bench_toolkit.TestHelpers.test_est_tokens) ... ok
test_jaccard (test_swe_bench_toolkit.TestHelpers.test_jaccard) ... ok
test_mean (test_swe_bench_toolkit.TestHelpers.test_mean) ... ok
test_rng_for_is_order_independent (test_swe_bench_toolkit.TestHelpers.test_rng_for_is_order_independent) ... ok
test_tokens_split_camel_and_snake (test_swe_bench_toolkit.TestHelpers.test_tokens_split_camel_and_snake) ... ok
test_wilson (test_swe_bench_toolkit.TestHelpers.test_wilson) ... ok
test_date_split_counts (test_swe_bench_toolkit.TestProbes.test_date_split_counts) ... ok
test_file_path_and_prefix_probes (test_swe_bench_toolkit.TestProbes.test_file_path_and_prefix_probes) ... ok
test_guess_path (test_swe_bench_toolkit.TestProbes.test_guess_path) ... ok
test_prefix_probe_with_one_added_line_uses_empty_prefix (test_swe_bench_toolkit.TestProbes.test_prefix_probe_with_one_added_line_uses_empty_prefix) ... ok
test_rename_does_not_change_an_honest_agent (test_swe_bench_toolkit.TestProbes.test_rename_does_not_change_an_honest_agent) ... ok
test_rename_task_stays_valid_and_changes_source (test_swe_bench_toolkit.TestProbes.test_rename_task_stays_valid_and_changes_source) ... ok
test_reword_keeps_examples (test_swe_bench_toolkit.TestProbes.test_reword_keeps_examples) ... ok
test_study_counts_duplicate_issues (test_swe_bench_toolkit.TestProbes.test_study_counts_duplicate_issues) ... ok
test_study_date_split_shows_the_gap (test_swe_bench_toolkit.TestProbes.test_study_date_split_shows_the_gap) ... ok
test_study_perturbation_levels (test_swe_bench_toolkit.TestProbes.test_study_perturbation_levels) ... ok
test_study_probes_separate_memorizer_from_honest (test_swe_bench_toolkit.TestProbes.test_study_probes_separate_memorizer_from_honest) ... ok
test_eval_snippet (test_swe_bench_toolkit.TestRunner.test_eval_snippet) ... ok
test_from_package_import_submodule (test_swe_bench_toolkit.TestRunner.test_from_package_import_submodule) ... ok
test_helpers (test_swe_bench_toolkit.TestRunner.test_helpers) ... ok
test_import_error_marks_all_tests_error (test_swe_bench_toolkit.TestRunner.test_import_error_marks_all_tests_error) ... ok
test_missing_name_is_error (test_swe_bench_toolkit.TestRunner.test_missing_name_is_error) ... ok
test_module_level_loop_times_out (test_swe_bench_toolkit.TestRunner.test_module_level_loop_times_out) ... ok
test_only_filter (test_swe_bench_toolkit.TestRunner.test_only_filter) ... ok
test_open_is_not_available_to_package_code (test_swe_bench_toolkit.TestRunner.test_open_is_not_available_to_package_code) ... ok
test_package_init_runs_and_star_import_works (test_swe_bench_toolkit.TestRunner.test_package_init_runs_and_star_import_works) ... ok
test_state_does_not_leak_between_runs (test_swe_bench_toolkit.TestRunner.test_state_does_not_leak_between_runs) ... ok
test_statuses (test_swe_bench_toolkit.TestRunner.test_statuses) ... ok
test_step_limit_threshold_is_pinned (test_swe_bench_toolkit.TestRunner.test_step_limit_threshold_is_pinned) ... ok
test_syntax_error_in_test_file (test_swe_bench_toolkit.TestRunner.test_syntax_error_in_test_file) ... ok
test_trace_function_is_restored (test_swe_bench_toolkit.TestRunner.test_trace_function_is_restored) ... ok
test_whitelisted_stdlib_import_works (test_swe_bench_toolkit.TestRunner.test_whitelisted_stdlib_import_works) ... ok
test_cap_boundary_is_inclusive (test_swe_bench_toolkit.TestScoring.test_cap_boundary_is_inclusive) ... ok
test_cost_tiers (test_swe_bench_toolkit.TestScoring.test_cost_tiers) ... ok
test_expected_ranking_facts (test_swe_bench_toolkit.TestScoring.test_expected_ranking_facts) ... ok
test_fix_counts_recovers_sloppy_agent (test_swe_bench_toolkit.TestScoring.test_fix_counts_recovers_sloppy_agent) ... ok
test_leaderboard_cap_lowers_rates (test_swe_bench_toolkit.TestScoring.test_leaderboard_cap_lowers_rates) ... ok
test_leaderboard_cost_arithmetic (test_swe_bench_toolkit.TestScoring.test_leaderboard_cost_arithmetic) ... ok
test_leaderboard_order_and_fields (test_swe_bench_toolkit.TestScoring.test_leaderboard_order_and_fields) ... ok
test_no_agent_resolves_the_narrow_or_wide_tasks (test_swe_bench_toolkit.TestScoring.test_no_agent_resolves_the_narrow_or_wide_tasks) ... ok
test_null_scores_zero_and_cost_positive (test_swe_bench_toolkit.TestScoring.test_null_scores_zero_and_cost_positive) ... ok
test_paired_gap (test_swe_bench_toolkit.TestScoring.test_paired_gap) ... ok
test_pareto_flags (test_swe_bench_toolkit.TestScoring.test_pareto_flags) ... ok
test_pareto_on_hand_data (test_swe_bench_toolkit.TestScoring.test_pareto_on_hand_data) ... ok
test_resolved_rate_and_cap (test_swe_bench_toolkit.TestScoring.test_resolved_rate_and_cap) ... ok
test_shapes (test_swe_bench_toolkit.TestScoring.test_shapes) ... ok
test_status_counts (test_swe_bench_toolkit.TestScoring.test_status_counts) ... ok
test_agents_solve_some_but_not_all (test_swe_bench_toolkit.TestSynthesis.test_agents_solve_some_but_not_all) ... ok
test_deterministic_and_seeded (test_swe_bench_toolkit.TestSynthesis.test_deterministic_and_seeded) ... ok
test_every_synthetic_task_is_valid_and_solvable_by_gold (test_swe_bench_toolkit.TestSynthesis.test_every_synthetic_task_is_valid_and_solvable_by_gold) ... ok
test_gold_is_the_reverse_of_a_single_edit (test_swe_bench_toolkit.TestSynthesis.test_gold_is_the_reverse_of_a_single_edit) ... ok
test_injection_operators_compile_on_some_site (test_swe_bench_toolkit.TestSynthesis.test_injection_operators_compile_on_some_site) ... ok
test_issue_has_repro_from_failing_test (test_swe_bench_toolkit.TestSynthesis.test_issue_has_repro_from_failing_test) ... ok
test_no_synthetic_task_needs_a_test_patch_and_none_vague (test_swe_bench_toolkit.TestSynthesis.test_no_synthetic_task_needs_a_test_patch_and_none_vague) ... ok
test_repo_filter_and_limit (test_swe_bench_toolkit.TestSynthesis.test_repo_filter_and_limit) ... ok
test_repro_from_test_directly (test_swe_bench_toolkit.TestSynthesis.test_repro_from_test_directly) ... ok
test_stats_add_up (test_swe_bench_toolkit.TestSynthesis.test_stats_add_up) ... ok
test_synthetic_repos_pass_their_own_tests (test_swe_bench_toolkit.TestSynthesis.test_synthetic_repos_pass_their_own_tests) ... ok
test_alt_patches_follow_planted_flaws (test_swe_bench_toolkit.TestTasks.test_alt_patches_follow_planted_flaws) ... ok
test_bug_substitutions_apply_exactly_once (test_swe_bench_toolkit.TestTasks.test_bug_substitutions_apply_exactly_once) ... ok
test_counts_and_ids (test_swe_bench_toolkit.TestTasks.test_counts_and_ids) ... ok
test_created_days_are_stable (test_swe_bench_toolkit.TestTasks.test_created_days_are_stable) ... ok
test_curated_tasks_are_cached (test_swe_bench_toolkit.TestTasks.test_curated_tasks_are_cached) ... ok
test_every_task_validates (test_swe_bench_toolkit.TestTasks.test_every_task_validates) ... ok
test_f2p_fail_on_base_and_pass_with_gold (test_swe_bench_toolkit.TestTasks.test_f2p_fail_on_base_and_pass_with_gold) ... ok
test_f2p_p2p_shape (test_swe_bench_toolkit.TestTasks.test_f2p_p2p_shape) ... ok
test_gold_patch_touches_no_tests (test_swe_bench_toolkit.TestTasks.test_gold_patch_touches_no_tests) ... ok
test_gold_resolves_and_empty_does_not (test_swe_bench_toolkit.TestTasks.test_gold_resolves_and_empty_does_not) ... ok
test_narrow_tests_import_names_the_issue_never_mentions (test_swe_bench_toolkit.TestTasks.test_narrow_tests_import_names_the_issue_never_mentions) ... ok
test_repo_files_layout (test_swe_bench_toolkit.TestTasks.test_repo_files_layout) ... ok
test_validate_flags_broken_tasks (test_swe_bench_toolkit.TestTasks.test_validate_flags_broken_tasks) ... ok
test_validate_flags_gold_that_breaks_a_passing_test (test_swe_bench_toolkit.TestTasks.test_validate_flags_gold_that_breaks_a_passing_test) ... ok
test_view_hides_test_patch_and_gold (test_swe_bench_toolkit.TestTasks.test_view_hides_test_patch_and_gold) ... ok
test_wide_tasks_need_two_fixes (test_swe_bench_toolkit.TestTasks.test_wide_tasks_need_two_fixes) ... ok

----------------------------------------------------------------------
Ran 151 tests in 4.176s

OK

5. Discussion questions

Use these in a study group or a class. Try to answer with an example and a source. Give each about 5 to 10 minutes.

1. The grading rule needs every FAIL_TO_PASS and PASS_TO_PASS test to pass. Name a case where a correct patch fails the rule and a case where a wrong patch passes it. Which one worries you more for a leaderboard [1]?
2. The authors drop tasks whose tests call a function that the real patch adds. The 2026 audit of Verified still found tests that import a helper the issue never names. What does that say about how well a filter written in advance can work, and what review step would you add [1] [8]?
3. Verified takes the highest of three annotators' severity labels. What does that choice do to the share of tasks that are kept, and what would you lose by taking the average instead [7]?
4. OpenAI says its Verified filter is "likely to be overzealous". Is it a problem that a benchmark throws away many good tasks to be sure about the rest? What would it mean for the tasks that remain [7]?
5. In the lab, a Lite-style shape filter keeps every planted flaw. What does a shape filter buy you, and what does the Lite page itself say that it is for [15]?
6. The same company that helped to make Verified later wrote that it no longer evaluates it, and later retracted its recommendation of Pro. How would you read each of these three posts differently from a paper by an outside group, and what would you check yourself [7] [8] [9]?
7. SWE-agent's edit command runs a linter, and an iterative search tool that returned too much made the agent worse than no search tool. What does this say about measuring a model on its own and measuring a model with its tools [4]?
8. A leaderboard fixes the scaffold (Bash Only) and another one lets each team use its own. Which question does each answer, and which would you want for choosing a model for your product [14] [16]?
9. SWE-smith only keeps a synthetic bug if it breaks an existing test. What kinds of real-world bug does that rule leave out, and what happens to an agent trained only on tasks of this kind [6]?
10. The Illusion paper finds 76% file-path accuracy on Verified and 53% outside SWE-bench. List three explanations other than memorization, and a test that would separate each from memorization [10].
11. The Live paper reports 43.2% on Verified and 19.25% on Live for one agent and model, and also finds no clear link between a task's recency and its success. Is that evidence for or against contamination as the cause of the gap? What else could explain it [11]?
12. Pro keeps a held-out set and a commercial set private. What does a private set protect, what does it cost the community, and who can then check the numbers [12]?
13. Several sources here are by the benchmark's own creators, and two instructors of this course are authors of some of them. How should a student weigh a creator's account of the benchmark's strengths, and what independent evidence is in this pack [5] [6] [10]?

6. Quiz (10 questions)

Pick one answer for each question, then press Check answers. Check answers 0 of 10 answered.

1. Under the rule in the SWE-bench paper's Appendix A.4, when is a task instance counted as resolved [1]?
   • When the patch changes the same files as the real pull request did and the model's explanation of the fix is judged correct
   • When the text of the patch is close enough to the real patch under a similarity score that the authors set
   • When the new test that the real pull request added passes, even if some older tests start to fail
   • When the patch applies and every test in both the FAIL_TO_PASS and PASS_TO_PASS lists has a passing status
2. What is the job of the PASS_TO_PASS tests in a task [1]?
   • They are tests that passed before and after the real fix, and they catch a patch that breaks other working code
   • They are the tests that the model is shown, so that it knows how its change will be judged before it submits
   • They are the tests that fail on the base commit, and they are the only ones that decide whether the issue is fixed
   • They are extra tests that the benchmark authors wrote by hand to replace tests that were flaky in the repository
3. According to the harness documentation, what happens if you run the same instance twice under one run id, with a different patch the second time [13]?
   • The second patch is graded fresh, because the harness rebuilds the instance image from scratch for every run it is given
   • The harness refuses to run, and prints an error that says that the run id is already taken and must be changed first
   • The cached result is returned, because results are keyed by the run and the instance and not by the patch
   • Both results are averaged, so the instance receives a score that is a mix of the two patches and is neither of them
4. A task's tests import a helper function that the real patch added, and the issue never names it. In the terms of the Verified audits, which problem is this [8]?
   • An underspecified issue, since the text should always give the name of every function that the fix needs to add
   • A wide test, since the test is checking more behavior than the issue asked for in its description of the problem
   • A narrow test, since it enforces a detail of the original implementation that a valid alternative fix would not have
   • An environment failure, since the helper can only be imported if the Docker image has been built with the right layers
5. Which statement about SWE-bench Lite and SWE-bench Verified matches their sources [15] [7]?
   • Both were chosen by human annotators who rated every issue for how well it was specified before the sample was drawn
   • Both keep only tasks that were written after the training cutoff of the model so that no task can have been seen
   • Lite is the larger set, since it adds new issues each month, while Verified is a small set that is never updated
   • Lite is chosen by shape rules (for example one file, at most three hunks) and Verified by human annotation
6. What does the SWE-agent edit command do that editing a file through plain bash commands does not [4]?
   • It runs a linter on the change and shows the agent the errors with a snippet, so a syntax mistake is caught at once
   • It lets the agent submit a patch without any tests being run, which saves cost by skipping the grading step on hard tasks
   • It hides the contents of the file from the agent, so that the agent can only change the lines that it has been told to
   • It asks a second model to review every change, and it rejects the edit unless that model agrees that it is correct
7. When SWE-smith creates a task by injecting a bug into a working repository, what must be true for the candidate to be kept [6]?
   • It must have been taken from a real merged pull request, because only those have an issue text that a person has written
   • It must change more than one file, because single-file bugs are too easy for agents to be useful as training data
   • It must break at least one existing test, so that there is a failing test to be fixed and a way to grade the repair
   • It must be checked by hand by an annotator, and each repository needs a new Docker image for each of its bugs
8. The Illusion paper finds 76% file-path accuracy on Verified issues and up to 53% on repositories outside SWE-bench. What is the fair reading [10]?
   • It proves that the models were trained on the benchmark, and so every score on Verified should be thrown away
   • It is a clue that points to possible memorization, though some of the gap may come from differences in the tasks
   • It shows that the models understand repositories, because the number is high even when the repository is not shown
   • It shows nothing at all, because the file path can be guessed from the issue text by a model that has no knowledge
9. In the lab, the memorizer's recall survives rewording the issue but not renaming the functions. What does that show [10]?
   • That my memorizer is keyed on the code, so a rename probe can tell code recall apart from skill, in the toy only
   • That real models are keyed on the issue wording, so a rewording probe is the most reliable test of memorization
   • That renaming functions is the same as writing a new task, so a renamed task can never be used to test a model again
   • That an honest scripted agent is also helped by renaming, which proves that the lab's agents are all memorizing
10. Which pair correctly describes how SWE-bench-Live and SWE-Bench Pro each try to limit contamination [11] [12]?
    • Live asks the model to promise not to use old data, and Pro removes every task that a model solved in an earlier test
    • Live and Pro both use a random sample of the original 2,294 tasks, and they differ only in how many tasks they keep
    • Live rewrites the text of old issues with a language model, and Pro hides the test patch from the evaluation harness
    • Live uses issues since 2024 with automated environments, and Pro uses copyleft public repositories plus private sets

Answer key

7. Graded assignment and rubric

Title: Build a small benchmark in a new domain, audit it, and report it honestly.
Time: about 3 hours. Total: 100 points. Work: solo.

The task

Extend the lab code and write a short memo (two pages at most). Everything is a toy: no real model, no real repository you do not own, no Docker and no network. Pick your own domain, such as a unit converter, a date parser or a small inventory library, and do not reuse the lab's five packages or its 20 tasks. Label every made-up number as made up, and every calculation of your own as your own. Say what a reader must not conclude from your numbers. Do not run any patch that you did not write, because the runner is not a sandbox.

Part A. Tasks (30 points)

• Write one small package with its own tests, and at least 8 tasks that each have an issue text, a bug, a gold patch and a test patch. Check that the FAIL_TO_PASS and PASS_TO_PASS lists come out as you expect, and that the gold patch breaks nothing.
• Plant at least 3 flaws: one underspecified issue, one narrow test and one wide test. Say in the memo how each one is built.
• Add one task for which a valid alternative fix exists, and show that your tests accept it.

Part B. Agents and scoring (25 points)

• Write at least 4 scripted agents with different repertoires or checks, and one null agent. Do not make them "smart": make them differ in a way that you can explain.
• Report the resolved rate of each with a 95% Wilson interval, the number of tasks, the cost per task from a price list that you make up, and the cost per solved task.
• Compare two agents with paired_gap or your own paired bootstrap. Say whether the data can tell them apart, and what number of tasks would be needed to do so (show your arithmetic).

Part C. Audit (25 points)

• Run a Verified-style filter and a Lite-style filter on your tasks. Report which of your planted flaws each one finds, with precision and recall against your labels.
• Design one flaw that your static check misses and show whether the dynamic alternative-fix check catches it. Design one that both miss, and say what a human reviewer would need.
• Report the leaderboard before and after the filter. Say what changed and what did not, and whether the new board has a ceiling.

Part D. Memorization and the memo (20 points)

• Add a memorizer and run a date split and a rename probe. Report the numbers with intervals, and say in one sentence why they are true by construction.
• Write a memo of two pages at most that answers: what does your benchmark measure, which sources in this pack does each design choice follow, what are three ways it could mislead a reader, and what would you need to see before you believed a real-world claim based on a benchmark like it. Name every vendor or author overlap in the sources you cite.
• State three limits of your tasks and three limits of your memo.

What to hand in

• Your swe_bench_toolkit.py (or your own file names), your package and tasks, and your tests. All tests must pass.
• A memo in PDF or Markdown with the tables from Parts A to C and the memo from Part D.
• A sources list for any reading you cite.

Rubric

Criterion	Full marks	Partial	Points
A. Tasks	A new package and 8 or more valid tasks, derived FAIL_TO_PASS and PASS_TO_PASS lists, three planted flaws explained, and one alternative fix that the tests accept.	Fewer tasks, flaws that are not explained, or lists that are written by hand and not derived.	30
B. Agents and scoring	Four agents and a null agent that differ for a stated reason, rates with Wilson intervals, cost per task and per solved task, and a paired comparison with a sample-size argument.	Rates without intervals, no cost, or a comparison of intervals that ignores that the tasks are shared.	25
C. Audit	Both filters with precision and recall, a flaw that the static check misses and the dynamic check finds, a flaw that both miss with the human step named, and a before and after leaderboard with a comment on the ceiling.	One filter only, no miss shown, or no before and after comparison.	25
D. Memorization and memo	Date split and rename probes with intervals and the by-construction caveat, a memo that answers all four questions, names the overlaps, and states three limits of each kind.	Probes without intervals, a memo that repeats the lab, or missing limits.	20
Total			100

Grade bands: 90 to 100 excellent. 75 to 89 good. 60 to 74 needs work. Under 60: redo and resubmit. Plain writing is rewarded: short sentences, no filler. Label your own arithmetic as your own.

8. Glossary

Term	Meaning
ACI (agent-computer interface)	The commands and feedback through which an agent uses a computer. SWE-agent's design has a linter, a windowed viewer and capped search [4] (2.8).
Base commit	The version of the repository just before the real fix. The model gets the repository at this state [1] (2.2).
BM25	A classic ranking function for text search. The original paper uses it to pick files, and the lab uses it to pick code units [1] (2.4).
Bootstrap interval	An interval from resampling the tasks many times. The lab uses it for differences between agents (2.11).
Cache level	The harness setting that decides which Docker images stay on disk between runs: none, base, env or instance [13] (2.5).
Contamination	When a benchmark's tasks or answers were in a model's training data, so a score may reflect recall (2.10).
Docker image	A packaged environment (operating system, language, dependencies) that the harness runs a task in. The harness has base, environment and instance layers [13] (2.5).
Execution-based filtering	Keeping a candidate task only if running its tests before and after the fix shows a fail-to-pass test, and the code installs and runs [1] (2.3).
FAIL_TO_PASS	The tests that fail before the real fix and pass after it. They define what the fix must do [1] (2.2).
Gold patch	The part of the real pull request that is not test files. It is the reference solution and is never shown to the model [1] (2.3).
Harness	The program that applies a patch in a clean environment, runs the tests and reports the result. Part of the benchmark's definition (2.5).
Hunk	One block of changes in a unified diff, with a header that gives line numbers and counts (2.2).
Instance	One task: issue, repository state, gold patch, test patch, test lists (2.3).
Lite	A 300-task subset of SWE-bench chosen by shape rules for cheaper evaluation [15] (2.6).
Memorizer	In the lab, an agent built to recall the fix of tasks it has "seen". Its score on those tasks is true by construction (2.10).
Narrow test	A test that rejects a valid fix because it demands a detail of the original implementation [8] (2.7).
Oracle retrieval	Showing the model the files that the real patch edited. Used in the paper for analysis, not as a fair task [1] (2.4).
Pareto frontier	The set of agents for which no other agent is both cheaper and at least as good. Used for cost and rate (2.11).
PASS_TO_PASS	The tests that pass both before and after the real fix. They guard against breaking other behavior [1] (2.2).
Resolved	The patch applies and every FAIL_TO_PASS and PASS_TO_PASS test passes [1] (2.2).
Scaffold	The program around a model that runs commands, shows files and manages the loop. A score belongs to model and scaffold together (2.8).
Test patch	The part of the real pull request that changes test files. It is applied before grading [1] (2.3).
Underspecified issue	An issue text that does not give enough information to know what a correct fix is [7] (2.7).
Unified diff	The standard text format for a patch: file headers, hunks, context lines, and added and removed lines (2.2).
Verified	A 500-task subset of SWE-bench that human annotators checked, made with OpenAI in 2024 [7] (2.7).
Wide test	A test that checks behavior that the issue never asked for [8] (2.7).
Wilson interval	An interval for a rate from counts that behaves well for small samples and rates near 0 or 1 (2.11).

9. References

Entries marked (required) are the course's required reading. Entries marked (added) are my own additions. The course site and the slide folder are listed without a mark. Each entry says how much of the source I read. A source by the benchmark's own authors, or by a company that uses the benchmark, is a claim by an interested party, and I say so where it applies.

• [1] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. arXiv:2310.06770 (submitted 10 October 2023; version 3, 11 November 2024). The first and third authors are the lecture's speakers, and John Yang is an instructor of the course. https://arxiv.org/abs/2310.06770 (I read the abstract, the main text and Appendix A on the arXiv HTML page, and skimmed the later appendices; the authors' own claims about their own benchmark) (required)
• [2] Stanford CS329Z: Engineering AI Agents, course site and schedule (Diyi Yang, Michael Ryan, John Yang). https://cs329z.stanford.edu/ (the course's own page, read on 29 October 2026; it lists "Guest Lecture: SWE-bench" on Monday, November 16, with Carlos Jimenez and Alexander Wettig, a one-sentence description and one reading, and lists "Agent Safety & Guardrails" on Wednesday, November 18)
• [3] CS329Z slide folder and the addresses lecture16.pdf and lecture17.pdf, each tried once on 29 October 2026 at 8:36 PM Pacific time. https://web.stanford.edu/class/cs329z/slides/lecture17.pdf (the two addresses returned "not found" (HTTP 404), and the folder index listed lecture01.pdf to lecture11.pdf only, so I did not see the slides)
• [4] Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS 2024, as the course lists it. arXiv:2405.15793 (submitted 6 May 2024; version 3, 11 November 2024). John Yang, an instructor of the course, is the first author, and the lecture's speakers are co-authors. https://arxiv.org/abs/2405.15793 (I read the abstract and the sections that I cite, not every section; the authors' own claims. The course lists this paper as a reading for a later lecture) (added)
• [5] Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J., Wu, X., Press, Ori, Muennighoff, N., Synnaeve, G., Narasimhan, K. R., Yang, D., Wang, S. I., and Press, Ofir. "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?" arXiv:2410.03859 (4 October 2024). John Yang and Diyi Yang, both instructors of the course, are authors, and Jimenez, one of the lecture's speakers, is a co-author. https://arxiv.org/abs/2410.03859 (I read the abstract and the sections that I cite, not every section; the authors' own claims. The abstract says 617 task instances, the body says 619 and the project site lists 480) (added)
• [6] Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., Khandpur, K., Zhang, Y., Hui, B., Press, O., Schmidt, L., and Yang, D. "SWE-smith: Scaling Data for Software Engineering Agents." arXiv:2504.21798 (30 April 2025; version 2, 21 May 2025). John Yang and Diyi Yang, both instructors of the course, are authors, and Jimenez and Wettig are co-authors. https://arxiv.org/abs/2504.21798 (I read the abstract and the sections that I cite, not every section; the authors' own claims, including the cost figures) (added)
• [7] OpenAI. "Introducing SWE-bench Verified." Company blog post, 13 August 2024, updated 24 February 2025. Vendor-authored; the author list includes C. E. Jimenez and J. Yang of the SWE-bench team. https://openai.com/index/introducing-swe-bench-verified/ (I read the whole post; it describes the company's own annotation work with the SWE-bench authors and the company's own scaffold runs) (added)
• [8] OpenAI. "Why we no longer evaluate SWE-bench Verified" (the page is headed "Why SWE-bench Verified no longer measures frontier coding capabilities"). Company blog post, 23 February 2026. Vendor-authored. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ (I read the whole post; it reports the company's own audit and red-team probing, and I did not repeat either) (added)
• [9] OpenAI. "Separating signal from noise in coding evaluations." Company blog post, 8 July 2026. Vendor-authored. https://openai.com/index/separating-signal-from-noise-coding-evaluations/ (I read the post; it reports the company's own audit of SWE-Bench Pro, which the company had recommended, and I did not re-check any task) (added)
• [10] Liang, S., Garg, S., and Zilouchian Moghaddam, R. "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason." arXiv:2506.12286 (submitted 14 June 2025; version 4, 1 December 2025). None of its authors is an author of SWE-bench. https://arxiv.org/abs/2506.12286 (I read the abstract and the sections that I cite, not every section; the authors' own claims and probes, with no access to the training data of the models) (added)
• [11] Zhang, L., He, S., Zhang, C., Kang, Y., Li, B., Xie, C., Wang, J., Wang, M., Huang, Y., Fu, S., Nallipogu, E., Lin, Q., Dang, Y., Rajmohan, S., and Zhang, D. "SWE-bench Goes Live!" arXiv:2505.23419 (submitted 29 May 2025; version 2, 1 June 2025). None of its authors is an author of SWE-bench. https://arxiv.org/abs/2505.23419 (I read the abstract and the sections that I cite, not every section; the authors' own claims and runs) (added)
• [12] Deng, X., Da, J., Pan, E., He, Y. Y., Ide, C., Garg, K., Lauffer, N., Park, A., Pasari, N., Rane, C., Sampath, K., Krishnan, M., Kundurthy, S., Hendryx, S., Wang, Z., Bharadwaj, V., Holm, J., Aluri, R., Zhang, C. B., Jacobson, N., Liu, B., and Kenstler, B. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" arXiv:2509.16941 (submitted 21 September 2025; version 2, 14 November 2025). None of its authors is an author of SWE-bench. https://arxiv.org/abs/2509.16941 (I read the abstract and the sections that I cite, not every section; the authors' own claims. Reference [9] reports an outside audit of this benchmark) (added)
• [13] SWE-bench documentation. "The Harness" (evaluation harness reference). https://www.swebench.com/SWE-bench/reference/harness/ (I read the page and did not run the harness; the project's own documentation) (added)
• [14] SWE-bench project site (swebench.com): home page, benchmark list, leaderboard views and news list. The site thanks Open Philanthropy, AWS, Modal, Andreessen Horowitz, OpenAI and Anthropic for support. https://www.swebench.com/ (I read the home page, including its news list, on 29 October 2026; the project's own page, which can change) (added)
• [15] Jimenez, C. E., Yang, J., and Geng, J. "SWE-bench Lite" (project page). https://www.swebench.com/lite.html (I read the whole page; the project's own description of its selection criteria) (added)
• [16] mini-SWE-agent (SWE-agent project), repository README on GitHub. https://github.com/SWE-agent/mini-swe-agent (I read the top of the README, not the code, and did not run it; the project's own claims, including a score above 74% on Verified that I did not check) (added)