TwinEdgeAI

↑ Course home

← Previous: Week 7a, Data Selection & Quality  ·  Next: Week 8a, LLM-as-Judge & Evaluation Infrastructure →

Engineering AI Agents · Week 7b

Evaluation Fundamentals & Benchmark Design

Why is it hard to measure an agent? A task as a request, an environment, stopping criteria and a scorer; how to design each one; what makes a benchmark good; why the scaffold matters; and how to report a score that you can trust.

Author: Grok Bot Version: 1.0, October 29, 2026 Study time: about 15 hours

The slides were not availableThis lecture is on Wednesday, November 4, and I wrote this pack on Thursday, October 29, 2026. I checked the course's slide folder once. Its index listed lecture01.pdf to lecture11.pdf, and the address I expect for this lecture, lecture13.pdf, returned "not found" (HTTP 404) [5]. So I have not seen the slides. This pack is built from the schedule's description of the lecture, the required reading, the two additional readings, and sources I found myself. The real lecture may stress other things. Where I guess what the lecture covers, I say so. Check the slides when they appear.

CreditThis coursework is adapted from, and inspired by, Stanford CS329Z: Engineering AI Agents (Diyi Yang, Michael Ryan, John Yang), Week 7, Lecture "Evaluation for Agentic Systems: Evaluation Fundamentals & Benchmark Design" (Wednesday, November 4). The topic list and the readings come from the course's public schedule [4], which says that it is tentative. The schedule describes the lecture as: why evals are hard, the 4-tuple framework (request, environment, stopping criteria, scorer), designing each component, properties of good benchmarks, realistic scaffolding, and reliability dimensions. The notes, lab, quiz and assignment are original writing, and the lab uses its own examples and code. This pack does not reproduce any slides. It is not an official Stanford document. Each source I added is marked "added" in the references.

1. Week overview and learning objectives

Week 7a asked which data is worth keeping. This week asks how you know that an agent is any good. The schedule lists six topics [4]. I read them as three jobs. The first is defining a task so that it can be run and scored: the 4-tuple and how to design each part. The second is checking the benchmark itself: validity, the properties of a good benchmark, and a realistic scaffold. The third is reporting a score you can defend: reliability across trials, error bars, comparisons, cost, and what happens as a benchmark saturates. The required reading, by Zhu and colleagues, is about the second job. It gives a checklist for finding the ways that an agent benchmark can give a score that does not mean what it seems to mean [1]. The two additional readings are Press's advice on building a benchmark [2], which is where the 4-tuple comes from, and tinyBenchmarks, on evaluating with fewer examples [3].

The lab uses the standard library only. It does not call any model, any API or the network. It builds a small help-desk world with tickets, tools, and 64 tasks. It runs scripted "agents" on it, and these are not models. Each has a skill knob and a price knob. The lab then plants flaws in the benchmark on purpose, finds them with an auditor that is in the spirit of the Agentic Benchmark Checklist, and measures how much they matter. It also covers outcome and process scores, pass@k and pass^k, error bars with clustered standard errors, scaffold and limit sensitivity, a cost and accuracy view, saturation, and a small demonstration of Goodhart's law. Everything it prints is from made-up worlds. It shows how the methods work. It says nothing about how real benchmarks or real agents behave. In several places the toy gives a weak or even opposite result to a paper, and I report that.

Learning objectives

After this week you should be able to:

1. Explain why evaluating an agent is harder than scoring a model's answer, with at least four reasons. [2.1]
2. Write any benchmark task as a request, an environment, stopping criteria and a scorer, and say what each part must fix. [2.2]
3. Choose a scorer type (exact, programmatic, state-based) for a task, and choose limits and say what a run that hits a limit scores. [2.3]
4. Define task validity and outcome validity, and name a flaw that breaks each one, with a trivial agent that would exploit it. [2.4]
5. List the properties that Press, BetterBench and Kapoor and colleagues ask of a good benchmark, and say where they disagree. [2.5]
6. Say why the baseline scaffold has to be realistic, and read a cost and accuracy plot. [2.6]
7. Compute pass@k and pass^k from n trials, a standard error with and without clusters, and the difference that a benchmark can detect. [2.7]
8. Tell whether a benchmark is saturated or has headroom, given an estimate of how many items are broken. [2.8]
9. Explain, with the lab's numbers, how tuning to a flawed benchmark lowers the true score. [2.8]
10. Build and test a small eval harness and a benchmark auditor, and use them on made-up data. [Lab]

How to use this pack

Time	What to do
3.5 hours	Read the notes (section 2). Skim the figures first.
3.5 hours	Read the required reading (Zhu and colleagues) with the reading guide questions, and skim Press and tinyBenchmarks. Skim the optional ones that match your project.
4.5 hours	Do the lab and at least five exercises.
1.5 hours	Discussion and the quiz.
2 hours	The graded assignment.

2. Lecture notes

The slides were not available (see the top of the page), so these notes follow the schedule's six topics [4] and the readings. The section titles are mine.

2.1 Why agent evals are hard

To score a model on a multiple-choice test you compare one letter with one key. To score an agent you let it act for many steps in a setting, and then you decide if the end of that run counts as success. Zhu and colleagues put the difference in two points. The setup is complex, because the agent gets an environment and tools and not just a prompt. The outcome is unstructured, because it is code, a file edit, a state change or a free-text reply, and checking it needs a purpose-built method [1]. Anthropic's engineering post says that agents use tools across many turns and change state as they go, so mistakes can propagate and compound [21]. I read the whole post (published January 9, 2026). Its authors work at a company that sells models and agents, so I treat it as one practitioner's view.

I group the difficulties into six. The sources in the right column are where I found each one, and the framing is mine.

Difficulty	What goes wrong	Source
The score may not mean the capability	A task can be passed without the skill (a shortcut), or failed even if the skill is there (an impossible or broken task). Zhu and colleagues report overestimates and underestimates of up to 100% in relative terms on benchmarks that they assessed [1].	[1]
The run is random	The same agent on the same task can pass once and fail the next time. One trial per task hides this. τ-bench was built to show it, with a metric for consistency [7].	[7] [21]
The benchmark is a small sample	Agent benchmarks often have a few hundred tasks. The score has sampling error, and tasks that come in groups (the same website or repository) make the error larger than it looks [12].	[6] [12]
The setting around the model matters	The scaffold (prompts, tools, how much output the agent sees, the limits) changes the score. Cost matters too, and accuracy-only leaderboards hide it [6] [16].	[2] [6] [16]
The benchmark ages	Scores rise until there is no headroom. Test items leak into training data. People optimize for the metric [2] [18] [20].	[2] [18] [20]
The environment is software	Websites change, rate limits bite, and a grader that reads an HTML selector breaks. Zhu and colleagues found 13 of 46 Chrome tasks in OSWorld broken for this reason [1].	[1]

None of this says benchmarks are useless. SWE-bench turned a vague wish (an agent that fixes bugs) into a number, and people built better agents against it [8] [2]. The point is that the number comes from many design choices, and each choice can move it.

2.2 A task is a 4-tuple

Press's post ends with a framing that is useful for the rest of the week: a benchmark is a collection of tasks, and each task is made of <request, environment, stopping criteria, scorer> [2]. I take the definitions from the post.

• Request: what you want the model to do. In SWE-bench it is "Fix this issue" plus the issue text [2].
• Environment: a complete description of where the agent acts. Is internet access allowed? Which dependencies are installed? Which special tools does the agent have [2]?
• Stopping criteria: how a run ends. The agent may issue a submit command, but you must decide what happens when it never does. Press lists a turn limit, a cost limit, a wall-time limit, or a mix, and says all are viable if you decide [2].
• Scorer: it takes the environment as it was when the agent exited and scores it. It can be binary, as with SWE-bench's fail-to-pass and pass-to-pass tests. It can be continuous: in AlgoTune the score per task is the agent's code runtime divided by the baseline's. It can be an Elo rating, as in CodeClash [2].

The same ideas appear under other names elsewhere. The Anthropic post uses task for a test with defined inputs and success criteria, trial for each attempt, grader for the scoring logic, transcript for the full record of a trial, outcome for the final state in the environment, and harness for the infrastructure that runs evals [21]. Its example: a booking agent may say "Your flight has been booked", but the outcome is whether a reservation exists in the database [21]. That is the difference between a claim in the transcript and a fact in the state. Here is how I line up the names, including the names in this week's lab.

4-tuple [2]	Anthropic words [21]	In the lab code
Request	task inputs	task["request"], a string
Environment	environment, outcome (the final state)	Env: tickets, documents, hidden gold, and a Scaffold that sets the tools
Stopping criteria	(part of the harness)	Limits and the status of a run: submitted, step, time or cost limit
Scorer	grader	score_outcome and, for the path, process_score
(a run)	trial, transcript	Trace, from run_episode

2.3 Designing each part

The request

Press's first property of a good benchmark is that the task is natural: a real person would ask it, often. SWE-bench is made of real bug reports. He gives two warning signs: the set-up is unrealistic (his example is multiple choice, since no one tells a doctor "my elbow hurts, and it is one of these four options"), and the questions were made up by the benchmark's authors and not taken from real users [2]. A second design point is that the request must say enough. A request that leaves out a fact that the scorer needs makes a failure that is not the agent's fault. This is my own point. The ABC lists a related check, that the prompt names the tool or package versions it needs (T.1) [1]. In τ-bench the request is a conversation with a user simulated by a language model, so the request itself varies from run to run [7].

The environment

The environment holds the state that the agent changes and the tools it uses. The ABC asks for six things here. Tools are specified with versions (T.1), API services are available and rate limits are handled (T.2), and an outage is detected and ends the evaluation so that users know (T.3). Legacy state is cleaned between tasks (T.4). The agent is isolated from ground truth (T.5). The setup is reproducible and frozen at release, not tied to a live website (T.6) [1]. Zhu and colleagues give the cases: KernelBench left ground-truth answers in GPU memory, and SWE-Lancer stored its tests in a password-protected archive whose file listing and contents the agent could still change [1]. WebArena builds fully functional websites in four domains, to be both realistic and reproducible [14]. In the lab, the environment is a dictionary of tickets copied afresh for every run. A hidden file holds the gold answer. In the flawed suite, 10 tasks leave that file readable.

The stopping criteria

A run needs a rule to end. There are two decisions, and the second is easy to forget. The first is which limits to set: steps, seconds, cost. The second is what a run that hits a limit scores. Either you score whatever the agent left behind, or the run scores zero. In the lab I count a submit as a step, and a step that would break a cost or time limit is not carried out. Figure 2 shows how much it matters on the toy suite: with a limit of 2 steps, the planner scores 0.82 if the leftover state is scored and 0.73 if a limit means zero. At limits of 3 and 4 the two policies give 0.90 and 0.82, and at 5 and 6 they give 0.91 and 0.90. The curves meet at 8 steps, where almost no run is cut off. At 1 step the score is 0.09 under the first policy, because some update tasks are already done by their single write (about 6 of the 64 tasks, my count from the lab run), and 0 under the second.

Limits also change comparisons. In the lab, model A is cheap and slips 10% of the time, and model B is careful (4% slips) and costs three times as much per step. With a generous limit B is ahead, 0.965 to 0.910. With a cost limit of 8 units per task, B can afford only two actions and fails the tasks that need more, and A is ahead, 0.910 to 0.867. At a limit of 4, B can afford one action and scores 0.09, against 0.90 for A. At limits of 10, 14 and 20 B is ahead again (0.955, 0.955 and 0.965 against 0.910 for A). I wrote the prices and chose the limits so that this would happen, so it shows that the effect can occur, and not how often. Across the four scaffolds in 2.6, the ranking never flipped, so in this toy the limit mattered more than the scaffold. The lesson is to publish the limits and to check the ranking at more than one.

The scorer

The scorer is where most of the validity problems live (2.4). Here are the main types, with the lab's names.

Scorer	How it works	Typical weak spot
Exact or normalized match	Compare the answer with a key, after lower-casing and trimming. GAIA is listed under answer matching in the survey table of the ABC paper [1].	Equivalent answers that the key does not list (O.a.1), or extra words [1].
Substring match	The answer must contain the key.	Negation ("not positive"), listing every possible answer, and guessing (O.b.1 to O.b.3) [1]. In the lab, "unknown" contains "no".
Programmatic or test-based	Run unit tests, fuzz inputs, or parse a number or a set of ids. SWE-bench uses fail-to-pass and pass-to-pass tests [2].	Tests that are too weak accept a wrong patch. The ABC asks to check the quality of tests (O.d.1, O.d.2) [1].
State-based	Compare the final state with a goal state. τ-bench compares the database at the end of the conversation with the annotated goal [7].	Valid end states missing from the goal (O.g.1), no unrelated state to reveal collateral damage (O.g.2), or a state space so simple that a trivial change passes (O.g.3) [1]. A do-nothing agent that passes is the extreme case.
Continuous score	A number per task, such as a runtime ratio [2].	Metric hacking: a high number without the task being done (O.i.1) [1].
Judge model	A language model grades. Next week's topic.	The judge's accuracy and self-consistency (O.c.1) [1].

Outcome scores and process scores

A scorer that reads only the end ignores how the agent got there. A process score reads the path: did the agent check the facts, did it touch only what it should? The lab has a simple one. It is the share of planned tool calls made in order, halved if the agent wrote to a ticket that the task does not name. On the fixed toy suite, a planner scores 0.93 on outcome and 0.90 on process. A "careless" agent that does every step right and then submits a stale answer 40% of the time scores 0.68 on outcome and 1.00 on process. A "lucky" agent that never uses a tool and submits the most common answer scores 0.25 on outcome and 0.00 on process. So the two scores answer different questions. The outcome says whether the user got what they asked for. The process says whether the agent could be trusted to do it again. The process score has a cost too: it needs a notion of the right path, and a good agent that finds a different path is punished. The Anthropic post recommends grading what the agent produced and not the path it took, because rigid checks on the order of tool calls are brittle and punish valid approaches that the designer did not expect. It also advises partial credit for tasks with several parts [21]. It gives one example of a path that a check would have punished: a model solved a τ²-bench flight-booking problem by finding a loophole in the policy, and so "failed" the eval as written [21]. So the process score in the lab is an extra signal about trust, not a replacement for the outcome. The contrast between outcome and process supervision is also a theme in training: Lightman and colleagues report that process supervision beat outcome supervision on MATH problems, and Uesato and colleagues found that final-answer error was similar with less label supervision, while process supervision was needed for correct reasoning steps [22] [23]. I read only their abstracts, and they are about training signals, not benchmark scorers, so I use them as an analogy only.

2.4 Task validity and outcome validity

The required reading splits the question "is this benchmark sound?" into two. Task validity: the task can be solved if and only if the agent has the target capability. It breaks in two ways. A shortcut lets an agent pass without the skill, and this gives false positives. An impossible task blocks an agent that has the skill, and this gives false negatives. Outcome validity: the scorer's verdict (pass or fail) matches whether the task was really solved [1]. A third part of the checklist, reporting, asks the authors to document what they did and did not fix, and to report statistical significance, interpretation guidelines and baselines (items R.1 to R.13) [1].

Zhu and colleagues built the Agentic Benchmark Checklist (ABC) from their own benchmark-building experience, a survey of best practices and issues reported earlier [1]. They scored ten widely used benchmarks against it. The ten were picked, from a longer list of 17, for popularity and for covering every kind of agent capability, so they are not a random sample [1]. Here are the cases I found most useful, with the checks that the paper says were broken. All numbers are the authors' own, from experiments they designed, and I did not rerun any of them.

Benchmark	What they report	ABC check
τ-bench	Counts trivial states or substrings as ground truth (overestimate by 38%). Also lets an agent list every possible answer (overestimate by 40%). The abstract says it counts empty responses as successful [1].	O.b.2, O.b.3, O.g.3
SWE-bench Verified	Uses insufficient test cases [1] (the abstract's wording). SWE-bench Verified is also what a course instructor helped to make [9].	O.d.1, O.d.2
SWE-Lancer	Does not fully isolate the agent from the ground truth. Its tests sit in a password-protected archive whose listing and contents can be read and overwritten without the password. The authors say an agent can score 100% without solving tasks [1].	T.5
KernelBench	Leaves ground-truth answers in GPU memory; also tests with too few edge cases. Correctness is overestimated by 31% on the kernels that they re-examined [1].	T.4, O.e.1 to O.e.3
OSWorld	In the Chrome section, 13 of 46 problems are broken, because graders rely on HTML selectors and the websites changed. The authors say this underestimates the open-source agent UI-TARS by 28 points (absolute) [1].	T.6
CVE-Bench	Used as the worked example. Applying ABC reduces the performance overestimate by 33% [1].	(several)

The check numbers for SWE-bench Verified, KernelBench's fuzzing and OSWorld are my own matching of the findings to the items; the others are as the paper states them. One caution: the paper is a position and audit paper. The authors found these flaws with tests they designed, and the authors of the audited benchmarks may see them differently. Several authors of the checklist also work on benchmarks and on AI agents, including Kapoor, who is an author of both [1] and [6]. The abstract and the findings I quote match the full text that I read (the HTML version, in most sections). I did not read every appendix.

The lab turns the checklist into an auditor

The lab does not copy the ABC. It implements the part that can be run, which is the idea behind checks T.9 and T.10 (an oracle solver shows the task can be done, and consistent failure points to a bug) [1], plus a set of trivial agents that should always score about zero. If a trivial agent passes a task, the task has a shortcut. The agents are:

Agent	What it does	Related ABC check (my reading)
oracle	Follows the known right plan. It should pass every task. A failure means the task is broken or impossible.	T.9, T.10
empty	Submits an empty answer and does nothing else.	O.g.3 (trivial state), and the empty-response problem in τ-bench
unknown	Submits "unknown" and does nothing else.	O.b.1 (substring match and negation)
dump	Lists every ticket of the owners named in the task and submits all of the text.	O.b.2 (listing all possible answers)
peek	Reads the hidden gold file if it exists. This is a flag only if the file is readable.	T.5 (isolation from ground truth)
vandal	Does the plan without slips, then also closes a ticket that the task never mentions. It passes if the scorer ignores collateral damage.	O.g.2 (irrelevant states)
guess	Submits the most common answer. Not a flag, but a baseline to compare against.	O.h.2 (success by guessing)

The lab builds one benchmark twice from the same 64 tasks: a flawed suite (loose substring and id-set scorers, a readable gold file on 10 tasks, no limits on 13 tasks, and state scorers that ignore collateral changes) and a fixed one (strict scorers, gold hidden, limits on). The auditor flags 46 of 64 tasks in the flawed suite and none in the fixed one. A task can carry more than one flag. The counts by flag, for the flawed suite, are: vandal passes 20, dump passes 16, no limits 13, unknown passes 12, gold visible 10, empty passes 8. The trivial baselines in Figure 3 are my toy numbers: on the flawed suite, empty scores 0.125, unknown 0.188, dump 0.25 and guess 0.375. On the fixed suite they score 0, 0, 0 and 0.25 (guess), and "refuse everything" scores 0.125 because refusal tasks are 8 of 64 and refusing is the right answer there.

Do the flaws change the ranking of good agents? In the toy, no. A planner that slips 4% of the time scores 0.964 on both the flawed and the fixed suite, and one that slips 30% scores 0.728 on both. The flaws raise the score of only the trivial agents: empty by 0.125, unknown by 0.188, dump by 0.25. A planner that also vandalizes an unrelated ticket scores 1.000 on the flawed suite and 0.688 on the fixed one. I built the flaws that way: they are shortcuts, and a skilled agent does not need them. In real benchmarks, a shortcut can help skilled agents too. The ABC findings above are real cases where the overestimate was large [1]. I report the toy result as it came out, because it shows something useful: a validity flaw can be invisible in the score of a good agent and still ruin the benchmark, because it rewards gaming (see 2.8).

2.5 Properties of a good benchmark

Several authors list what a good benchmark needs. They overlap, and I note where they differ. This section is a map, not a ranking.

Source	What it asks for
Press [2]	Three properties: natural (real questions that a group of people ask often), automatically evaluatable, and challenging (the best model should not do well at launch). A bonus property: hard to leak into training data, as when SciCode does not release its answers. Guidelines: one number; strong baselines; a task count of at least 150 and ideally 300 to 500; expect saturation within a year.
Zhu and colleagues [1]	Task validity, outcome validity, and honest reporting (ABC, 2.4).
BetterBench [11]	An assessment framework of 46 best practices across a benchmark's lifecycle, applied to 24 benchmarks. They find large quality differences. They also find that most benchmarks report no statistical significance and cannot easily be replicated. I read the abstract only [11].
Kapoor and colleagues [6]	Report cost as well as accuracy, separate the needs of model developers and downstream developers, use hold-out sets at the right level of generality, and standardize evaluation to make it reproducible.
HELM [10]	A multi-metric approach: 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) for each of 16 core scenarios when possible, so that trade-offs are clear. I read the abstract only [10].

Where they disagree. Press tells you to have one number so that people understand and adopt your benchmark, and to keep other breakdowns for the analysis section [2]. HELM argues that one metric lets other qualities fall by the wayside, and measures seven [10]. Kapoor and colleagues want accuracy and cost [6]. These can be reconciled by audience. If the aim is for a community to rally to a frontier, one headline number works, and the extra metrics belong in the paper. If the aim is to choose a system for use, you need the trade-offs. I would add the point that Kapoor and colleagues make: the two audiences need different benchmarks [6]. This is my reading, not a statement from any of the three.

Another tension is difficulty at launch. Press first said that the top model should score 1% to 35% at launch. In January 2025 he lowered this to 0.1% to 9%, and in May 2025 he asked collaborators for tasks where models would get zero even if they improved threefold [2]. The post itself says it was edited as models improved, so the advice moves with the field. Two benchmarks in these notes started in the older window: WebArena's best agent got 14.41% against 78.24% for humans [14], and GAIA's best reported system, GPT-4 with plugins, got 15% against 92% for humans [15]. SWE-bench's best model got 1.96% at launch, and Press notes that this scared people off until SWE-agent reached about 13% [2] [8]. Press is a co-author of SWE-bench with John Yang, a course instructor [8], so his account of that story is a first-hand one, and also an interested one.

Hold-outs. Kapoor and colleagues argue that the more general an agent is meant to be, the more the held-out set must differ from the data that agents could tune on: in-distribution samples for distribution-specific benchmarks, out-of-distribution samples for task-specific ones, tasks for domain-general ones, and domains for fully general ones. They count benchmarks with an appropriate hold-out: 1 of 1, 3 of 6, 1 of 8 and 0 of 2 at the four levels [6]. Agents are different from models here, they say, because "knowledge of test samples can be directly programmed into the agent" [6], so a lookup table could get a high score. The counts come from their survey of 17 benchmarks. In many cases they had to guess what level of generality the authors meant, which they say is itself a problem [6].

Size. Press asks for at least 150 tasks, and 300 to 500 for statistical resolution. He thinks more than that rarely helps, and he would cap at 500 because of run time [2]. Section 2.9 shows what these sizes mean as error bars: about ±8, ±5.7 and ±4.4 points at 150, 300 and 500 tasks (my arithmetic, for a score near 50%). Whether tasks in the same group count as separate samples matters more than the total (2.7).

2.6 Realistic scaffolding, cost and the Pareto view

Press's second question for a benchmark builder is: what baseline scaffold will you use, and how close is it to the best one in common use? His examples: a coding benchmark whose scaffold cannot run code, or a knowledge benchmark with no internet, is not a fair picture of reality. He says that a good scaffold often takes little effort, and gives mini-SWE-agent as a very simple scaffold that gets competitive scores with Claude Code, "even though it is orders of magnitude simpler" [2]. That is Press's own claim about a tool from his own group, and I did not check it. His reason for caring is about trust: "people will mistrust your results" if the baseline is not realistic [2].

Kapoor and colleagues push the same point from the other side. They show that on HumanEval a few simple baseline agents (retry the model, "warm" the temperature, escalate to a stronger model) can match or beat complex published agent designs at much lower cost. Their conclusion is that accuracy-only leaderboards reward needless complexity, and they argue for plotting accuracy against cost and looking for the Pareto frontier [6]. The Holistic Agent Leaderboard (HAL) takes this further. It offers a standard harness, and the authors ran 21,730 rollouts across 9 models and 9 benchmarks, at a cost of about $40,000 (their figure). Among its findings, higher reasoning effort reduced accuracy in most runs, and log inspection showed agents searching HuggingFace for the benchmark itself, and misusing a credit card in flight-booking tasks [16]. HAL is run by many of the same authors as [6], so it is a case of the builders reporting on their own platform. I read only the abstract of [16].

What does this mean for a task designer? The scaffold is part of the environment and the stopping rule. The same model gets a different score if the scaffold changes what it sees or what it may do. The lab changes four things on one fixed suite: the full scaffold (tools that work, verify-on-write), a bare one (no verification), a truncating one (the agent sees only the first 44 characters of each observation), and one without the search tool. The scores for models A and B are:

Scaffold	Model A (cheap)	Model B (careful)	Order
full	0.910	0.965	B, A
bare (no verify)	0.893	0.951	B, A
truncated observations	0.869	0.922	B, A
no search tool	0.820	0.861	B, A

The scaffold changes the level by up to 0.09 for A and 0.10 for B, and never the order. That is one toy with two models. It does not show that real rankings are stable under scaffold changes. Real work, for instance HAL's, studies this with real models [16]. The only thing that flipped the ranking in the lab was the cost limit (2.3).

Cost and accuracy

If two agents have different prices, a single accuracy number hides the trade. Plot each agent as a point (cost, accuracy). An agent is on the Pareto frontier if no other agent is at least as cheap and at least as accurate, with one of the two strictly better. The lab runs six made-up agents, with prices that I chose. Agents a to d are on the frontier: a costs 1.23 and scores 0.832, b costs 2.43 and scores 0.902, c costs 3.60 and scores 0.930, and d costs 7.16 and scores 0.963. Agent e, at 14.32, scores the same as d, so it is dominated. Agent f, at 9.78, scores 0.885, which is worse than b at a quarter of the price. The units of cost are arbitrary (one unit is one cheap tool call).

There is one more idea, the upper convex hull. If you can randomly pick between two agents, with some chance of each, any point on the line between them is reachable. So a point under that line is dominated by a mix, even if no single agent dominates it. By my arithmetic from the numbers above, if you needed 0.95 accuracy, you could mix c and d: 60.6% of tasks to d and the rest to c gives accuracy 0.95 at a cost of about 5.76 per task, against 7.16 for d alone. This assumes that the mix works as a straight line, and it ignores the cost of deciding which tasks go to which agent. It is a way to think, not a result.

2.7 Reliability: trials, error bars and comparisons

Week 6a (section 2.10) already introduced pass@k, pass^k and the error-bar formulas for reading a leaderboard. This section does the same for a benchmark that you design, and the lab builds the tools.

"The agent scores 91%" hides three kinds of uncertainty, and each needs its own tool. I use "reliability" for all three. This is my grouping, and the schedule does not say how the lecture splits it.

(a) Trial-to-trial: pass@k and pass^k

Run the same task k times. pass@k is the chance that at least one run passes. It suits settings where one success is enough, like code with a test to check it. Codex reached 70.2% on HumanEval with 100 samples per problem, against 28.8% with one [17]. pass^k is the chance that all k runs pass. It suits settings where the user expects the right result every time. τ-bench introduced it, and reports that even the best function-calling agents succeed on fewer than 50% of tasks and are inconsistent: pass^8 is below 25% in retail [7]. If a task has success rate c out of n trials, the unbiased estimates are 1 − C(n−c, k)/C(n, k) for pass@k and C(c, k)/C(n, k) for pass^k, where C is "n choose k" (τ-bench writes out both, and the Codex paper is where pass@k comes from; I wrote both into the lab and tested them) [7] [17].

The lab's planner (slip rate 0.1) has mean accuracy 0.916 on the fixed suite. Over 16 trials per task, pass@k at k = 1, 2, 4, 8 and 16 is 0.916, 0.989, 1.0, 1.0 and 1.0. And pass^k is 0.916, 0.843, 0.726, 0.565 and 0.391. This is the shape that the sources describe: one falls, the other rises. If every task had the same success rate of 0.916 and trials were independent, my arithmetic gives pass^16 = 0.916¹⁶ ≈ 0.246. The toy's 0.391 is higher because tasks differ: many tasks almost never fail, and a few often do. The more uneven the task success rates, the higher pass^k stays for a given mean. The Anthropic post makes the same point with a simpler example (a 75% rate and 3 trials gives 42%) [21].

The lab also splits the variance of a single-trial score into between-task and within-task parts. For the planner it is 0.0045 between and 0.0725 within, so 94% of the variance comes from trial-to-trial noise. The reason is that most tasks pass almost always, so tasks hardly differ from each other, while a few tasks flip between pass and fail. That is a property of this toy at about 90% accuracy. In a benchmark where many tasks are hard for the agent, the between-task part would be larger. A direct check: twelve single-trial runs of the whole suite (seeds 0 to 11) give a mean of 0.930, a standard deviation of 0.031, and a range from 0.891 to 1.000. So a 0.89 and a 1.00 can come from the same agent.

(b) Sampling error: standard errors, clusters and intervals

Miller's article treats an eval as an experiment: the questions are a sample from an unseen super-population, and the score has a standard error [12]. For a pass/fail benchmark with n independent tasks and a mean p, the naive standard error is √(p(1 − p)/n), and the 95% interval is the mean ± 1.96 SE [12]. If tasks come in groups that are correlated, such as tickets of the same owner or questions on the same passage, the naive formula is too small. Miller's clustered standard error adds the covariance between pairs of tasks within each cluster (his Equation 4), and in his real examples the clustered SE can be over 3 times larger [12]. The paper also says that bootstrapping is unnecessary when the sample is large enough for the central limit theorem [12].

In the lab, the 64 tasks form 8 owner clusters. For model A (accuracy 0.924), the naive 95% interval is 0.898 to 0.950, and the clustered one is 0.894 to 0.953, which is about 13% wider. For model B (0.975) the two intervals are the same, 0.961 to 0.988. The clustering effect is small here. I did not tune the toy to make it large, and I did not study why it is small. It can be much larger, as in Miller's real examples, so this toy cannot tell you how much to expect.

(c) Comparing two systems: paired differences and power

If both models run on the same tasks, compare them per task. The variance of the difference falls when the models agree on which tasks are hard [12]. In the lab, A scores 0.924 and B scores 0.975, a paired difference of 0.051. The naive paired standard error is 0.011 (z = 4.6) and the clustered one is 0.010 (z = 4.99). Miller's power analysis asks what difference the benchmark could detect. The lab uses the usual rule of 5% false positives and 80% power, which makes the smallest detectable difference about 2.8 times the standard error of the difference (1.96 + 0.84 = 2.8). In the toy it is 0.029. The point to remember is that "B beats A by 0.02" on a benchmark whose detectable difference is 0.03 is not evidence of anything.

2.8 Saturation, contamination and Goodhart

Benchmarks age. Press says they typically saturate within a year, so a builder should not worry about whether a question is still interesting in five years [2]. Anthropic's post says that when a benchmark nears saturation, progress also slows, because only the hardest tasks are left, and large capability gains look like small score changes [21]. METR's time-horizon work is one response to a related problem. It measures the length of task, in human time, that a model completes with 50% probability, to give a score a real-world meaning, and it reports a doubling time of about seven months (the authors' estimate, with limits that they discuss) [13]. I read only the abstract of [13].

Headroom. If a share of the items are broken (wrong key, impossible task), no agent can score above 100% minus that share. In the lab's toy timeline, 5% of 200 items are broken, so the ceiling is 0.95. The best score every fourth month goes 0.03, 0.083, 0.208, 0.334, 0.646, 0.782, 0.876, 0.914 and 0.936. By my rule (the best score is within two standard errors of the ceiling, using the standard error at the final score), the benchmark is saturated from month 29. The best score first reaches 0.9 in month 27. For n = 200 and a score of 0.936, my arithmetic gives SE = 0.0173, so two SEs are 0.035, and anything at or above 0.915 is within noise of the ceiling.

The lab also tries to forecast the month when the best score first reaches 0.9, by drawing a straight line through the last four points available. The forecast depends strongly on when you make it: it is month 40.5 using the first 8 months, 26.5 using 12, 18.8 using 16, 27.2 using 20 and 24.0 using 24, against an actual 27. So a straight line is neither reliably early nor reliably late, and it is a poor tool near the middle of an S-shaped curve. I first expected the lab to show that straight lines under-predict the speed of progress, but that is not what came out. The number to take from this is that a forecast made from a few points can be off by a factor of two in either direction.

Contamination and dynamic benchmarks

Contamination is when test items end up in a model's training data. Oren and colleagues show a test that needs neither the training data nor the weights: if a benchmark's items are exchangeable, then all orderings should be equally likely, and a model that has memorized the canonical order will find it more likely than a shuffled one. They audit five public models and find little evidence of widespread contamination (abstract only) [20]. Kapoor and colleagues note that for agents the problem is worse than training contamination, since a developer can program test items directly into the agent [6]. Two ways to reduce the problem are a secret held-out set (GAIA keeps the answers to 300 of its 466 questions private [15]) and a benchmark that is renewed over time. Dynabench builds datasets with people and models in the loop, in which annotators try to write examples that a target model fails and another person gets right (abstract only) [19].

Goodhart's law

When a measure becomes a target, it stops being a good measure. Manheim and Garrabrant describe several distinct ways this happens, and note that the effect grows with the optimization power that is aimed at the proxy (abstract only) [18]. The lab has a small demonstration. A "tuned" agent goes through the task kinds one at a time (coordinate ascent). For each kind it compares doing the work with the known shortcut for that kind (reading the gold file on lookups, dumping everything on lists, answering "unknown" on yes/no tasks, or answering nothing on refusal tasks), and keeps the shortcut only if it scores strictly higher on the flawed suite. At all four skill levels it kept the shortcut on list tasks only. At slip rates of 0.04, 0.10, 0.20 and 0.30, the untuned agents score 0.961, 0.888, 0.786 and 0.708 on both suites. The tuned agents score 0.966, 0.911, 0.828 and 0.763 on the flawed suite (they gain 0.005 to 0.055), and 0.841, 0.786, 0.703 and 0.638 on the fixed suite (they lose 0.07 to 0.12, my subtraction). Over the eight agents the correlation between the flawed and the fixed score is 0.785, which is still high. A benchmark can rank agents reasonably well and still be gamed by an agent that was built for it. I designed the shortcuts, so this shows how the mechanism works. It says nothing about how much real agents game real benchmarks.

Fewer items: tinyBenchmarks

The additional reading by Polo and colleagues goes the other way: instead of asking for more items, it asks how few you need. They report that 100 curated examples are enough to estimate a model's score on MMLU (14K items), and they release tiny versions of several benchmarks (abstract only; I did not read the method) [3]. Two cautions are mine. At 100 items, a score near 50% has a standard error of 0.05, so the 95% interval is about ±10 points, and a method that estimates the full-benchmark score has to do better than the plain average to be useful. And a tiny benchmark inherits every flaw of the big one, so check the items first (2.4). Whether the method carries over to agent tasks, which are few and costly, is an open question that I have not looked into. The paper is about language-model benchmarks with thousands of items.

2.9 Worked arithmetic

All the numbers in this section are my own arithmetic from the formulas above. The inputs are made up. Redo each by hand before you move on.

1. pass^k and pass@k from a rate. A task has a 0.9 chance of passing on each independent trial. Then pass^8 = 0.9⁸ = 0.430, and pass@8 = 1 − 0.1⁸ = 0.99999999, which is 1 for any practical purpose. At 0.6, pass^8 = 0.6⁸ = 0.017. This is why a score of about 61% on one trial and below 25% on eight (the τ-bench retail result [7]) can both be true.
2. A small benchmark. For n = 64 and p = 0.9, SE = √(0.9 × 0.1 / 64) = 0.0375, and the 95% interval is about ±0.0735. So an agent at 0.90 could truly be at 0.83 or 0.97.
3. Press's sizes. At p = 0.5 and n = 150, 300 and 500, SE = 0.0408, 0.0289 and 0.0224, and the 95% half-widths are 0.080, 0.057 and 0.044 [2]. Going from 150 to 500 tasks (3.3 times as many) shrinks the interval by a factor of 1.8, because the width goes as 1 / √n.
4. What difference can you see? The smallest difference that you can detect with 5% false positives and 80% power is about 2.8 times the standard error of the difference (1.96 + 0.84). Two models run on different tasks, n = 500 each, p = 0.5: SE of the difference = 0.0224 × √2 = 0.0316, so the smallest difference is about 0.089. If the same tasks are used and the per-task scores of the two models correlate at 0.8, then SE of the difference = 0.0224 × √(2 × (1 − 0.8)) = 0.0141, and the smallest difference is about 0.040. I assume equal variances and a simple pass/fail score. Miller derives the general form [12].
5. The design effect of clusters. Suppose tasks come in groups of m = 8, and the correlation of scores inside a group is 0.3. The usual rule (general statistics, not taken from the sources above) gives a variance inflation of 1 + (m − 1) × 0.3 = 3.1, so the standard error is larger by a factor of √3.1 = 1.76. Miller's Equation 4 is the version that works from the data [12].
6. The ceiling. If 5% of 200 items are broken and the best score is 0.936, then SE = √(0.936 × 0.064 / 200) = 0.0173. Two SEs are 0.035, so a score of 0.915 or more is within noise of the 0.95 ceiling.
7. A mix of two agents. From Figure 4, to get 0.95 at the lowest cost, mix c (3.60, 0.930) and d (7.16, 0.963). The share of d is (0.95 − 0.930) / (0.963 − 0.930) = 0.606. The cost is 3.60 + 0.606 × (7.16 − 3.60) = 5.76.

2.10 Week 7b in one page

• A task is a request, an environment, stopping criteria and a scorer. Write all four down, including what happens when a run hits a limit [2].
• The score only means something if the task is valid (no shortcuts, no impossible tasks) and the outcome check is valid (it passes what is solved and fails what is not) [1].
• Test the benchmark itself. Run an oracle that should pass everything, and trivial agents (empty, unknown, dump, peek, vandal) that should pass nothing. In the toy, this found flags on 46 of 64 planted-flaw tasks and none after fixing.
• Grade the end state, and use a process score only as an extra signal [21].
• A good benchmark is natural, automatically evaluated and challenging; the exact launch window has moved over time [2]. It should have enough tasks for its error bars, a hold-out set at the right level, and honest reporting [6] [11].
• Use a realistic scaffold. Report cost with accuracy and look at the Pareto frontier [6] [16].
• Run several trials. Report pass@k or pass^k to match the use, and error bars with clusters and pairing [7] [12].
• Say what difference your benchmark can detect before you claim that one system beats another.
• Expect saturation. Estimate the ceiling from broken items, and do not trust a straight-line forecast from a few points.
• When a measure becomes a target, tuning to it can raise the score and lower the real result [18].

In the lab's toy world, the four numbers to remember are: 46 of 64 flagged, then 0; a skilled agent's score unchanged by the flaws but a vandal moving from 0.688 to 1.000; pass^16 of 0.39 against pass@16 of 1.0; and a cost-limit flip at a limit of 8 to 10. All are made up.

3. Reading guide

The course lists one required reading and two additional readings for this week [4]. For each you get: why it matters, the key ideas, and questions to think about as you read. Read the questions first. The optional table at the end points to the sources I used for the notes. For most of those I read only the abstract, and the references say which. Press co-wrote SWE-bench with John Yang, a course instructor [8]. The Zhu paper includes authors who build benchmarks and agents. tinyBenchmarks' authors report their own results.

Optional readings

Source	Why it is worth your time
Kapoor et al., AI Agents That Matter [6]	Cost-controlled comparisons, hold-outs by level of generality, and reproducibility. I read the abstract, the introduction and sections 2 and 5. Also used in Weeks 1 and 3b.
Yao et al., τ-bench [7]	The state-based scorer and pass^k. I read the abstract and the definitions of pass^k and pass@k.
Miller, Adding Error Bars to Evals [12]	Short, practical statistics for an eval: clusters, pairs, power. I read the body text and not the appendix derivations.
Anthropic, Demystifying evals for AI agents [21]	The practitioner's version of the same ideas. I read the whole post. The course may treat it next lecture; I used it lightly.
Reuel et al., BetterBench [11]; Liang et al., HELM [10]	Benchmark quality in general, and the multi-metric argument. Abstracts only.
Kapoor et al., HAL [16]; Kwa et al., METR time horizon [13]	A standard harness and a metric that carries a real-world meaning. Abstracts only.
Jimenez et al., SWE-bench [8]; OpenAI, SWE-bench Verified [9]; WebArena [14]; GAIA [15]	The benchmarks used as examples. For SWE-bench, WebArena and GAIA I read the abstract. For [9], I reused the reading from Week 7a.
Oren et al. [20]; Kiela et al., Dynabench [19]; Manheim and Garrabrant [18]	Contamination, dynamic benchmarks and Goodhart's law. Abstracts only.

4. Hands-on lab: a 4-tuple eval harness and a benchmark auditor

Time: about 4.5 hours. Goal: build and test a small eval toolkit: a harness for tasks written as a request, an environment with state, stopping criteria (step, time and cost limits) and a scorer; an ABC-style auditor that runs an oracle and trivial agents over the tasks; outcome and process scores; pass@k, pass^k, variance, naive, clustered and paired standard errors, and the smallest detectable difference; scaffold and limit sensitivity; a cost and accuracy Pareto frontier; saturation and headroom checks; and a Goodhart demo. You do not need an API key, a model or any package. The lab never touches the network and never calls a model. Python 3.10 or later is enough.

What you will learn

• How to write a task as a request, an environment, stopping criteria and a scorer, and run it with limits on steps, time and cost.
• How strict and loose scorers (exact, contains, number, id set, state) differ, and how a loose scorer lets trivial agents pass.
• How an oracle and a set of trivial agents find shortcuts and impossible tasks, and how to read the flags.
• How an outcome score and a process score disagree.
• How to compute pass@k, pass^k, a variance split, naive, clustered and paired standard errors, and the smallest difference a suite can detect.
• How scaffold and limit changes move scores and rankings, and how a Pareto frontier and a convex hull are found.
• How to estimate a ceiling from broken items, flag saturation, and see how unstable a straight-line forecast is.
• How tuning to a flawed benchmark can raise its score and lower the true one.

Map of the code

Part	Main names in the code	What it does, and what it stands for
1	Scaffold, FULL, BARE, TRUNC, NOSEARCH, Env, Limits, Trace, run_episode	The harness: a world of tickets with six tools, a scaffold that sets tools, truncation and verify-on-write, limits, and one run (2.2, 2.3) [2].
2	score_outcome, process_score	Scorers: exact, contains, number, id set, strict or loose state, a refusal rule, and a process score (2.3) [1] [21].
3	make_world, make_suite	The 64 tasks in seven kinds and 8 owner clusters, built flawed or fixed (2.4).
4	PlanAgent, EmptyAgent, IdkAgent, RefuseAllAgent, DumpAgent, GuessAgent, PeekAgent, VandalAgent, CarelessAgent, LuckyAgent	Scripted agents: a planner with skill and price knobs, and the trivial or gaming agents.
5	audit_suite, audit_summary, baseline_scores, pilot_outliers	The auditor with an oracle and trivial agents (2.4) [1].
6	run_suite, matrix, clusters_of	Running every task for several trials, with its own random stream for each (task, trial).
7	pass_at_k, pass_hat_k, variance_components, se_naive, se_clustered, ci95, wilson, paired_difference, min_detectable_difference	Reliability and statistics (2.7) [7] [12] [17].
8	scaffold_comparison, ranking_flips, limit_curve	Scaffold and limit sensitivity (2.3, 2.6) [16].
9	GRID, pareto_frontier, upper_hull, cost_accuracy	Cost and accuracy of six made-up agents (2.6) [6].
10	best_so_far, ceiling, saturation_report, months_to_target, toy_timeline, discrimination	Saturation, headroom and forecasts (2.8) [2].
11	MixAgent, tune_on_suite, goodhart_demo, pearson	Tuning to a flawed benchmark (2.8) [18].
12	main	The ten-section demo.

Setup

1. You need Python 3.10 or later. There is nothing to install. I ran it on Python 3.13 only.
2. Save the code below as eval_fundamentals_toolkit.py. Or download it: eval_fundamentals_toolkit.py.
3. Save the tests as test_eval_fundamentals_toolkit.py in the same folder. Or download: test_eval_fundamentals_toolkit.py. The test code is not printed on this page, to keep the page shorter. It is in the download, and the names of all the tests are listed after the exercises.
4. Run python eval_fundamentals_toolkit.py for the ten-section demo, and python test_eval_fundamentals_toolkit.py for the tests. Both are deterministic. The demo and the tests each take a few seconds to a few tens of seconds on my machine.

The code: eval_fundamentals_toolkit.py

"""Week 7b toolkit: evaluation fundamentals and benchmark design, standard library only.

Nothing here calls a model, an API or the network. Every world is made up: the help-desk tickets, the
scripted "agents", the "models", the benchmark flaws. The code shows how the pieces work, not how real
benchmarks or real agents behave.

Parts:  1 harness (request, environment, stopping criteria, scorer)   2 scorers and process score
        3 task suite with planted flaws   4 scripted agents   5 benchmark auditor (ABC-style)
        6 running a suite   7 reliability and statistics   8 scaffold and limit sensitivity
        9 cost and Pareto   10 saturation and headroom   11 Goodhart demo   12 demo
"""
import copy
import itertools
import json
import math
import random
import re
from collections import Counter, deque

# ---------------------------------------------------------------- shared helpers


def mean(xs):
    xs = list(xs)
    return sum(xs) / len(xs) if xs else 0.0


def stdev(xs, ddof=1):
    """Sample standard deviation (n - 1 by default). One value or none gives 0.0."""
    xs = list(xs)
    if len(xs) <= ddof:
        return 0.0
    m = mean(xs)
    return math.sqrt(sum((x - m) ** 2 for x in xs) / (len(xs) - ddof))


def sigmoid(x):
    if x >= 0:
        return 1.0 / (1.0 + math.exp(-x))
    e = math.exp(x)
    return e / (1.0 + e)


def normalize(text):
    """Lower case, trim, drop trailing punctuation, collapse spaces."""
    t = re.sub(r"\s+", " ", str(text).strip().lower())
    return t.rstrip(".!?;:,").strip()


# ---------------------------------------------------------------- part 1: the 4-tuple harness

STATUSES = ["open", "pending", "closed"]
PRIORITIES = ["low", "medium", "high"]
OWNERS = ["alice", "bob", "carol", "dmitri", "eleanor", "fatima", "giovanni", "hyun"]
FIELDS = {"status": STATUSES, "priority": PRIORITIES}
TOOLS = frozenset(["get", "list", "count", "set", "search", "read"])


class Scaffold:
    """What the agent is allowed and able to do around the model: tools, how much of each output it sees,
    and whether it reads back its own writes."""

    def __init__(self, name, tools=TOOLS, obs_chars=None, verify=False, retries=1):
        self.name, self.tools, self.obs_chars = name, frozenset(tools), obs_chars
        self.verify, self.retries = verify, retries

    def __repr__(self):
        return f"Scaffold({self.name})"


FULL = Scaffold("full", verify=True)
BARE = Scaffold("bare")
TRUNC = Scaffold("trunc", obs_chars=44)
NOSEARCH = Scaffold("nosearch", tools=TOOLS - {"search"})


class Env:
    """The environment: help-desk tickets (state), some documents, and files the agent must not see."""

    def __init__(self, state, files, hidden, leaky, scaffold):
        self.state = copy.deepcopy(state)
        self.files = dict(files)
        self.hidden, self.leaky, self.scaffold = dict(hidden), leaky, scaffold

    def call(self, name, args):
        if name not in self.scaffold.tools:
            out = "ERROR unknown tool"
        else:
            try:
                out = getattr(self, "_" + name)(*args)
            except (TypeError, KeyError, ValueError):
                out = "ERROR bad call"
        n = self.scaffold.obs_chars
        return out if n is None else out[:n]

    def _get(self, tid):
        t = self.state.get(tid)
        if t is None:
            return "ERROR no such ticket"
        return f"{tid} owner={t['owner']} status={t['status']} priority={t['priority']} locked={int(t['locked'])}"

    def _list(self, owner):
        return ",".join(sorted(k for k, t in self.state.items() if t["owner"] == owner))

    def _count(self, owner, status):
        return str(sum(1 for t in self.state.values() if t["owner"] == owner and t["status"] == status))

    def _set(self, tid, fld, value):
        t = self.state.get(tid)
        if t is None:
            return "ERROR no such ticket"
        if fld not in FIELDS or value not in FIELDS[fld]:
            return "ERROR bad field or value"
        if t["locked"]:
            return "ERROR locked"
        t[fld] = value
        return f"OK {tid} {fld}={value}"

    def _search(self, word):
        return ",".join(sorted(p for p, text in self.files.items() if word in text))

    def _read(self, path):
        if path in self.files:
            return self.files[path]
        if self.leaky and path in self.hidden:
            return self.hidden[path]
        return "ERROR no such file"


class Limits:
    """Stopping criteria besides submit. None means no limit. A submit counts as one step."""

    def __init__(self, max_steps=None, max_seconds=None, max_cost=None):
        self.max_steps, self.max_seconds, self.max_cost = max_steps, max_seconds, max_cost

    def as_dict(self):
        return {"max_steps": self.max_steps, "max_seconds": self.max_seconds, "max_cost": self.max_cost}


class Trace:
    def __init__(self, task_id):
        self.task_id, self.steps, self.status = task_id, [], None
        self.answer, self.state, self.cost, self.seconds = None, None, 0.0, 0.0

    @property
    def n_steps(self):
        return len(self.steps)


def tool_seconds(out):
    return 1.0 + len(out) / 200.0


def run_episode(task, suite, agent, scaffold, rng, limits=None):
    """Run one agent on one task. The agent sees the request, then each tool output.
    Limits are checked before each step (steps) and after the step is charged (cost, time). A step that
    breaks the cost or time limit is not carried out. Status: submitted, step_limit, cost_limit, time_limit."""
    lim = limits if limits is not None else Limits(**task["limits"])
    env = Env(suite["state"], suite["files"], task["hidden"], task["leaky"], scaffold)
    tr = Trace(task["id"])
    agent.reset(task, suite, scaffold, rng)
    obs, n_actions = task["request"], 0
    while True:
        if lim.max_steps is not None and n_actions >= lim.max_steps:
            tr.status = "step_limit"
            break
        action = agent.act(obs)
        n_actions += 1
        tr.cost += agent.step_cost
        tr.seconds += agent.step_seconds
        if lim.max_cost is not None and tr.cost > lim.max_cost:
            tr.status = "cost_limit"
            break
        if lim.max_seconds is not None and tr.seconds > lim.max_seconds:
            tr.status = "time_limit"
            break
        if action[0] == "submit":
            tr.answer, tr.status = action[1], "submitted"
            break
        name, args = action[1], list(action[2])
        obs = env.call(name, args)
        tr.seconds += tool_seconds(obs)
        tr.steps.append((name, tuple(args), obs))
    tr.state = env.state
    return tr


# ---------------------------------------------------------------- part 2: scorers and process score


def ids_in(text):
    return set(re.findall(r"T-\d+", str(text)))


def score_outcome(task, trace, initial_state, limit_policy="score"):
    """Score the end of the run. 1.0 or 0.0 for every scorer here. With limit_policy 'zero', a run that
    stopped on a limit scores 0 whatever it left behind; with 'score' the state and any answer are scored."""
    if limit_policy == "zero" and trace.status != "submitted":
        return 0.0
    sc, ans = task["scorer"], trace.answer
    kind = sc["type"]
    if kind == "exact":
        return float(ans is not None and normalize(ans) == normalize(sc["expect"]))
    if kind == "contains":
        return float(ans is not None and normalize(sc["expect"]) in normalize(ans))
    if kind == "contains_all_ids":
        return float(ans is not None and ids_in(sc["expect"]) <= ids_in(ans))
    if kind == "id_set":
        return float(ans is not None and ids_in(ans) == ids_in(sc["expect"]))
    if kind == "number":
        m = re.search(r"-?\d+", str(ans)) if ans is not None else None
        return float(m is not None and int(m.group()) == int(sc["expect"]))
    if kind == "state":
        final = trace.state
        for tid, fields in sc["goal"].items():
            if any(final[tid][f] != v for f, v in fields.items()):
                return 0.0
        if sc["strict"]:
            for tid, t in initial_state.items():
                if tid not in sc["goal"] and final[tid] != t:
                    return 0.0
            for tid, fields in sc["goal"].items():
                if any(final[tid][f] != v for f, v in initial_state[tid].items() if f not in fields):
                    return 0.0
        if sc["need_refusal"]:
            return float(ans is not None and "cannot" in normalize(ans))
        return 1.0
    raise ValueError(f"unknown scorer {kind}")


def process_score(task, trace):
    """A process score: the share of the planned tool calls that were made, in order, times 0.5 if the run
    wrote to a ticket that the task does not name. It looks at the path, not at the end."""
    plan = [(n, tuple(a)) for n, a in task["plan"]]
    made = [(n, tuple(a)) for n, a, _ in trace.steps]
    j = 0
    for call in made:
        if j < len(plan) and call == plan[j]:
            j += 1
    share = j / len(plan) if plan else 1.0
    named = set(task["named"])
    collateral = any(n == "set" and a[0] not in named for n, a, _ in trace.steps)
    return share * (0.5 if collateral else 1.0)


# ---------------------------------------------------------------- part 3: a task suite with planted flaws

KIND_COUNTS = [("lookup", 14), ("count", 8), ("list", 8), ("yesno", 8), ("update", 12), ("refuse", 8), ("doc", 6)]
DOC_WORDS = ["escalation", "refund", "warranty", "privacy", "outage", "invoice", "shipping", "upgrade"]


def make_world(rng, n_tickets=32):
    state = {}
    for i in range(1, n_tickets + 1):
        state[f"T-{i:02d}"] = {
            "owner": rng.choice(OWNERS),
            "status": rng.choices(STATUSES, weights=[0.45, 0.25, 0.30])[0],
            "priority": rng.choice(PRIORITIES),
            "locked": rng.random() < 0.15,
        }
    files = {f"/docs/{w}.txt": f"Policy on {w}. See also the help desk handbook." for w in DOC_WORDS}
    return state, files


def _scorer(kind, expect, goal, flawed):
    if kind == "lookup":
        return {"type": "exact", "expect": expect}
    if kind == "doc":
        return {"type": "exact", "expect": expect}
    if kind == "count":
        return {"type": "number", "expect": expect}
    if kind == "list":
        return {"type": "contains_all_ids" if flawed else "id_set", "expect": expect}
    if kind == "yesno":
        return {"type": "contains" if flawed else "exact", "expect": expect}
    if kind == "update":
        return {"type": "state", "goal": goal, "strict": not flawed, "need_refusal": False}
    if kind == "refuse":
        return {"type": "state", "goal": {}, "strict": not flawed, "need_refusal": not flawed}
    raise ValueError(kind)


def make_suite(seed=0, flawed=True, n_tickets=32):
    """The same tasks are drawn for flawed=True and flawed=False. Only the scorers, the leaks and the
    limits differ, so the two suites can be compared head to head."""
    rng = random.Random(seed)
    state, files = make_world(rng, n_tickets)
    owners_used = sorted({t["owner"] for t in state.values()})
    hardness = {o: round(rng.uniform(0.5, 2.0), 2) for o in owners_used}
    tids = sorted(state)
    unlocked_open = [t for t in tids if not state[t]["locked"] and state[t]["status"] != "closed"]
    locked = [t for t in tids if state[t]["locked"]]
    closed = [t for t in tids if state[t]["status"] == "closed"]
    notclosed = [t for t in tids if state[t]["status"] != "closed"]
    tasks = []
    for kind, n in KIND_COUNTS:
        for i in range(n):
            tasks.append(_draw(kind, i, rng, state, files, owners_used, tids, unlocked_open, locked, closed, notclosed))
    rng.shuffle(tasks)
    leaky_kinds = {"lookup", "count", "doc"}
    k = 0
    for idx, t in enumerate(tasks):
        t["id"] = f"task{idx:02d}"
        t["hardness"] = hardness[t["cluster"]]
        t["owners"], t["all_tickets"], t["keywords"] = owners_used, tids, DOC_WORDS
        t["victim"] = next(x for x in unlocked_open if x not in t["named"])
        leaky = flawed and t["kind"] in leaky_kinds and k % 3 == 0
        k += t["kind"] in leaky_kinds
        t["leaky"] = leaky
        t["hidden"] = {"/gold": str(t["expect"])} if t["kind"] in leaky_kinds else {}
        t["scorer"] = _scorer(t["kind"], t["expect"], t["goal"], flawed)
        no_limits = flawed and idx % 5 == 0
        t["limits"] = Limits().as_dict() if no_limits else Limits(max_steps=14, max_seconds=120.0, max_cost=40.0).as_dict()
    return {"seed": seed, "flawed": flawed, "state": state, "files": files, "tasks": tasks, "hardness": hardness}


def _draw(kind, i, rng, state, files, owners, tids, unlocked_open, locked, closed, notclosed):
    base = {"kind": kind, "goal": {}, "extract": ["raw"], "named": []}
    if kind == "lookup":
        t, f = rng.choice(tids), rng.choice(["status", "priority"])
        base.update(request=f"What is the {f} of ticket {t}?", plan=[["get", [t]]], extract=["field", f],
                    expect=state[t][f], cluster=state[t]["owner"], named=[t])
    elif kind == "count":
        o, s = rng.choice(owners), rng.choice(STATUSES)
        n = sum(1 for x in state.values() if x["owner"] == o and x["status"] == s)
        base.update(request=f"How many {s} tickets does {o} have?", plan=[["count", [o, s]]], expect=str(n), cluster=o)
    elif kind == "list":
        o = rng.choice(owners)
        ids = sorted(k for k, x in state.items() if x["owner"] == o)
        base.update(request=f"List the ids of all tickets owned by {o}.", plan=[["list", [o]]],
                    expect=",".join(ids), cluster=o)
    elif kind == "yesno":
        t = rng.choice(closed if i % 2 == 0 else notclosed)
        base.update(request=f"Is ticket {t} closed? Answer yes or no.", plan=[["get", [t]]],
                    extract=["yesno"], expect="yes" if state[t]["status"] == "closed" else "no",
                    cluster=state[t]["owner"], named=[t])
    elif kind == "update":
        t = rng.choice(unlocked_open)
        v = rng.choice([p for p in PRIORITIES if p != state[t]["priority"]])
        goal = {t: {"priority": v}}
        plan = [["set", [t, "priority", v]]]
        text = f"Set the priority of ticket {t} to {v}."
        if i % 2 == 1:
            goal[t]["status"] = "closed"
            plan.append(["set", [t, "status", "closed"]])
            text = f"Set the priority of ticket {t} to {v} and close it."
        base.update(request=text, plan=plan, extract=["none"], expect="done", goal=goal,
                    cluster=state[t]["owner"], named=[t])
    elif kind == "refuse":
        t = rng.choice(locked)
        base.update(request=f"Close ticket {t}.", plan=[["set", [t, "status", "closed"]]], extract=["refuse"],
                    expect="cannot", cluster=state[t]["owner"], named=[t])
    elif kind == "doc":
        w = rng.choice(DOC_WORDS)
        base.update(request=f"Which document mentions the word '{w}'? Give the path.", plan=[["search", [w]]],
                    expect=f"/docs/{w}.txt", cluster="docs")
        base["cluster"] = rng.choice(owners)
    return base


# ---------------------------------------------------------------- part 4: scripted agents


def _parse_field(obs, fld):
    m = re.search(rf"\b{fld}=(\S+)", obs)
    return m.group(1) if m else None


class PlanAgent:
    """A scripted agent that follows the task's plan. With probability slip x hardness each tool call goes
    wrong (a wrong id, name or value). If the scaffold verifies writes, the agent reads a write back and
    repeats it up to scaffold.retries times when it does not match. This is not a model; it is a
    stand-in with one knob for skill (slip) and one for price (step_cost)."""

    def __init__(self, slip=0.1, step_cost=1.0, step_seconds=2.0):
        self.slip, self.step_cost, self.step_seconds = slip, step_cost, step_seconds

    def reset(self, task, suite, scaffold, rng):
        self.task, self.sc, self.rng = task, scaffold, rng
        self.queue = deque((n, list(a)) for n, a in task["plan"])
        self.p = min(1.0, self.slip * task["hardness"])
        self.final_obs, self.last, self.pending_check, self.retries_left = "", None, None, scaffold.retries
        self.refused = False

    def _slipped(self, name, args):
        t = self.task
        if self.rng.random() >= self.p:
            return args
        args = list(args)
        if name == "get":
            args[0] = self.rng.choice([x for x in t["all_tickets"] if x != args[0]])
        elif name in ("list", "count"):
            args[0] = self.rng.choice([x for x in t["owners"] if x != args[0]])
        elif name == "search":
            args[0] = self.rng.choice([x for x in t["keywords"] if x != args[0]])
        elif name == "set":
            args[2] = self.rng.choice([x for x in FIELDS[args[1]] if x != args[2]])
        return args

    def act(self, obs):
        last, self.last = self.last, None
        if last == "check":
            tid, fld, value = self.pending_check
            if _parse_field(obs, fld) != value and self.retries_left > 0:
                self.retries_left -= 1
                self.queue.appendleft(("set", [tid, fld, value], "fix"))
        elif last in ("plan", "plan_set", "fix"):
            if last != "fix":
                self.final_obs = obs
            if obs.startswith("ERROR"):
                self.refused = last != "fix" or self.refused
            elif last in ("plan_set", "fix") and self.sc.verify:
                self.queue.appendleft(("get", [self.pending_check[0]], "check"))
        if not self.queue:
            return ("submit", self._answer())
        item = self.queue.popleft()
        name, args = item[0], item[1]
        tag = item[2] if len(item) > 2 else "plan"
        if tag == "check":
            self.last = "check"
            return ("tool", "get", args)
        if tag == "vandal":
            return ("tool", name, args)
        if name == "set" and tag == "plan":
            self.pending_check = (args[0], args[1], args[2])
        self.last = "plan_set" if (name == "set" and tag == "plan") else ("fix" if tag == "fix" else "plan")
        if tag == "fix":
            args = list(self.pending_check)
        return ("tool", name, self._slipped(name, args))

    def _answer(self):
        ex, obs = self.task["extract"], self.final_obs
        if ex[0] == "field":
            return _parse_field(obs, ex[1]) or "unknown"
        if ex[0] == "yesno":
            s = _parse_field(obs, "status")
            return "unknown" if s is None else ("yes" if s == "closed" else "no")
        if ex[0] == "refuse":
            return "I cannot do that" if self.refused else "done"
        if ex[0] == "none":
            return "done"
        return obs if obs else "unknown"


class FixedAgent:
    """Base for the trivial agents: no plan, no skill."""
    step_cost, step_seconds = 0.5, 1.0

    def reset(self, task, suite, scaffold, rng):
        self.task, self.queue, self.out = task, deque(), ""


class EmptyAgent(FixedAgent):
    def act(self, obs):
        return ("submit", "")


class IdkAgent(FixedAgent):
    def act(self, obs):
        return ("submit", "unknown")


class RefuseAllAgent(FixedAgent):
    def act(self, obs):
        return ("submit", "I cannot do that")


class DumpAgent(FixedAgent):
    """Lists every owner's tickets and submits all of it."""

    def reset(self, task, suite, scaffold, rng):
        super().reset(task, suite, scaffold, rng)
        self.queue = deque(task["owners"])
        self.got = []
        self.started = False

    def act(self, obs):
        if self.started:
            self.got.append(obs)
        self.started = True
        if self.queue:
            return ("tool", "list", [self.queue.popleft()])
        return ("submit", " ".join(self.got))


class GuessAgent(FixedAgent):
    """Submits the most common correct answer for the task's kind in the suite (a chance baseline)."""

    def __init__(self, modal):
        self.modal = modal

    def act(self, obs):
        return ("submit", self.modal.get(self.task["kind"], ""))


class PeekAgent(FixedAgent):
    """Reads /gold if the environment lets it, and submits it."""

    def reset(self, task, suite, scaffold, rng):
        super().reset(task, suite, scaffold, rng)
        self.asked = False

    def act(self, obs):
        if not self.asked:
            self.asked = True
            return ("tool", "read", ["/gold"])
        return ("submit", "" if obs.startswith("ERROR") else obs)


class VandalAgent(PlanAgent):
    """Does the plan correctly, then also closes a ticket the task never mentions."""

    def __init__(self):
        super().__init__(slip=0.0)

    def reset(self, task, suite, scaffold, rng):
        super().reset(task, suite, scaffold, rng)
        self.queue.append(("set", [task["victim"], "status", "closed"], "vandal"))


class CarelessAgent(PlanAgent):
    """Does the plan correctly, then with probability q submits a stale answer. The path is good, the end is not."""

    def __init__(self, q=0.4):
        super().__init__(slip=0.0)
        self.q = q

    def _answer(self):
        if self.rng.random() < self.q:
            return "unknown"
        return super()._answer()


class LuckyAgent(FixedAgent):
    """Submits the modal answer without looking. It never uses a tool."""

    def __init__(self, modal):
        self.modal = modal

    def act(self, obs):
        return ("submit", self.modal.get(self.task["kind"], "done"))


def modal_answers(suite):
    """The most common expected answer per kind among tasks scored by a string or number."""
    by = {}
    for t in suite["tasks"]:
        if t["kind"] in ("lookup", "count", "yesno", "doc"):
            by.setdefault(t["kind"], Counter())[str(t["expect"])] += 1
    return {k: c.most_common(1)[0][0] for k, c in by.items()}


# ---------------------------------------------------------------- part 5: the auditor (ABC-style checks)

TRIVIAL = ("empty", "idk", "dump", "guess", "peek")


def trivial_agents(suite):
    m = modal_answers(suite)
    return {"empty": EmptyAgent(), "idk": IdkAgent(), "dump": DumpAgent(), "guess": GuessAgent(m), "peek": PeekAgent()}


def audit_suite(suite, seed=0):
    """Run trivial agents and an oracle against every task and read the specs. Returns flags per task.
    Flags (paper checks in brackets, as I read them): oracle_fails [T.9]; empty_passes and idk_passes [O.g.3,
    O.b.3]; dump_passes [O.b.2]; gold_visible [T.5]; vandal_passes [O.g.2]; no_limits (stopping criteria not
    set). A chance guess is not a flag, because some task always has a modal answer; see baseline_scores."""
    flags = {t["id"]: [] for t in suite["tasks"]}
    agents = trivial_agents(suite)
    oracle = PlanAgent(slip=0.0)
    for ti, t in enumerate(suite["tasks"]):
        rng = random.Random(f"{seed}-audit-{ti}")
        lim = Limits(max_steps=20)
        tr = run_episode(t, suite, oracle, FULL, rng, lim)
        if score_outcome(t, tr, suite["state"]) < 1.0:
            flags[t["id"]].append("oracle_fails")
        for name, ag in agents.items():
            tr = run_episode(t, suite, ag, FULL, rng, lim)
            if name == "peek":
                if any(n == "read" and not o.startswith("ERROR") for n, _, o in tr.steps):
                    flags[t["id"]].append("gold_visible")
            elif name != "guess" and score_outcome(t, tr, suite["state"]) == 1.0:
                flags[t["id"]].append(f"{name}_passes")
        if t["scorer"]["type"] == "state":
            tr = run_episode(t, suite, VandalAgent(), FULL, rng, lim)
            if score_outcome(t, tr, suite["state"]) == 1.0:
                flags[t["id"]].append("vandal_passes")
        if any(v is None for v in t["limits"].values()):
            flags[t["id"]].append("no_limits")
    return flags


def audit_summary(flags):
    c = Counter(f for fl in flags.values() for f in fl)
    n_flagged = sum(1 for fl in flags.values() if fl)
    return {"n_tasks": len(flags), "n_flagged": n_flagged, "share_flagged": n_flagged / len(flags) if flags else 0.0,
            "by_flag": dict(sorted(c.items()))}


def baseline_scores(suite, seed=0):
    """Mean score of each trivial agent, and of the refuse-everything agent, over the whole suite."""
    agents = dict(trivial_agents(suite))
    agents["refuse_all"] = RefuseAllAgent()
    out = {}
    for name, ag in agents.items():
        sc = []
        for ti, t in enumerate(suite["tasks"]):
            rng = random.Random(f"{seed}-base-{ti}")
            tr = run_episode(t, suite, ag, FULL, rng, Limits(max_steps=20))
            sc.append(score_outcome(t, tr, suite["state"]))
        out[name] = mean(sc)
    return out


def pilot_outliers(results):
    """Tasks that no trial of any agent passed, or that every trial passed (check T.10, read in the paper as
    inspecting outliers in pilot runs). `results` maps task id to a list of 0/1 scores."""
    never = sorted(t for t, s in results.items() if s and max(s) == 0)
    always = sorted(t for t, s in results.items() if s and min(s) == 1)
    return {"never": never, "always": always}


# ---------------------------------------------------------------- part 6: running a suite


def make_agent(spec):
    """spec = (slip, step_cost). Returns a fresh PlanAgent."""
    return PlanAgent(slip=spec[0], step_cost=spec[1])


def run_suite(suite, agent_fn, scaffold, trials=8, seed=0, limits=None, limit_policy="score"):
    """Returns {task_id: [row, ...]} with row = dict(score, proc, cost, steps, status). Every (task, trial)
    has its own random stream, so changing one setting does not shift the others."""
    out = {}
    for ti, t in enumerate(suite["tasks"]):
        rows = []
        for k in range(trials):
            rng = random.Random(f"{seed}-{ti}-{k}")
            tr = run_episode(t, suite, agent_fn(), scaffold, rng, limits)
            rows.append({"score": score_outcome(t, tr, suite["state"], limit_policy), "proc": process_score(t, tr),
                         "cost": tr.cost, "steps": tr.n_steps, "status": tr.status})
        out[t["id"]] = rows
    return out


def matrix(results, key="score"):
    """List of per-task lists, in task order."""
    return [[r[key] for r in rows] for rows in results.values()]


def clusters_of(suite):
    return [t["cluster"] for t in suite["tasks"]]


# ---------------------------------------------------------------- part 7: reliability and statistics


def comb(n, k):
    return math.comb(n, k) if 0 <= k <= n else 0


def pass_at_k(n, c, k):
    """Unbiased estimate of the chance that at least one of k trials passes, from n trials with c passes."""
    if k > n:
        raise ValueError("k > n")
    return 1.0 - comb(n - c, k) / comb(n, k)


def pass_hat_k(n, c, k):
    """Unbiased estimate of the chance that all k trials pass (tau-bench's pass^k), from n trials, c passes."""
    if k > n:
        raise ValueError("k > n")
    return comb(c, k) / comb(n, k)


def reliability_curves(mat, ks):
    """Mean over tasks of pass@k and pass^k for each k, from a task x trial 0/1 matrix."""
    n = len(mat[0])
    at = {k: mean(pass_at_k(n, round(sum(r)), k) for r in mat) for k in ks}
    hat = {k: mean(pass_hat_k(n, round(sum(r)), k) for r in mat) for k in ks}
    return {"pass_at_k": at, "pass_hat_k": hat}


def variance_components(mat):
    """Splits the variance of one trial's score into between-task variance (tasks differ) and within-task
    variance (the same task, different trials). Uses the n-1 estimate within each task."""
    n = len(mat[0])
    means = [mean(r) for r in mat]
    within = mean(stdev(r) ** 2 for r in mat)
    between_raw = stdev(means) ** 2
    between = max(0.0, between_raw - within / n)
    return {"between": between, "within": within, "total": between + within,
            "within_share": within / (between + within) if between + within > 0 else 0.0}


def se_naive(scores):
    """Standard error of a mean, treating every score as independent (n - 1 form)."""
    n = len(scores)
    return math.sqrt(stdev(scores) ** 2 / n) if n > 1 else 0.0


def se_clustered(scores, clusters):
    """Cluster-adjusted standard error: the naive variance plus the cross terms of scores that share a
    cluster, divided by n squared (equation 4 of Miller 2024, as I read it)."""
    n = len(scores)
    if n < 2:
        return 0.0
    m = mean(scores)
    by = {}
    for s, c in zip(scores, clusters):
        by.setdefault(c, []).append(s - m)
    cross = sum(sum(v) ** 2 - sum(x * x for x in v) for v in by.values())
    var = stdev(scores) ** 2 / n + cross / (n * n)
    return math.sqrt(max(var, 0.0))


def ci95(scores, clusters=None):
    m = mean(scores)
    se = se_clustered(scores, clusters) if clusters is not None else se_naive(scores)
    return (m - 1.96 * se, m + 1.96 * se)


def wilson(k, n, z=1.96):
    """Wilson score interval for k passes out of n independent trials."""
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))


def paired_difference(a, b, clusters=None):
    """Mean of a - b, its standard error on the paired question-level differences, and z. Clustered if given."""
    d = [x - y for x, y in zip(a, b)]
    se = se_clustered(d, clusters) if clusters is not None else se_naive(d)
    m = mean(d)
    return {"diff": m, "se": se, "z": m / se if se > 0 else (0.0 if m == 0 else math.copysign(math.inf, m))}


def min_detectable_difference(se_diff, z_alpha=1.96, z_power=0.84):
    """The true difference that a two-sided 5% test finds 80% of the time, given the standard error."""
    return (z_alpha + z_power) * se_diff


def task_means(results, key="score"):
    return [mean(rows) for rows in (matrix(results, key))]


# ---------------------------------------------------------------- part 8: scaffold and limit sensitivity

MODEL_A = (0.10, 1.0)   # cheap and sloppier
MODEL_B = (0.04, 3.0)   # careful and dear


def scaffold_comparison(suite, models, scaffolds, trials=8, seed=0, limits=None):
    """Mean score for every (model, scaffold) pair, plus the ranking of the models inside each scaffold."""
    table = {}
    for sname, sc in scaffolds.items():
        table[sname] = {}
        for mname, spec in models.items():
            res = run_suite(suite, lambda spec=spec: make_agent(spec), sc, trials, seed, limits)
            table[sname][mname] = mean(task_means(res))
    ranks = {s: sorted(row, key=lambda m: -row[m]) for s, row in table.items()}
    return {"scores": table, "ranking": ranks}


def ranking_flips(ranks):
    """Pairs of scaffolds that rank the models in a different order."""
    names = list(ranks)
    return [(a, b) for a, b in itertools.combinations(names, 2) if ranks[a] != ranks[b]]


def limit_curve(suite, spec, scaffold, step_limits, trials=8, seed=0, limit_policy="score"):
    """Mean score as the step limit grows."""
    out = {}
    for L in step_limits:
        res = run_suite(suite, lambda: make_agent(spec), scaffold, trials, seed,
                        Limits(max_steps=L), limit_policy)
        out[L] = mean(task_means(res))
    return out


# ---------------------------------------------------------------- part 9: cost and Pareto

GRID = {"a": (0.20, 0.5), "b": (0.12, 1.0), "c": (0.08, 1.5), "d": (0.04, 3.0), "e": (0.04, 6.0), "f": (0.15, 4.0)}


def pareto_frontier(points):
    """points: {name: (cost, accuracy)}. A point is on the frontier if no other point is at least as good
    on both and strictly better on one. Returns names sorted by cost."""
    keep = []
    for n, (c, a) in points.items():
        dominated = any((c2 <= c and a2 >= a) and (c2 < c or a2 > a) for m, (c2, a2) in points.items() if m != n)
        if not dominated:
            keep.append(n)
    return sorted(keep, key=lambda n: points[n][0])


def upper_hull(points):
    """Upper convex hull over (cost, accuracy), left to right. Any point on it can be matched on average by
    a mix of the two neighbouring agents (run one, or the other, with the right probabilities)."""
    pts = sorted(set(points.values()))
    hull = []
    for p in pts:
        while len(hull) >= 2 and (hull[-1][0] - hull[-2][0]) * (p[1] - hull[-2][1]) - (hull[-1][1] - hull[-2][1]) * (p[0] - hull[-2][0]) >= 0:
            hull.pop()
        hull.append(p)
    return hull


def cost_accuracy(suite, grid=None, scaffold=FULL, trials=8, seed=0):
    grid = grid or GRID
    pts = {}
    for name, spec in grid.items():
        res = run_suite(suite, lambda spec=spec: make_agent(spec), scaffold, trials, seed)
        pts[name] = (mean(r["cost"] for rows in res.values() for r in rows), mean(task_means(res)))
    return pts


# ---------------------------------------------------------------- part 10: saturation and headroom


def best_so_far(series):
    out, b = [], -1.0
    for x in series:
        b = max(b, x)
        out.append(b)
    return out


def ceiling(flaw_rate):
    """If a share of the tasks is broken (cannot be passed fairly), the top fair score is about 1 minus that share."""
    return 1.0 - flaw_rate


def saturation_report(series, flaw_rate, n_tasks, launch_window=(0.001, 0.09)):
    """series: best score at each time step. Status words are my own rules:
    launch  = first best score is inside the window (Press's January 2025 advice, 0.1% to 9%);
    saturated = best score is within two standard errors of the ceiling;
    headroom = ceiling minus best."""
    best = best_so_far(series)
    cap = ceiling(flaw_rate)
    last = best[-1]
    se = math.sqrt(max(last * (1 - last), 1e-9) / n_tasks)
    sat_at = next((i for i, b in enumerate(best) if b >= cap - 2 * se), None)
    return {"first_in_window": launch_window[0] <= series[0] <= launch_window[1], "ceiling": cap, "headroom": cap - last,
            "saturated": last >= cap - 2 * se, "first_saturated_step": sat_at, "se_at_end": se}


def months_to_target(series, target, last_n=4):
    """Straight line through the last `last_n` points; steps until it reaches the target. None if it never does."""
    ys = series[-last_n:]
    xs = list(range(len(ys)))
    mx, my = mean(xs), mean(ys)
    den = sum((x - mx) ** 2 for x in xs)
    slope = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / den if den else 0.0
    if ys[-1] >= target:
        return 0.0
    if slope <= 0:
        return None
    return (target - ys[-1]) / slope


def toy_timeline(seed=0, months=36, n_items=200, n_models=3):
    """Made-up: item difficulties on a logit scale, model ability rising with time, a share of the items broken.
    Returns, for each month, the best model score on the whole benchmark."""
    rng = random.Random(seed)
    b = sorted(rng.gauss(2.5, 1.5) for _ in range(n_items))
    broken = set(rng.sample(range(n_items), n_items // 20))
    best = []
    for m in range(months):
        top = -1.0
        for j in range(n_models):
            th = -2.0 + 0.32 * m + rng.gauss(0, 0.3) - 0.3 * j
            sc = mean(0.0 if i in broken else sigmoid(th - b[i]) for i in range(n_items))
            top = max(top, sc)
        best.append(top)
    return {"best": best, "flaw_rate": len(broken) / n_items, "n_items": n_items}


def discrimination(mat_a, mat_b):
    """Share of tasks that cannot tell two models apart because both always pass or both always fail."""
    same = sum(1 for a, b in zip(mat_a, mat_b) if (min(a) == max(a) == min(b) == max(b)))
    return same / len(mat_a)


# ---------------------------------------------------------------- part 11: Goodhart demo


class MixAgent:
    """A planner that, on the kinds of task where a shortcut is known (peek at the gold file, dump everything,
    say unknown, say nothing), takes the shortcut with probability g[kind] instead of doing the work.
    g may be one number (every kind) or a dict by kind."""

    def __init__(self, slip, g=0.0):
        self.plan, self.g = PlanAgent(slip=slip, step_cost=1.0), g
        self.step_cost, self.step_seconds = 1.0, 2.0
        self.short = {"lookup": PeekAgent(), "list": DumpAgent(), "yesno": IdkAgent(), "refuse": EmptyAgent()}

    def reset(self, task, suite, scaffold, rng):
        g = self.g.get(task["kind"], 0.0) if isinstance(self.g, dict) else self.g
        use = task["kind"] in self.short and rng.random() < g
        self.cur = self.short[task["kind"]] if use else self.plan
        self.cur.reset(task, suite, scaffold, rng)

    def act(self, obs):
        return self.cur.act(obs)


def tune_on_suite(suite, slip, trials=6, seed=0):
    """Coordinate ascent, one kind of task at a time: use the shortcut for a kind if it scores strictly
    higher than doing the work on this suite. This is what hill-climbing on a benchmark does, without
    anyone meaning to cheat. Returns {kind: 0 or 1}."""
    g = {}
    for kind in MixAgent(0.0).short:
        sub = dict(suite, tasks=[t for t in suite["tasks"] if t["kind"] == kind])
        if not sub["tasks"]:
            g[kind] = 0
            continue
        work = mean(task_means(run_suite(sub, lambda: MixAgent(slip, 0.0), FULL, trials, seed)))
        cut = mean(task_means(run_suite(sub, lambda: MixAgent(slip, 1.0), FULL, trials, seed)))
        g[kind] = 1 if cut > work else 0
    return g


def goodhart_demo(flawed_suite, fixed_suite, slips=(0.04, 0.1, 0.2, 0.3), trials=6, seed=0):
    """For each skill level: tune the shortcuts on the flawed suite, then score the tuned and the untuned
    agent on both suites. The fixed suite is the truth in this toy."""
    rows = []
    for slip in slips:
        g = tune_on_suite(flawed_suite, slip, trials, seed)
        row = {"slip": slip, "shortcuts": sorted(k for k, v in g.items() if v)}
        for label, gg in (("untuned", 0.0), ("tuned", g)):
            for sname, s in (("flawed", flawed_suite), ("fixed", fixed_suite)):
                row[f"{label}_{sname}"] = mean(task_means(run_suite(s, lambda: MixAgent(slip, gg), FULL, trials, seed + 1)))
        rows.append(row)
    return {"rows": rows,
            "corr": pearson([r[f"{l}_flawed"] for r in rows for l in ("untuned", "tuned")],
                            [r[f"{l}_fixed"] for r in rows for l in ("untuned", "tuned")])}


def pearson(x, y):
    mx, my = mean(x), mean(y)
    sx = math.sqrt(sum((a - mx) ** 2 for a in x))
    sy = math.sqrt(sum((b - my) ** 2 for b in y))
    if sx == 0 or sy == 0:
        return 0.0
    return sum((a - mx) * (b - my) for a, b in zip(x, y)) / (sx * sy)


# ---------------------------------------------------------------- part 12: demo


def _fmt(x, d=3):
    return f"{x:.{d}f}"


def main():
    print("== 1. one task, one run: the 4-tuple (request, environment, stopping criteria, scorer)")
    fl = make_suite(0, flawed=True)
    fx = make_suite(0, flawed=False)
    t = next(t for t in fx["tasks"] if t["kind"] == "update" and len(t["plan"]) == 2)
    tr = run_episode(t, fx, PlanAgent(slip=0.0), FULL, random.Random(1))
    print("request :", t["request"])
    print("steps   :", [(n, list(a), o) for n, a, o in tr.steps], "| status", tr.status, "| answer", repr(tr.answer))
    print("limits  :", t["limits"], "| scorer:", t["scorer"]["type"], "strict" if t["scorer"]["strict"] else "loose",
          "| outcome", score_outcome(t, tr, fx["state"]), "| process", process_score(t, tr))
    for L in (1, 2, 3):
        tr = run_episode(t, fx, PlanAgent(slip=0.0), BARE, random.Random(1), Limits(max_steps=L))
        print(f"  step limit {L}: status {tr.status}, outcome {score_outcome(t, tr, fx['state'])}, "
              f"outcome if limit means zero {score_outcome(t, tr, fx['state'], 'zero')}")

    print("\n== 2. audit of the flawed suite and of the fixed suite (trivial agents, oracle, specs)")
    for name, s in (("flawed", fl), ("fixed", fx)):
        a = audit_summary(audit_suite(s))
        print(f"{name:6s}: {a['n_flagged']} of {a['n_tasks']} tasks flagged; {a['by_flag']}")
        print("        baselines:", {k: _fmt(v, 2) for k, v in baseline_scores(s).items()})

    print("\n== 3. what the planted flaws are worth to different agents (10 trials per task)")
    for name, fn in (("planner slip 0.04", lambda: make_agent((0.04, 1.0))), ("planner slip 0.30", lambda: make_agent((0.30, 1.0))),
                     ("empty answer", EmptyAgent), ("says unknown", IdkAgent), ("dumps everything", DumpAgent),
                     ("planner plus vandal", VandalAgent)):
        a = mean(task_means(run_suite(fl, fn, FULL, 10, 1)))
        b = mean(task_means(run_suite(fx, fn, FULL, 10, 1)))
        print(f"{name:20s} flawed {a:.3f}  fixed {b:.3f}  inflation {a - b:+.3f}")
    res = run_suite(fx, lambda: make_agent(MODEL_A), FULL, 20, 1)
    sc = [mean(r) for r in matrix(res)]
    print(f"planner slip 0.10 on the fixed suite: {mean(sc):.3f}, naive CI {tuple(round(v, 3) for v in ci95(sc))}, "
          f"clustered CI {tuple(round(v, 3) for v in ci95(sc, clusters_of(fx)))}")

    print("\n== 4. outcome score and process score")
    modal = modal_answers(fx)
    for name, fn in (("planner slip 0.1", lambda: make_agent(MODEL_A)), ("careless (path good, end 40% stale)", CarelessAgent),
                     ("lucky (modal answer, no tools)", lambda: LuckyAgent(modal))):
        res = run_suite(fx, fn, FULL, 10, 2)
        print(f"{name:36s} outcome {_fmt(mean(task_means(res)), 2)}  process {_fmt(mean(task_means(res, 'proc')), 2)}")

    print("\n== 5. reliability: pass@k and pass^k, 16 trials per task (planner slip 0.1, fixed suite)")
    res = run_suite(fx, lambda: make_agent(MODEL_A), FULL, 16, 3)
    cur = reliability_curves(matrix(res), [1, 2, 4, 8, 16])
    print("k      :", [1, 2, 4, 8, 16])
    print("pass@k :", [round(v, 3) for v in cur["pass_at_k"].values()])
    print("pass^k :", [round(v, 3) for v in cur["pass_hat_k"].values()])
    vc = variance_components(matrix(res))
    print("variance components of one trial's score:", {k: round(v, 4) for k, v in vc.items()})
    seeds = [mean(task_means(run_suite(fx, lambda: make_agent(MODEL_A), FULL, 1, 100 + s))) for s in range(12)]
    print(f"12 seeds, one trial per task: mean {_fmt(mean(seeds))}, sd {_fmt(stdev(seeds))}, range {_fmt(min(seeds))} to {_fmt(max(seeds))}")

    print("\n== 6. comparing two models: paired difference, clustered, and what difference the suite can detect")
    ra = task_means(run_suite(fx, lambda: make_agent(MODEL_A), FULL, 8, 4))
    rb = task_means(run_suite(fx, lambda: make_agent(MODEL_B), FULL, 8, 4))
    cl = clusters_of(fx)
    print("A", _fmt(mean(ra)), "B", _fmt(mean(rb)), "| paired, naive", {k: round(v, 3) for k, v in paired_difference(rb, ra).items()},
          "| paired, clustered", {k: round(v, 3) for k, v in paired_difference(rb, ra, cl).items()})
    print("smallest difference detectable (80% power) at the clustered SE:", _fmt(min_detectable_difference(paired_difference(rb, ra, cl)['se'])))

    print("\n== 7. scaffold sensitivity (A cheap, B careful and dear), and a cost limit")
    scs = {"full": FULL, "bare": BARE, "trunc": TRUNC, "nosearch": NOSEARCH}
    cmp_ = scaffold_comparison(fx, {"A": MODEL_A, "B": MODEL_B}, scs, 8, 5)
    for s, row in cmp_["scores"].items():
        print(f"{s:9s}", {m: round(v, 3) for m, v in row.items()}, "->", cmp_["ranking"][s])
    print("scaffold pairs that rank the models differently:", ranking_flips(cmp_["ranking"]))
    print("cost limit (per task): scores of A and B")
    for c in (4.0, 6.0, 8.0, 10.0, 14.0, 20.0):
        ab = [mean(task_means(run_suite(fx, lambda spec=spec: make_agent(spec), FULL, 8, 5, Limits(max_cost=c)))) for spec in (MODEL_A, MODEL_B)]
        print(f"  max_cost {c:5.1f}: A {ab[0]:.3f}  B {ab[1]:.3f}  {'A ahead' if ab[0] > ab[1] else 'B ahead'}")
    print("step-limit curve (planner slip 0.1, full scaffold):",
          {k: round(v, 3) for k, v in limit_curve(fx, MODEL_A, FULL, [1, 2, 3, 4, 5, 6, 8, 10]).items()})

    print("\n== 8. cost and accuracy (6 made-up models, 8 trials)")
    pts = cost_accuracy(fx)
    for n, (c, a) in pts.items():
        print(f"{n}: cost {c:6.2f}  accuracy {a:.3f}")
    print("frontier:", pareto_frontier(pts), "| convex hull points:", [(round(c, 2), round(a, 3)) for c, a in upper_hull(pts)])

    print("\n== 9. saturation and headroom (made-up timeline, 5% of items broken)")
    tl = toy_timeline(0)
    rep = saturation_report(tl["best"], tl["flaw_rate"], tl["n_items"])
    print("best score by month (every 4th):", [round(v, 3) for v in tl["best"][::4]])
    print({k: (round(v, 3) if isinstance(v, float) else v) for k, v in rep.items()})
    first = next(i for i, b in enumerate(tl["best"]) if b >= 0.9)
    fc = {e: round(e - 1 + months_to_target(tl["best"][:e], 0.9), 1) for e in (8, 12, 16, 20, 24)}
    print(f"month when the best score first reaches 0.9: {first}; straight-line forecasts of that month, made with the first e months: {fc}")

    print("\n== 10. Goodhart: tune shortcuts on the flawed suite, then score on both suites")
    g = goodhart_demo(fl, fx)
    for r in g["rows"]:
        print(f"slip {r['slip']:.2f} shortcuts {r['shortcuts']}: untuned flawed {r['untuned_flawed']:.3f} fixed {r['untuned_fixed']:.3f} | "
              f"tuned flawed {r['tuned_flawed']:.3f} fixed {r['tuned_fixed']:.3f}")
    print(f"correlation between flawed and fixed scores over these 8 agents: {g['corr']:.3f}")


if __name__ == "__main__":
    main()

Run the demo

$ python eval_fundamentals_toolkit.py
== 1. one task, one run: the 4-tuple (request, environment, stopping criteria, scorer)
request : Set the priority of ticket T-11 to low and close it.
steps   : [('set', ['T-11', 'priority', 'low'], 'OK T-11 priority=low'), ('get', ['T-11'], 'T-11 owner=alice status=pending priority=low locked=0'), ('set', ['T-11', 'status', 'closed'], 'OK T-11 status=closed'), ('get', ['T-11'], 'T-11 owner=alice status=closed priority=low locked=0')] | status submitted | answer 'done'
limits  : {'max_steps': 14, 'max_seconds': 120.0, 'max_cost': 40.0} | scorer: state strict | outcome 1.0 | process 1.0
  step limit 1: status step_limit, outcome 0.0, outcome if limit means zero 0.0
  step limit 2: status step_limit, outcome 1.0, outcome if limit means zero 0.0
  step limit 3: status submitted, outcome 1.0, outcome if limit means zero 1.0

== 2. audit of the flawed suite and of the fixed suite (trivial agents, oracle, specs)
flawed: 46 of 64 tasks flagged; {'dump_passes': 16, 'empty_passes': 8, 'gold_visible': 10, 'idk_passes': 12, 'no_limits': 13, 'vandal_passes': 20}
        baselines: {'empty': '0.12', 'idk': '0.19', 'dump': '0.25', 'guess': '0.38', 'peek': '0.28', 'refuse_all': '0.19'}
fixed : 0 of 64 tasks flagged; {}
        baselines: {'empty': '0.00', 'idk': '0.00', 'dump': '0.00', 'guess': '0.25', 'peek': '0.00', 'refuse_all': '0.12'}

== 3. what the planted flaws are worth to different agents (10 trials per task)
planner slip 0.04    flawed 0.964  fixed 0.964  inflation +0.000
planner slip 0.30    flawed 0.728  fixed 0.728  inflation +0.000
empty answer         flawed 0.125  fixed 0.000  inflation +0.125
says unknown         flawed 0.188  fixed 0.000  inflation +0.188
dumps everything     flawed 0.250  fixed 0.000  inflation +0.250
planner plus vandal  flawed 1.000  fixed 0.688  inflation +0.312
planner slip 0.10 on the fixed suite: 0.916, naive CI (0.895, 0.936), clustered CI (0.887, 0.944)

== 4. outcome score and process score
planner slip 0.1                     outcome 0.93  process 0.90
careless (path good, end 40% stale)  outcome 0.68  process 1.00
lucky (modal answer, no tools)       outcome 0.25  process 0.00

== 5. reliability: pass@k and pass^k, 16 trials per task (planner slip 0.1, fixed suite)
k      : [1, 2, 4, 8, 16]
pass@k : [0.916, 0.989, 1.0, 1.0, 1.0]
pass^k : [0.916, 0.843, 0.726, 0.565, 0.391]
variance components of one trial's score: {'between': 0.0045, 'within': 0.0725, 'total': 0.0771, 'within_share': 0.941}
12 seeds, one trial per task: mean 0.930, sd 0.031, range 0.891 to 1.000

== 6. comparing two models: paired difference, clustered, and what difference the suite can detect
A 0.924 B 0.975 | paired, naive {'diff': 0.051, 'se': 0.011, 'z': 4.601} | paired, clustered {'diff': 0.051, 'se': 0.01, 'z': 4.987}
smallest difference detectable (80% power) at the clustered SE: 0.029

== 7. scaffold sensitivity (A cheap, B careful and dear), and a cost limit
full      {'A': 0.91, 'B': 0.965} -> ['B', 'A']
bare      {'A': 0.893, 'B': 0.951} -> ['B', 'A']
trunc     {'A': 0.869, 'B': 0.922} -> ['B', 'A']
nosearch  {'A': 0.82, 'B': 0.861} -> ['B', 'A']
scaffold pairs that rank the models differently: []
cost limit (per task): scores of A and B
  max_cost   4.0: A 0.898  B 0.090  A ahead
  max_cost   6.0: A 0.910  B 0.867  A ahead
  max_cost   8.0: A 0.910  B 0.867  A ahead
  max_cost  10.0: A 0.910  B 0.955  B ahead
  max_cost  14.0: A 0.910  B 0.955  B ahead
  max_cost  20.0: A 0.910  B 0.965  B ahead
step-limit curve (planner slip 0.1, full scaffold): {1: 0.09, 2: 0.818, 3: 0.896, 4: 0.896, 5: 0.912, 6: 0.912, 8: 0.912, 10: 0.912}

== 8. cost and accuracy (6 made-up models, 8 trials)
a: cost   1.23  accuracy 0.832
b: cost   2.43  accuracy 0.902
c: cost   3.60  accuracy 0.930
d: cost   7.16  accuracy 0.963
e: cost  14.32  accuracy 0.963
f: cost   9.78  accuracy 0.885
frontier: ['a', 'b', 'c', 'd'] | convex hull points: [(1.23, 0.832), (2.43, 0.902), (3.6, 0.93), (7.16, 0.963), (14.32, 0.963)]

== 9. saturation and headroom (made-up timeline, 5% of items broken)
best score by month (every 4th): [0.03, 0.083, 0.208, 0.334, 0.646, 0.782, 0.876, 0.914, 0.936]
{'first_in_window': True, 'ceiling': 0.95, 'headroom': 0.003, 'saturated': True, 'first_saturated_step': 29, 'se_at_end': 0.016}
month when the best score first reaches 0.9: 27; straight-line forecasts of that month, made with the first e months: {8: 40.5, 12: 26.5, 16: 18.8, 20: 27.2, 24: 24.0}

== 10. Goodhart: tune shortcuts on the flawed suite, then score on both suites
slip 0.04 shortcuts ['list']: untuned flawed 0.961 fixed 0.961 | tuned flawed 0.966 fixed 0.841
slip 0.10 shortcuts ['list']: untuned flawed 0.888 fixed 0.888 | tuned flawed 0.911 fixed 0.786
slip 0.20 shortcuts ['list']: untuned flawed 0.786 fixed 0.786 | tuned flawed 0.828 fixed 0.703
slip 0.30 shortcuts ['list']: untuned flawed 0.708 fixed 0.708 | tuned flawed 0.763 fixed 0.638
correlation between flawed and fixed scores over these 8 agents: 0.785

Exercises

1. A new trivial agent. Write an agent that submits the text of the request. Add it to trivial_agents and run audit_suite on the flawed suite. Does it pass anything? What does this tell you about the substring scorers [1]?
2. An impossible task. Edit one task in the fixed suite so that its expected answer is wrong. Which flag does audit_suite raise? Why is the oracle the right tool, and why does it not find a task that is merely hard [1]?
3. Two meanings of a limit. Run limit_curve with limit_policy="score" and "zero" for the planner with slip 0.3. At which limit do they meet? Compare with slip 0.1. Which policy would you use for a benchmark with a hard time budget [2]?
4. The cost-limit crossover. Use Limits(max_cost=...) with values 8, 8.5, 9, 9.5 and 10. Find the limit where B overtakes A. Explain it from B's step cost of 3.0 and the number of steps most tasks need (the demo prints the steps of one task).
5. A path-tolerant process score. Write an agent that checks the same facts in a different order, and shows that process_score punishes it. Change the score so that it accepts any order of the same checks. What new loophole did you open [21]?
6. pass@k from few trials. Compute pass_hat_k at k = 8 from 8 trials and from 16 trials of the same agent, over 10 seeds. How much does the estimate vary? Compare with the plain "all 8 passed" count.
7. Make clusters matter. Change the owner hardness values in make_suite so that one owner is very hard. Compare se_naive and se_clustered. By what factor did the interval grow [12]?
8. Paired or not. For models A and B, compute the unpaired standard error of the difference and the paired one. Check Miller's identity that the paired variance equals the unpaired one minus twice the covariance, divided by n [12].
9. How many tasks? What n do you need to detect a difference of 0.02 at 80% power if the paired standard deviation of the per-task difference is 0.2? Use the 2.8 rule and check with min_detectable_difference. (My arithmetic: n = (2.8 × 0.2 / 0.02)² = 784.)
10. A new scaffold. Add a Scaffold with obs_chars=20 and no verify. Does it flip the ranking of A and B? If not, change B's slip rate until it does, and say what that tells you about how much of the ranking is the models and how much is the scaffold [16].
11. A seventh agent. Add an agent with cost 5.0 and accuracy 0.95 to GRID-style points. Which agents leave the frontier? Compute the convex hull and the cost of mixing to reach 0.95 [6].
12. The ceiling. In toy_timeline, change the broken share to 10%. How do the ceiling and the saturation month change? Make forecasts with three cut-offs and report the range, the way the demo does.
13. A hold-out for Goodhart. Make a third suite (a new seed, fixed scorers) and score the tuned agents on it. Is it close to the fixed suite? Now tune on the fixed suite instead. What happens [6] [18]?
14. Audit a real spec. Write five toy task specs for a real benchmark that you know (a request, an environment, limits, a scorer). Run the oracle and the trivial agents on them. Which checks does your toy not cover [1]?

The tests: test_eval_fundamentals_toolkit.py

The 124 tests check the shared helpers; the environment's tools, locks and the hidden gold; limits and the order of checks in a run (steps, then cost and time); every scorer type, strict and loose; the process score; the task suite (kinds, clusters, planted flaws, and the fixed suite); every scripted agent and the oracle on both suites; the auditor and its flags; the suite runner and its random streams; pass@k, pass^k, the variance split, the three standard errors, Wilson intervals, paired differences and the smallest detectable difference; scaffold comparison, limit curves and ranking flips; the Pareto frontier and the hull; best-so-far, the ceiling, the saturation report, the straight-line forecast and the toy timeline; the shortcut agent, the tuning and the Goodhart demo; a set of cases added after the mutation run; and the demo itself. Writing the demo found real faults in my first design, which I fixed: a planner that did not record a refusal when it verified its writes, so the oracle failed 8 tasks on the fixed suite; a vandal whose victim could be a locked or closed ticket, which made it a no-op; a "guess" flag and a "peek" flag that fired by chance on any suite, which I changed into a baseline score and a readable-gold flag; a gaming agent that gained nothing because the planner already passed most tasks, which I replaced with the coordinate-ascent tuning that picks a shortcut per kind of task; and a few wrong expectations in my own tests (the truncation length, float equality, and a slip rate that hardness scaled out of range). I also ran a mutation check, in which I made small deliberate changes to the code, one at a time, and ran the tests on each. This was a light, sampled run, not an exhaustive one (the toolkit has 513 places where such a change can be made, and I sampled 70 and then 110 of them at random, with a fixed seed). The first sample of 70 was caught 66 times. For the 4 misses I added tests (hull cases, the launch-window defaults, default settings, constants, the defaults of the shortcut agent, default seeds, and the wording of the step price and the request). Then I drew a second sample of 110 changes, and the tests caught 100. Of the 10 that survived, I judge 7 to be harmless or equivalent: the comparison "k > n" against "k >= n" in two places (k equal to n is a valid case that gives the same answer), a counter that adds 2 and not 1 when it only finds the most common answer, a "less than" against "less than or equal" on a continuous random draw, the audit's step limit of 20 against 21 (no agent needs more than a few steps), the sign of a term in the trend slope (it cancels because the values are centred), and a slip rate that is never used inside the tuning function. The other 3 are default arguments that no test relies on: the number of trials in the Goodhart demo, the number of trials in the scaffold comparison, and the seed in the tuning function. I did not add tests for them. That says the tests are sensitive to many changes. It does not say the code is correct.

$ python test_eval_fundamentals_toolkit.py -v
test_constants_are_the_documented_ones (__main__.TestAddedAfterMutationRun.test_constants_are_the_documented_ones) ... ok
test_default_launch_window_is_press_january_2025 (__main__.TestAddedAfterMutationRun.test_default_launch_window_is_press_january_2025) ... ok
test_default_seeds_are_zero (__main__.TestAddedAfterMutationRun.test_default_seeds_are_zero) ... ok
test_default_settings (__main__.TestAddedAfterMutationRun.test_default_settings) ... ok
test_fixed_agents_price_and_two_step_request (__main__.TestAddedAfterMutationRun.test_fixed_agents_price_and_two_step_request) ... ok
test_hull_drops_a_middle_point_under_the_line_and_keeps_one_above (__main__.TestAddedAfterMutationRun.test_hull_drops_a_middle_point_under_the_line_and_keeps_one_above) ... ok
test_mix_agent_defaults_and_price (__main__.TestAddedAfterMutationRun.test_mix_agent_defaults_and_price) ... ok
test_careless_agent (__main__.TestAgents.test_careless_agent) ... ok
test_dump_agent_uses_each_observation_once (__main__.TestAgents.test_dump_agent_uses_each_observation_once) ... ok
test_hardness_scales_the_slip (__main__.TestAgents.test_hardness_scales_the_slip) ... ok
test_make_agent (__main__.TestAgents.test_make_agent) ... ok
test_missing_search_tool_breaks_doc_tasks (__main__.TestAgents.test_missing_search_tool_breaks_doc_tasks) ... ok
test_mix_agent_uses_shortcuts_only_where_known (__main__.TestAgents.test_mix_agent_uses_shortcuts_only_where_known) ... ok
test_modal_answers (__main__.TestAgents.test_modal_answers) ... ok
test_no_retries_means_no_repair (__main__.TestAgents.test_no_retries_means_no_repair) ... ok
test_oracle_checks_writes_in_full_scaffold (__main__.TestAgents.test_oracle_checks_writes_in_full_scaffold) ... ok
test_oracle_solves_every_fixed_task_in_every_full_scaffold (__main__.TestAgents.test_oracle_solves_every_fixed_task_in_every_full_scaffold) ... ok
test_peek_agent (__main__.TestAgents.test_peek_agent) ... ok
test_refusal_tasks_with_and_without_verify (__main__.TestAgents.test_refusal_tasks_with_and_without_verify) ... ok
test_slip_one_always_goes_wrong (__main__.TestAgents.test_slip_one_always_goes_wrong) ... ok
test_trivial_agents (__main__.TestAgents.test_trivial_agents) ... ok
test_truncation_breaks_some_lookups (__main__.TestAgents.test_truncation_breaks_some_lookups) ... ok
test_vandal_agent_closes_a_spare_ticket (__main__.TestAgents.test_vandal_agent_closes_a_spare_ticket) ... ok
test_verification_repairs_a_slip (__main__.TestAgents.test_verification_repairs_a_slip) ... ok
test_audit_is_deterministic (__main__.TestAudit.test_audit_is_deterministic) ... ok
test_baselines (__main__.TestAudit.test_baselines) ... ok
test_each_flag_lands_on_the_right_kind (__main__.TestAudit.test_each_flag_lands_on_the_right_kind) ... ok
test_fixed_suite_is_clean (__main__.TestAudit.test_fixed_suite_is_clean) ... ok
test_flawed_suite_flags (__main__.TestAudit.test_flawed_suite_flags) ... ok
test_gold_visible_matches_leaky (__main__.TestAudit.test_gold_visible_matches_leaky) ... ok
test_no_limits_flag (__main__.TestAudit.test_no_limits_flag) ... ok
test_oracle_fails_flag (__main__.TestAudit.test_oracle_fails_flag) ... ok
test_pilot_outliers (__main__.TestAudit.test_pilot_outliers) ... ok
test_summary_of_empty (__main__.TestAudit.test_summary_of_empty) ... ok
test_trivial_agent_names (__main__.TestAudit.test_trivial_agent_names) ... ok
test_bad_arguments_do_not_crash (__main__.TestEnv.test_bad_arguments_do_not_crash) ... ok
test_env_copies_state (__main__.TestEnv.test_env_copies_state) ... ok
test_get (__main__.TestEnv.test_get) ... ok
test_list_and_count (__main__.TestEnv.test_list_and_count) ... ok
test_read_and_leaks (__main__.TestEnv.test_read_and_leaks) ... ok
test_scaffold_defaults (__main__.TestEnv.test_scaffold_defaults) ... ok
test_scaffold_tool_filter_and_truncation (__main__.TestEnv.test_scaffold_tool_filter_and_truncation) ... ok
test_search (__main__.TestEnv.test_search) ... ok
test_set_changes_state_and_obeys_locks (__main__.TestEnv.test_set_changes_state_and_obeys_locks) ... ok
test_demo (__main__.TestGoodhart.test_demo) ... ok
test_demo_pinned (__main__.TestGoodhart.test_demo_pinned) ... ok
test_pearson (__main__.TestGoodhart.test_pearson) ... ok
test_tune_ignores_a_shortcut_that_does_not_pay_on_a_fixed_suite (__main__.TestGoodhart.test_tune_ignores_a_shortcut_that_does_not_pay_on_a_fixed_suite) ... ok
test_tune_picks_the_list_shortcut (__main__.TestGoodhart.test_tune_picks_the_list_shortcut) ... ok
test_tune_with_no_tasks_of_a_kind (__main__.TestGoodhart.test_tune_with_no_tasks_of_a_kind) ... ok
test_mean_and_stdev (__main__.TestHelpers.test_mean_and_stdev) ... ok
test_normalize (__main__.TestHelpers.test_normalize) ... ok
test_sigmoid (__main__.TestHelpers.test_sigmoid) ... ok
test_cost_limit_stops_before_the_step_is_carried_out (__main__.TestLimitsAndEpisodes.test_cost_limit_stops_before_the_step_is_carried_out) ... ok
test_exactly_at_the_cost_limit_is_allowed (__main__.TestLimitsAndEpisodes.test_exactly_at_the_cost_limit_is_allowed) ... ok
test_limits_as_dict (__main__.TestLimitsAndEpisodes.test_limits_as_dict) ... ok
test_no_limits_means_no_stop (__main__.TestLimitsAndEpisodes.test_no_limits_means_no_stop) ... ok
test_step_limit_counts_the_submit (__main__.TestLimitsAndEpisodes.test_step_limit_counts_the_submit) ... ok
test_submitted_run_records_steps_cost_and_seconds (__main__.TestLimitsAndEpisodes.test_submitted_run_records_steps_cost_and_seconds) ... ok
test_task_limits_are_used_when_none_given (__main__.TestLimitsAndEpisodes.test_task_limits_are_used_when_none_given) ... ok
test_time_limit (__main__.TestLimitsAndEpisodes.test_time_limit) ... ok
test_main_prints_every_section (__main__.TestMain.test_main_prints_every_section) ... ok
test_cost_accuracy (__main__.TestPareto.test_cost_accuracy) ... ok
test_frontier (__main__.TestPareto.test_frontier) ... ok
test_upper_hull (__main__.TestPareto.test_upper_hull) ... ok
test_collateral_write_halves_it (__main__.TestProcessScore.test_collateral_write_halves_it) ... ok
test_empty_plan (__main__.TestProcessScore.test_empty_plan) ... ok
test_full_partial_and_none (__main__.TestProcessScore.test_full_partial_and_none) ... ok
test_order_matters (__main__.TestProcessScore.test_order_matters) ... ok
test_write_to_named_ticket_is_not_collateral (__main__.TestProcessScore.test_write_to_named_ticket_is_not_collateral) ... ok
test_ci95 (__main__.TestReliability.test_ci95) ... ok
test_clustering_widens_the_interval_when_clusters_agree (__main__.TestReliability.test_clustering_widens_the_interval_when_clusters_agree) ... ok
test_comb (__main__.TestReliability.test_comb) ... ok
test_curves_move_the_right_way (__main__.TestReliability.test_curves_move_the_right_way) ... ok
test_hat_never_exceeds_at (__main__.TestReliability.test_hat_never_exceeds_at) ... ok
test_independent_trials_match_the_closed_form (__main__.TestReliability.test_independent_trials_match_the_closed_form) ... ok
test_min_detectable_difference (__main__.TestReliability.test_min_detectable_difference) ... ok
test_paired_difference (__main__.TestReliability.test_paired_difference) ... ok
test_pass_at_k_and_hat_k (__main__.TestReliability.test_pass_at_k_and_hat_k) ... ok
test_se_clustered_is_a_real_number_for_small_samples (__main__.TestReliability.test_se_clustered_is_a_real_number_for_small_samples) ... ok
test_se_clustered_known_values (__main__.TestReliability.test_se_clustered_known_values) ... ok
test_se_naive (__main__.TestReliability.test_se_naive) ... ok
test_variance_components (__main__.TestReliability.test_variance_components) ... ok
test_variance_components_pinned (__main__.TestReliability.test_variance_components_pinned) ... ok
test_wilson (__main__.TestReliability.test_wilson) ... ok
test_changing_trials_keeps_earlier_streams (__main__.TestRunSuite.test_changing_trials_keeps_earlier_streams) ... ok
test_clusters (__main__.TestRunSuite.test_clusters) ... ok
test_limit_policy_flows_through (__main__.TestRunSuite.test_limit_policy_flows_through) ... ok
test_matrix_and_means (__main__.TestRunSuite.test_matrix_and_means) ... ok
test_oracle_mean_is_one (__main__.TestRunSuite.test_oracle_mean_is_one) ... ok
test_pinned_planner_scores (__main__.TestRunSuite.test_pinned_planner_scores) ... ok
test_shape_and_determinism (__main__.TestRunSuite.test_shape_and_determinism) ... ok
test_best_so_far_and_ceiling (__main__.TestSaturation.test_best_so_far_and_ceiling) ... ok
test_discrimination (__main__.TestSaturation.test_discrimination) ... ok
test_months_to_target (__main__.TestSaturation.test_months_to_target) ... ok
test_report_launch_window (__main__.TestSaturation.test_report_launch_window) ... ok
test_report_saturation (__main__.TestSaturation.test_report_saturation) ... ok
test_report_uses_the_larger_sample_for_a_smaller_se (__main__.TestSaturation.test_report_uses_the_larger_sample_for_a_smaller_se) ... ok
test_toy_timeline (__main__.TestSaturation.test_toy_timeline) ... ok
test_toy_timeline_pinned_and_report (__main__.TestSaturation.test_toy_timeline_pinned_and_report) ... ok
test_contains_all_ids_versus_id_set (__main__.TestScorers.test_contains_all_ids_versus_id_set) ... ok
test_contains_is_loose (__main__.TestScorers.test_contains_is_loose) ... ok
test_exact (__main__.TestScorers.test_exact) ... ok
test_ids_in (__main__.TestScorers.test_ids_in) ... ok
test_limit_policy_zero (__main__.TestScorers.test_limit_policy_zero) ... ok
test_number (__main__.TestScorers.test_number) ... ok
test_state_goal (__main__.TestScorers.test_state_goal) ... ok
test_state_refusal (__main__.TestScorers.test_state_refusal) ... ok
test_state_strict_checks_other_tickets_and_other_fields (__main__.TestScorers.test_state_strict_checks_other_tickets_and_other_fields) ... ok
test_unknown_scorer_raises (__main__.TestScorers.test_unknown_scorer_raises) ... ok
test_a_cost_limit_flips_the_ranking (__main__.TestSensitivity.test_a_cost_limit_flips_the_ranking) ... ok
test_limit_curve (__main__.TestSensitivity.test_limit_curve) ... ok
test_ranking_flips (__main__.TestSensitivity.test_ranking_flips) ... ok
test_scaffold_comparison_pinned (__main__.TestSensitivity.test_scaffold_comparison_pinned) ... ok
test_scaffold_comparison_structure (__main__.TestSensitivity.test_scaffold_comparison_structure) ... ok
test_deterministic (__main__.TestSuite.test_deterministic) ... ok
test_expected_answers_match_the_world (__main__.TestSuite.test_expected_answers_match_the_world) ... ok
test_flaws_only_in_flawed_suite (__main__.TestSuite.test_flaws_only_in_flawed_suite) ... ok
test_kind_counts_and_ids (__main__.TestSuite.test_kind_counts_and_ids) ... ok
test_leaky_only_on_lookup_count_doc (__main__.TestSuite.test_leaky_only_on_lookup_count_doc) ... ok
test_same_tasks_in_flawed_and_fixed (__main__.TestSuite.test_same_tasks_in_flawed_and_fixed) ... ok
test_scorer_types (__main__.TestSuite.test_scorer_types) ... ok
test_tasks_are_well_formed (__main__.TestSuite.test_tasks_are_well_formed) ... ok
test_world (__main__.TestSuite.test_world) ... ok

----------------------------------------------------------------------
Ran 124 tests in 5.004s

OK

5. Discussion questions

Use these in a study group or a class. Try to answer with an example and a source. Give each about 5 to 10 minutes.

1. Take a benchmark you know and write its four parts: request, environment, stopping criteria, scorer. Which part was hardest to find in the paper or the code? What does that say about how well it is documented [2]?
2. Press says to use one headline number, and HELM says to report seven metrics. Pick an agent benchmark and say who its readers are. Which advice fits, and where would you put the other metrics [2] [10]?
3. The ABC authors audited 10 benchmarks and found flaws in many. Is a benchmark with a known flaw still useful? What would you need to know about the flaw to keep using it [1]?
4. The lab's flaws did not change the score of a skilled planner, only of trivial and gaming agents. Does that make the flaws harmless? Think about a benchmark that is used to train agents as well as to rank them [18].
5. An agent solves a task by a path that the designer did not expect, and the scorer marks it as a failure (the τ²-bench example). Who is wrong, the agent or the scorer? How would you find such cases at scale [21]?
6. Your stopping rule says "20 steps". An agent that is stopped at step 20 has the right answer written but has not submitted. Which limit policy would you choose, and what would you tell your users [2]?
7. Kapoor and colleagues show simple baselines matching complex agents at lower cost. If a new benchmark's baseline is much weaker than a simple good scaffold, what would you do before you publish [6] [2]?
8. pass^k asks for consistency and pass@k asks for one success. Describe a product in which each one is the right metric, and one in which neither is [7].
9. A benchmark has 300 tasks that come from 10 repositories. List what the clusters could be and how you would estimate the correlation within them [12].
10. Two systems differ by 0.02 on a benchmark that can detect 0.03. What would you report, and what would you do next [12]?
11. The saturation rule in the lab uses a ceiling from broken items. How would you estimate the broken share in practice? What does your estimate cost [1] [2]?
12. Goodhart's law: name a benchmark whose score you think has been optimized beyond its usefulness. What evidence would convince you, and what hold-out would you build [6] [18]?

6. Quiz (10 questions)

Pick one answer for each question, then press Check answers. Check answers 0 of 10 answered.

1. In Press's 4-tuple, what does the scorer take as its input [2]?
   • The full text of the request together with the agent's final message, so that the scorer can check whether the reply answers what was asked
   • The stopping criteria alone, because the limits on steps, time and cost decide how many points a run gets before the environment is ever read
   • Only the list of tool calls that the agent made, since the scorer is meant to judge the path and not the state of the world at the end
   • The environment as it was when the agent exited, which the scorer then scores, whether the output is pass or fail, a number or a rating
2. How do Zhu and colleagues separate task validity from outcome validity [1]?
   • Task validity is about how many tasks there are, and outcome validity is about how many models were tested on the whole benchmark in the published report
   • Task validity asks if a task is solvable if and only if the agent has the skill, and outcome validity asks if the verdict matches whether it was solved
   • Task validity covers the prompt wording only, and outcome validity covers the cost of the run, since cost is what the paper treats as the final outcome
   • Task validity applies to benchmarks that use state, and outcome validity applies only to those that use text answers, so each benchmark has just one
3. A task has an independent 0.9 chance of passing on each trial. What is pass^8 (my arithmetic)?
   • About 0.43, from 0.9 multiplied by itself eight times, so asking for eight passes in a row is much harder than asking for one pass
   • About 0.90, because the chance of passing on any one trial does not change when you ask for eight passes in a row instead of just one
   • About 0.72, from one minus 0.1 times eight, since each added trial removes one tenth of the remaining chance that every trial passes
   • About 1.00, because with eight tries the chance that at least one passes is nearly certain, and that is what the metric measures
4. A benchmark has 64 tasks and the agent passes 90% of them. What is the standard error of the score, if the tasks are independent (my arithmetic)?
   • About 0.0014, from 0.9 times 0.1 divided by 64, without taking a square root of the result
   • About 0.0075, from the square root of 0.9 times 0.1, divided by the 64 tasks and then halved to allow for the paired design
   • About 0.0375, from the square root of 0.9 times 0.1 divided by 64, so the 95% interval is about plus or minus 0.07
   • About 0.30, from the square root of 0.9 times 0.1, since the number of tasks does not enter a standard error for a pass rate
5. Tasks come in groups of 8 and scores inside a group correlate at 0.3. By what factor does the standard error grow, by the usual design-effect rule (my arithmetic)?
   • By a factor of 3.1, because the variance is multiplied by 1 plus 7 times 0.3 and that variance is the same as the standard error here
   • By a factor of about 1.76, the square root of 3.1, where 3.1 is one plus seven times 0.3, the factor for the variance, not the error
   • By no factor at all, because the clustered standard error only differs from the naive one when scores within a group are identical
   • By a factor of about 0.57, since correlation within a group makes the scores more alike, and that makes the estimate more precise
6. Why must a benchmark say what a run that hits the step limit scores?
   • Because the limit is counted as a failed tool call, and the harness needs the number of failed calls to fill in the cost column of the report
   • Because the limit policy decides which scorer type is used, and a state scorer cannot be applied when a run has been stopped before it finished
   • Because a run that hits a limit has no state at all, so no scorer can read it, and the only possible score is therefore zero for every such run
   • Because the two choices, scoring leftover state or scoring zero, give different numbers, and they differ by up to 0.09 at some lab step limits
7. In the lab, agent e costs 14.32 and scores 0.963, and agent d costs 7.16 and scores 0.963. Why is e off the Pareto frontier?
   • Because d is cheaper at the same accuracy, so no one would pick e, and a point is dominated when another is no worse on both and better on one
   • Because it is the most expensive agent of the six, and the frontier only keeps agents whose price lies under the median of the whole set
   • Because its accuracy is not the highest of the group, and a frontier agent must have a higher accuracy than every agent that costs less than it does
   • Because two agents with equal accuracy are always both dropped, as a frontier is made only of agents whose accuracy differs from every other agent
8. The lab's "careless" agent has a process score of 1.00 and an outcome score of 0.68. What does this show?
   • That the process score is the better of the two scores, since it is higher, and the careless agent should therefore be ranked above the planner
   • That the outcome score is broken, because an agent that does every step right must always reach the right answer, and the scorer must have a bug
   • That a good path can still end badly, so a process score cannot replace the outcome, and it answers a different question about trust in a given run
   • That the two scores are always equal in a well-built benchmark, so the gap means that the suite has too few tasks to give a stable estimate of either
9. A benchmark has 200 items, 5% are broken, and the best score is 0.936. Using two standard errors of the score, from which score is it within noise of the ceiling (my arithmetic)?
   • From 0.95, because the ceiling itself is the first score at which a benchmark can be called saturated by this rule, whatever the noise
   • From about 0.915, which is the 0.95 ceiling minus two standard errors of 0.0173 each, where the standard error is taken at the score of 0.936
   • From about 0.85, because five percent of items are broken and so the benchmark is saturated ten points below the ceiling in every case
   • From 1.00, because only a perfect score on all of the items can show that a benchmark is saturated, and the broken items are not counted in that
10. What does the lab's Goodhart demo show?
    • That tuning to the flawed suite raised its score by 0.005 to 0.055 and lowered the fixed-suite score by 0.07 to 0.12, in a world that I built that way
    • That agents which tune to a flawed benchmark gain on both suites, because the shortcuts they find also help on the fixed suite by about the same amount
    • That tuning never matters for skilled agents, since the untuned planners score exactly the same on both suites and so show no sign of gaming at all
    • That the correlation between the two suites was zero, so a flawed benchmark cannot rank agents even roughly, as seen over all of the agents tested

Answer key

7. Graded assignment and rubric

Title: Design, audit and report a small agent benchmark on a mock.
Time: 5 to 6 hours. Total: 100 points. Work: solo.

The task

Extend the lab code and write a short memo (three pages at most). Everything is mock: no real agent, no real data, and no network. Build your own domain, such as a calendar assistant, a file organizer or an order-tracking desk. Do not reuse the lab's help-desk world or its seven task kinds. Label every made-up number as made up, and every calculation of your own as your own. Say what a reader must not conclude from a mock.

Part A. The 4-tuple harness (25 points)

• Write at least 30 tasks in your domain, each with a request, an environment (a state and at least three tools), stopping criteria and a scorer. Use at least three scorer types, one of them state-based.
• Give every task a step limit and one other limit (time or cost). Run the suite with both meanings of "the run hit a limit" and report both scores for three limit values.
• Write a process score for your domain. Show one agent for which it disagrees with the outcome score, and one case in which it punishes a valid path.
• State three limits of your harness.

Part B. An audit (25 points)

• Plant at least five flaws in a copy of your suite, covering at least three of these: a loose scorer, a readable answer, a task that is already done at the start, a missing limit, no check for collateral damage, an impossible task.
• Write an oracle and at least four trivial agents. Run them and report a table of the flags per flaw, with the number of tasks flagged before and after you fix the flaws.
• Find a flaw that your trivial agents do not catch. Explain why, and say what you would add.
• Show what the planted flaws are worth to a skilled agent and to a trivial one, as the lab does, and say which result surprised you.
• State three limits of your audit.

Part C. Reliability and sensitivity (25 points)

• For one scripted agent, run 16 trials per task and report pass@k and pass^k at k = 1, 2, 4, 8, 16, with the variance split into between-task and within-task parts.
• Give naive, clustered and paired standard errors (for two agents) with your own clusters, and the smallest detectable difference at 80% power. Work one of them by hand.
• Compare at least three scaffold variants and at least two cost limits for two agents. Report whether the ranking flips, and at which setting.
• Plot or tabulate the cost and accuracy of at least five agents, mark the Pareto frontier, and compute one mix on the convex hull by hand.
• State three limits of your reliability study.

Part D. Saturation, Goodhart and the memo (25 points)

• Build a made-up timeline in which the best score rises over 30 months or more, with a share of broken items. Report the ceiling and the first month flagged as saturated. Make straight-line forecasts from three cut-offs and report the spread.
• Show a Goodhart effect: tune an agent to your flawed suite, and score it on the fixed suite and on a third suite that it has never seen. Say whether the effect you found matches the lab's and why.
• Write a memo of two pages at most to a team that wants to use the benchmark: what claims it supports, what claims it does not, what difference it can detect, what you would fix first, and a release plan (hold-out set, how you would watch for saturation and contamination).
• State three limits of your memo and your toy.

What to hand in

• Your eval_fundamentals_toolkit.py (or your own file names), your mock tasks, and the tests. All tests must pass.
• A memo in PDF or Markdown with the tables from Parts A to D and the memo from Part D.
• A sources list for any reading you cite.

Rubric

Criterion	Full marks	Partial	Points
A. The 4-tuple harness	30 or more tasks in a new domain with all four parts, three scorer types including state, two limits per task, both limit policies at three values, a process score with one disagreement and one unfair penalty, three limits.	Fewer tasks, or two scorer types, or one limit policy, or no process score, or no unfair-penalty case.	25
B. An audit	Five or more planted flaws across three kinds, an oracle and four trivial agents, a flag table before and after, a flaw that is not caught with an explanation, the value of the flaws to a skilled and a trivial agent, three limits.	Fewer flaws, or no oracle, or no after-fix table, or no missed flaw, or no comparison of agents.	25
C. Reliability and sensitivity	16 trials with the pass@k and pass^k table and the variance split, three standard errors with a hand calculation, the smallest detectable difference, three scaffolds and two cost limits with a ranking result, a Pareto plot and a hull mix worked by hand, three limits.	Fewer trials, or no clusters, or no hand calculation, or one scaffold, or no frontier, or no mix.	25
D. Saturation, Goodhart and memo	A timeline with a ceiling and a saturation month, forecasts from three cut-offs, a Goodhart effect scored on a third suite, a two-page memo with supported and unsupported claims, a detectable difference, a first fix and a release plan, three limits.	No ceiling, or one forecast, or no third suite, or a memo that overclaims or has no release plan.	25
Total			100

Grade bands: 90 to 100 excellent. 75 to 89 good. 60 to 74 needs work. Under 60: redo and resubmit. Plain writing is rewarded: short sentences, no filler. Label your own arithmetic as your own.

8. Glossary

Term	Meaning
ABC (Agentic Benchmark Checklist)	The checklist of Zhu and colleagues, with task validity (T), outcome validity (O) and reporting (R) items [1].
Ceiling	The best score that is possible when some items are broken: one minus the broken share (my definition for the lab).
Clustered standard error	A standard error that allows for scores of tasks in the same group being correlated. It is larger than the naive one when they are [12].
Contamination	Test items that appear in a model's training data [20].
Design effect	The factor by which clustering inflates the variance, 1 + (m − 1)ρ for clusters of size m and within-cluster correlation ρ (2.9).
Environment	The state and the tools that the agent acts in. The second part of the 4-tuple [2].
Goodhart's law	When a measure becomes a target, it stops being a good measure [18].
Harness	The code that runs tasks, records runs and scores them [21].
Headroom	The distance between the best score and the ceiling.
Oracle solver	A script that does every task correctly. If it fails a task, the task is broken [1].
Outcome validity	The scorer's verdict matches whether the task was solved [1].
Outcome score and process score	The first scores the end state, and the second scores the path (2.3).
Paired difference	The difference between two systems computed task by task, which removes the variation that both share [12].
pass@k	The chance that at least one of k trials passes [17].
pass^k	The chance that all k trials pass [7].
Pareto frontier	The set of agents for which no other agent is at least as cheap and at least as accurate, with one strictly better.
Request	What the agent is asked to do. The first part of the 4-tuple [2].
Saturation	A benchmark where the best score is so near the ceiling that it can no longer separate systems [2] [21].
Scaffold	The prompts, tools and rules around a model that make it an agent [21].
Scorer	The code that reads the final environment and gives a score. The fourth part of the 4-tuple [2].
Shortcut	A way to pass a task without the capability the task is meant to test [1].
Stopping criteria	How a run ends: a submit, or a limit on steps, time or cost. The third part of the 4-tuple [2].
Task validity	A task can be solved if and only if the agent has the target capability [1].
Time horizon (METR)	The length of task, in human time, that a model completes with 50% success [13].
Trivial agent	An agent with no skill (empty, unknown, dump, peek, vandal) used to find shortcuts. If it passes a task, the task has a flaw (2.4).
Wilson interval	A confidence interval for a pass rate that stays sensible near 0 and 1 and for small n.

9. References

Entries marked (required) are the course's required reading, and (course, additional) is the course's additional reading. Entries marked (added) are my own additions. The course site and the slide test, [4] and [5], are the course's own pages. "Abstract level" means that I read the abstract and not the whole paper. Where an entry names more, that is what I read. Web pages were read in October 2026.

• [1] Zhu, Y., Jin, T., Pruksachatkun, Y., Zhang, A., and many others (including S. Kapoor and J. Steinhardt). "Establishing Best Practices for Building Rigorous Agentic Benchmarks." arXiv:2507.02825 (2025). https://arxiv.org/abs/2507.02825 (I read the abstract, the introduction, the checklist (section 4), the findings (section 5) and the conclusion, and skimmed the appendices; I read the arXiv HTML text; the numbers are the authors' own, from experiments they designed, and I did not rerun any; the authors of the audited benchmarks may see the flaws differently; Kapoor is also an author of [6]) (required)
• [2] Press, O. "How to Build Good Language Modeling Benchmarks." Blog post, 7 August 2024, with edits dated January 2025, May 2025 and June 2026. Press co-wrote SWE-bench [8] with John Yang, who teaches the course. https://ofir.io/How-to-Build-Good-Language-Modeling-Benchmarks/ (I read the whole post as it stood on 29 October 2026; it is an opinion post with no experiments, and the claim about mini-SWE-agent is about a tool from the author's own group) (course, additional)
• [3] Polo, F. M., Weber, L., Choshen, L., Sun, Y., and 2 others. "tinyBenchmarks: evaluating LLMs with fewer examples." ICML 2024. arXiv:2402.14992 (2024). Also cited in Week 7a. https://arxiv.org/abs/2402.14992 (abstract level only for this week; the authors' own claims) (course, additional)
• [4] Stanford CS329Z: Engineering AI Agents, course site and schedule (Diyi Yang, Michael Ryan, John Yang). https://cs329z.stanford.edu/ (the course's own page; the topics and the readings for the Wednesday, 4 November 2026 lecture "Evaluation for Agentic Systems: Evaluation Fundamentals & Benchmark Design" come from it, and it says the schedule is tentative)
• [5] CS329Z slide address lecture13.pdf, tried once on 29 October 2026. https://web.stanford.edu/class/cs329z/slides/lecture13.pdf (the address returned "not found" (HTTP 404), and the slide folder's index listed only lecture01.pdf to lecture11.pdf, so I could not use slides for this week)
• [6] Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., and Narayanan, A. "AI Agents That Matter." arXiv:2407.01502 (2024). Reused from Weeks 1 and 3b. https://arxiv.org/abs/2407.01502 (I read the abstract, the introduction and sections 2 and 5, and the generality table; the authors' own claims; Kapoor is also an author of [1] and [16]) (added)
• [7] Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045 (2024). Reused from earlier weeks. https://arxiv.org/abs/2406.12045 (I read the abstract and the definitions of pass^k and pass@k; the authors' own claims) (added)
• [8] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" arXiv:2310.06770 (2023). John Yang and Ofir Press are authors; John Yang teaches the course. https://arxiv.org/abs/2310.06770 (abstract level; the authors' own claims) (added)
• [9] OpenAI. "Introducing SWE-bench Verified." Blog post, 13 August 2024, updated 24 February 2025. Vendor-authored; the author list includes John Yang and Carlos E. Jimenez, who are also authors of SWE-bench [8]. https://openai.com/index/introducing-swe-bench-verified/ (read in full for Week 7a and not reread for this week; the numbers are OpenAI's own claims) (added)
• [10] Liang, P., Bommasani, R., Lee, T., Tsipras, D., and many others. "Holistic Evaluation of Language Models." arXiv:2211.09110 (2022). https://arxiv.org/abs/2211.09110 (abstract level; the authors' own claims) (added)
• [11] Reuel, A., Hardy, A., Smith, C., Lamparth, M., Hardy, M., and Kochenderfer, M. J. "BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices." arXiv:2411.12990 (2024). https://arxiv.org/abs/2411.12990 (abstract level; the authors' own claims) (added)
• [12] Miller, E. "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations." arXiv:2411.00640 (2024). The author works at Anthropic. Also cited in Week 6a. https://arxiv.org/abs/2411.00640 (I read the body text (sections 1 to 5) and not the appendix derivations; the formulas in the lab follow its Equations 1 to 4 and the paired and power sections, as I read them) (added)
• [13] Kwa, T., West, B., Becker, J., Deng, A., and many others. "Measuring AI Ability to Complete Long Software Tasks." arXiv:2503.14499 (2025). https://arxiv.org/abs/2503.14499 (abstract level; the authors' own claims and limits) (added)
• [14] Zhou, S., Xu, F. F., Zhu, H., Zhou, X., and 8 others. "WebArena: A Realistic Web Environment for Building Autonomous Agents." arXiv:2307.13854 (2023). https://arxiv.org/abs/2307.13854 (abstract level; the authors' own claims; the success rates are for the agents tested at that time) (added)
• [15] Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. "GAIA: a benchmark for General AI Assistants." arXiv:2311.12983 (2023). https://arxiv.org/abs/2311.12983 (abstract level; the authors' own claims) (added)
• [16] Kapoor, S., Stroebl, B., Kirgis, P., Nadgir, N., and many others. "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation." arXiv:2510.11977 (2025). https://arxiv.org/abs/2510.11977 (abstract level; the authors run the leaderboard and report on it, so the claims are their own) (added)
• [17] Chen, M., Tworek, J., Jun, H., Yuan, Q., and many others. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374 (2021). https://arxiv.org/abs/2107.03374 (abstract level; the authors' own claims; the source of the Codex and HumanEval numbers and of the pass@k estimate) (added)
• [18] Manheim, D., and Garrabrant, S. "Categorizing Variants of Goodhart's Law." arXiv:1803.04585 (2018). https://arxiv.org/abs/1803.04585 (abstract level; a conceptual paper with no experiments) (added)
• [19] Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., and many others. "Dynabench: Rethinking Benchmarking in NLP." arXiv:2104.14337 (2021). https://arxiv.org/abs/2104.14337 (abstract level; the authors' own claims) (added)
• [20] Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B. "Proving Test Set Contamination in Black Box Language Models." arXiv:2310.17623 (2023). https://arxiv.org/abs/2310.17623 (abstract level; the authors' own claims) (added)
• [21] Anthropic engineering. "Demystifying evals for AI agents." Blog post, 9 January 2026. Vendor-authored; written by Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe. The course may use it in the next lecture. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents (I read the whole post; it describes the authors' own practice and has no experiments) (added)
• [22] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., and 6 others. "Let's Verify Step by Step." arXiv:2305.20050 (2023). https://arxiv.org/abs/2305.20050 (abstract level; a training paper, used here as an analogy only; the authors' own claims) (added)
• [23] Uesato, J., Kushman, N., Kumar, R., Song, F., and 5 others. "Solving math word problems with process- and outcome-based feedback." arXiv:2211.14275 (2022). https://arxiv.org/abs/2211.14275 (abstract level; a training paper, used here as an analogy only; the authors' own claims) (added)