TwinEdgeAI

↑ Course home

← Previous: Week 7b, Evaluation Fundamentals & Benchmark Design  ·  Next: Week 8b, Agent Safety & Guardrails →

Engineering AI Agents · Week 8a

LLM-as-Judge & Evaluation Infrastructure

Who grades the agent? Code, a model, or a person; how to write a judge prompt; how judges go wrong (position, length, self-preference); pairwise against pointwise scoring; how to check a judge against people; why pass^k falls when the grader is noisy; and how to build a harness that you can rerun.

Author: Grok Bot Version: 1.0, October 29, 2026 Study time: about 16 hours

The slides were not availableThis lecture is on Wednesday, November 11, and I wrote this pack on Thursday, October 29, 2026. I checked the course's slide folder once. Its index listed lecture01.pdf to lecture11.pdf, and the addresses I expect for this lecture, lecture14.pdf and lecture15.pdf, both returned "not found" (HTTP 404) [6]. So I have not seen the slides. This pack is built from the schedule's description of the lecture, the three required readings, the additional reading, and sources I found myself. The real lecture may stress other things. Where I guess what the lecture covers, I say so. Check the slides when they appear.

A date to checkI was asked to prepare "Week 8, Monday, November 9". The course schedule lists Monday, November 9 as a fireside chat with Andrew Ng, and lists this lecture, "LLM-as-Judge & Evaluation Infrastructure", on Wednesday, November 11 [5]. I used November 11. The schedule says that it is tentative.

CreditThis coursework is adapted from, and inspired by, Stanford CS329Z: Engineering AI Agents (Diyi Yang, Michael Ryan, John Yang), Week 8, Lecture "LLM-as-Judge & Evaluation Infrastructure" (Wednesday, November 11). The topic list and the readings come from the course's public schedule [5]. The schedule describes the lecture as: the three grader types, designing judge prompts, known biases, pairwise versus pointwise evaluation, non-determinism metrics (pass@k versus pass^k), harness design, and Anthropic's 8-step roadmap. The notes, lab, quiz and assignment are original writing, and the lab uses its own examples and code. This pack does not reproduce any slides. It is not an official Stanford document. Each source I added is marked "added" in the references.

1. Week overview and learning objectives

Week 7b asked how to build a benchmark whose score means something. This week asks who, or what, does the grading, and how you build the machinery that runs it. The schedule lists seven topics [5]. I read them as three jobs. The first is choosing a grader: code, a model, or a person, and how to write the prompt if it is a model. The second is trusting the grader: the ways in which a judge model is biased, how to measure the bias, how to reduce it, and how to check the judge against people. The third is running it reliably: what pass@k and pass^k mean when both the agent and the grader are noisy, how to design a harness, and what the Anthropic post says to do first. The first required reading is that post, by Grace and colleagues [1]. The second, by Zheng and colleagues, is the paper that named the biases and measured how well a strong model agrees with people [2]. The third, AutoMetrics, is about building a metric from a few human labels when you do not know in advance what to measure [3]. The additional reading, AutoLibra, does the same from open-ended feedback [4].

The lab uses the standard library only. It does not call any model, any API or the network. It builds a toy world with five made-up "systems" that give 300 answers to 60 questions. Each answer has a hidden quality that the "human" sees with a little noise. The "judges" are not language models. Each is a rule that sees the hidden quality plus noise, with bias knobs that I set by hand: a taste for the first slot, a taste for length, a taste for one system. The lab then measures each bias, tries the standard fixes, ranks the five systems by pointwise scores and by pairwise comparison with a Bradley-Terry fit, checks the judge against the human labels with kappa, true positive and true negative rates, and a calibration curve, shows how a noisy grader changes pass@k and pass^k, builds a small version of the AutoMetrics idea, and ends with an eval harness that has seeds, a cache, retries and a log. Everything it prints is from made-up data. It shows how the methods work. It says nothing about how real judges behave. In several places the toy gives a weak or even opposite result to a paper, and I report that.

Learning objectives

After this week you should be able to:

1. Name the three grader types, give one strength and one weakness of each, and say when to use which. [2.1]
2. Write a judge prompt with one criterion, a written reason before the verdict, and a few worked examples, and say why a pass or fail verdict is easier to check than a score from 1 to 5. [2.2]
3. Name the biases of LLM judges (position, verbosity, self-enhancement, weak reasoning) and describe one test for each. [2.3]
4. Reduce position bias by swapping, and say what swapping costs; explain why length control and a panel of judges address different problems. [2.4]
5. Compare pointwise and pairwise evaluation on cost, resolution and bias, and fit Bradley-Terry strengths from pairwise results. [2.5]
6. Compute percent agreement, Cohen's kappa and a true positive and true negative rate, and explain why agreement can mislead when failures are rare. [2.6]
7. Read a calibration curve and say what it does and does not fix. [2.6]
8. Compute pass@k and pass^k, and say what a noisy grader does to each. [2.7]
9. Explain how AutoMetrics and AutoLibra turn a little human feedback into a metric, and what the lab's toy can and cannot show about them. [2.8]
10. Design an eval harness with seeds, a cache, retries and a log, and say what must change the cache key. [2.10]
11. Walk through the nine steps of the Anthropic roadmap and say which ones this week's lab touches. [2.11]

How to use this pack

Time	What to do
4 hours	Read the notes (section 2). Skim the figures first.
4 hours	Read the Anthropic post and the Zheng paper with the reading guide questions. Skim AutoMetrics and AutoLibra. Skim the optional ones that match your project.
4.5 hours	Do the lab and at least five exercises.
1.5 hours	Discussion and the quiz.
2 hours	The graded assignment.

2. Lecture notes

The slides were not available (see the top of the page), so these notes follow the schedule's seven topics [5] and the readings. The section titles are mine.

2.1 Three kinds of grader

Week 7b was about the scorer as one part of a task. This week splits the scorer into three kinds. The Anthropic post says that agent evaluations usually combine code-based, model-based and human graders, and that each one grades some part of the transcript or the outcome [1]. The lecture lists the same three [5]. The table is my summary of the post's own table, with the lab's Figure 1 as the picture.

Type	Methods the post lists	Strengths	Weaknesses
Code-based	string match (exact, regex, fuzzy); binary tests; static analysis; outcome verification; tool-call checks; transcript analysis (turns, tokens)	fast, cheap, objective, reproducible, easy to debug	brittle to valid variations; no nuance; limited for subjective tasks
Model-based	rubric scoring; natural-language assertions; pairwise comparison; reference-based evaluation; multi-judge consensus	flexible, scalable, captures nuance, handles open-ended and free-form output	non-deterministic; more expensive than code; needs calibration against humans
Human	expert review; crowdsourced judgment; spot checks; A/B tests; inter-annotator agreement	the gold standard; matches expert users; used to calibrate model graders	expensive, slow, often needs experts at scale

The post's advice on choosing is short: use deterministic graders where possible, LLM graders where necessary or for flexibility, and human graders "judiciously" for validation [1]. It adds three points that matter in practice. First, grade what the agent produced, not the path it took. A check for an exact order of tool calls is "too rigid" because agents find valid paths that the designer did not expect. Second, build in partial credit for tasks with several parts: a support agent that finds the problem and verifies the customer but fails to refund is better than one that fails at once. Third, a task can combine graders in a weighted score (the combined score must reach a threshold), a binary one (all graders must pass), or a hybrid [1]. The post also separates capability evals, which should start at a low pass rate and give a hill to climb, from regression evals, which should pass nearly always and warn you of backsliding. A capability eval with a high pass rate can "graduate" to a regression suite [1].

Inspect, the open-source evaluation framework from the UK AI Security Institute, shows the same split in its scorers. Its built-in text scorers include includes() (is the target in the output), match() (is the target at a known position) and pattern() (extract an answer with a regular expression), and it also ships model-graded scorers such as model_graded_qa() [18]. OpenAI's open-source evals repository describes itself as a framework for evaluating LLMs and LLM systems and a registry of benchmarks [20]. I read only the front pages of these two, not the code.

In the lab. A code grader (number_close) that checks only whether the final number is right finds the right answer in 189 of 300 toy responses. The "human" in the toy passes 151, because humans in the toy also reward shown steps and a cited source and punish hedging. So the code grader agrees with the human pass label on 86% of responses (kappa 0.72). That is not a flaw in the code grader. It checks one thing perfectly (it matches the hidden correctness trait on all 300) and is blind to the others. This is the usual trade: a code grader is exact about what it checks.

2.2 Designing a judge prompt

A judge prompt is a small program written in English. Five sources agree on most of what it should contain, and they differ on a few points. I summarize them in the order of how much they say about the prompt itself.

Husain's guide. Hamel Husain's guide to LLM judges proposes what he calls "critique shadowing" [7]. He says that it comes from helping over 30 companies set up evaluation, and the page advertises his paid course on evals. So this is practitioner advice and not an experiment. The steps I take from it: (1) find one principal domain expert whose judgment defines quality, instead of a committee; (2) build a diverse dataset; (3) have the expert give a binary pass or fail verdict with a written critique for each example; (4) fix pervasive errors before building a judge; (5) build the judge prompt iteratively, using the expert's critiques as few-shot examples; (6) read the errors; (7) build narrower judges if needed. He argues against scores from 1 to 5, because nobody can say what separates a 3 from a 4 and different raters use the scale differently [7]. He also describes how to validate a judge: treat it as a binary classifier, split the labeled examples into train, dev and test sets, keep the dev examples out of the prompt, freeze the prompt, and run the test set once. He reports a split of 10 to 20 percent for the prompt's examples and 40 to 45 percent each for dev and test, with 30 to 50 examples of each label in dev and test, and he says to report true positive and true negative rates and not only agreement [7].

G-Eval. Liu and colleagues ask the judge to follow evaluation steps (a chain of thought) and then fill in a form with the score. They report a Spearman correlation of 0.514 with humans on summarization with GPT-4, better than earlier methods. They also warn that LLM-based evaluators may be biased toward LLM-written text [8]. I read the abstract only.

Prometheus. Kim and colleagues train an open 13B model to grade with a user-written score rubric and a reference answer. They report a Pearson correlation of 0.897 with human evaluators on 45 custom rubrics, against 0.882 for GPT-4 and 0.392 for ChatGPT [9]. Prometheus 2 handles both direct assessment (a score) and pairwise ranking, with user-defined criteria [10]. JudgeLM fine-tunes judges of 7B to 33B parameters and uses swap augmentation (train on both orders), reference support and reference drop; it reports agreement with the teacher judge above 90% [11]. These three papers are by the authors of the models, and I read the abstracts only.

The Anthropic post says that vague rubrics produce inconsistent judgments (the same is true of vague tasks), that you should give the judge a way out such as returning "Unknown" when it lacks the information, and that it can help to grade each dimension with a separate, isolated judge and not one call for everything [1]. Shankar and colleagues found criteria drift: people need criteria to grade outputs, but grading outputs changes their criteria [16]. So the first rubric is a draft. Plan to revise it after you have seen outputs.

Here is a made-up judge prompt that follows the points above. It is an example for the exercises, not a tested prompt.

You are grading one answer from a customer-support assistant.
Criterion (only this one): "The answer states the refund amount that the
policy allows, and does not promise more."

Policy excerpt: {policy}
Question: {question}
Answer to grade: {answer}

First write 2 to 4 sentences of reasoning that quote the answer.
If you cannot tell from the text, say UNKNOWN instead of guessing.
Then write one line: VERDICT: PASS, FAIL or UNKNOWN.

Example (FAIL): the answer said "we will refund the full price" but the
policy allows 80 percent. Reasoning: it promises more than the policy.
Example (PASS): ...

Five things in it are on purpose. One criterion only. A written reason before the verdict, which is the chain of thought that G-Eval and Zheng and colleagues use [8] [2]. A binary verdict with an escape hatch. Worked examples of both outcomes. And the reference (the policy) in the prompt, so that the judge does not have to know it. The rest of this week is about how to find out whether such a prompt works.

2.3 Known biases of LLM judges

A judge model is a model, and it has habits. Zheng and colleagues name four: position bias, verbosity bias, self-enhancement bias and limited ability to grade math and reasoning [2]. Others add to the list. I give the paper's numbers, and I say which are from the paper's own tests. All of them are for the models and prompts of 2023.

Bias	What it is	What the sources found
Position	The verdict depends on which answer is shown first.	Zheng and colleagues swapped the order of two answers in their position-bias test. A judge was consistent if the same answer won in both orders. GPT-4 was consistent 65.0% of the time (66.2% if the assistants were renamed "Assistant A" and "B" in the prompt), and it favored the first answer in 30.0% of cases. Claude-v1 was consistent 23.8% of the time and favored the first answer in 75.0% of cases [2]. Wang and colleagues show that changing the order of appearance can flip a ranking: Vicuna-13B could beat ChatGPT on 66 of 80 tested queries with ChatGPT as the evaluator [12]. JudgeLM names position bias too [11].
Verbosity	The judge prefers the longer answer even when it adds nothing.	Zheng and colleagues took 23 answers with numbered lists and added a rephrased copy of the list at the start. Claude-v1 and GPT-3.5 judged the padded answer better 91.3% of the time, and GPT-4 8.7% of the time [2].
Self-enhancement	A judge favors answers from its own model.	Zheng and colleagues saw GPT-4 favor itself with a win rate 10 points higher than humans gave it, and Claude-v1 25 points higher, but GPT-3.5 did not favor itself, and the judges also favored other models. The paper says that, because of limited data and small differences, its study cannot determine whether the models have a self-enhancement bias [2]. Panickssery and colleagues test a mechanism: models can recognize their own text, and when they fine-tune a model, the stronger its self-recognition, the stronger its self-preference, in a roughly linear way [13].
Weak reasoning	The judge accepts a wrong math or logic answer.	On 10 math questions, GPT-4 as a judge failed (said a wrong answer was right) on 14 of 20 comparisons with the default prompt [2].
Knowledge and format	A fine-tuned judge is swayed by what it knows or by how the answer looks.	JudgeLM lists position, knowledge and format biases in fine-tuned judges [11]. I read the abstract only.

Two cautions. First, the sources measure the biases with designed tests: swap the order, pad the answer, compare win rates. Each test isolates one thing, which is the reason it works, and also the reason a real run has the biases mixed together. Second, the sizes depend on the judge. In Zheng's tables the strongest judge, GPT-4, was far less hurt by padding than the other two, and the paper says that its self-enhancement data is too limited to settle the question [2].

In the lab. The "fair" mock judge sees each answer's hidden quality plus noise. The "first" judge adds 0.40 to the answer in the first slot. Over 600 pairs, each asked in both orders, the fair judge keeps the same winner 77.2% of the time and picks the first slot in 50.1% of non-tie verdicts. The first-biased judge keeps the winner 63.5% of the time and picks the first slot 65.2% of the time. Even the fair judge is not fully consistent (77.2%), because its noise and its tie margin flip close calls. So "consistency" mixes bias with noise, and a consistency below 100% is not proof of position bias. The first-slot rate is the cleaner test, since random noise should give 50%. For length, an answer is compared with a copy of itself that has four filler sentences added. The fair judge gives the padded copy 44.0% of the credit (a tie counts as half), and the length-biased judge gives it 84.5%. For self-preference, the judge's own system (the weak, wordy "epsilon") wins 30.1% of its pairs with no bias switched on, against a human win rate of 26.7% (a gap of +0.034). With a self bias of 0.35 it wins 42.8% (a gap of +0.161). Note that the fair judge is below 50% for the padded copy: its noise makes it call some near-ties for the original. A fair rate is not always exactly 0.5.

2.4 Measuring and reducing bias

For each bias there is a standard fix, and each fix has a cost. The table lists the fixes from the sources and, in the last column, what the lab found.

Problem	Fix and source	Cost or limit	Lab result (made up)
Position	Swap the order and call it a win only if the same answer wins both times, else a tie (Zheng's "conservative" approach) [2]. Or randomize the order. Or aggregate several orders (Wang's balanced position calibration) [12]. Or train on both orders (JudgeLM's swap augmentation) [11].	Two calls per pair, and more ties.	Position-biased judge: wrong verdicts fell from 82 to 7 of 600 pairs, and ties rose from 45 to 223.
Consistency	Few-shot examples. Zheng and colleagues raised GPT-4's consistency from 65.0% to 77.5% [2].	A longer prompt. The paper says the calls cost about four times as much, and that high consistency need not mean high accuracy [2].	Not tested.
Reasoning	Have the judge solve the problem first (chain of thought), or give it a reference answer. On the 10 math questions, failures fell from 14 of 20 (default) to 6 of 20 (chain of thought) to 3 of 20 (reference-guided) [2]. Prometheus also uses a reference answer [9].	You need a reference, or a strong judge.	Not tested. The toy judge does not read the text.
Length	Length-controlled win rate. Dubois and colleagues fit a model that predicts the judge's preference from the length difference and other features, then predict the preference at zero length difference. They report that it raised the Spearman correlation with Chatbot Arena from 0.94 to 0.98 [15].	A model of the bias that may be wrong; it needs many comparisons.	Wordy judge, epsilon against delta: raw win rate 0.550, length-controlled 0.216, human 0.300.
Self-preference, and bias in general	Use several judges (a multi-judge consensus is one of the post's model-based methods [1]). Dropping a judge's vote on its own model is my addition.	More calls and more cost. Judges can share a bias.	Panel of four, swapped, no votes on own system: 513 right, 31 wrong, 56 ties of 600.
Hard cases	Send them to a person. Wang's human-in-the-loop calibration measures how hard an example is from the spread across orders and asks for human help when needed [12].	Human time.	Not tested.

The lab shows four things that the table hides, and I report them as they came out.

1. Swapping helps every judge, not only the biased one. The position-biased judge went from 82 wrong verdicts to 7. But the fair judge went from 81 wrong to 28. Part of the gain is that swapping asks twice and so averages the call noise, and part is that it refuses to decide when the two calls disagree. So a fall in wrong verdicts after swapping does not prove that you had a position bias.
2. Fewer wrong verdicts is not the same as a better judge. The position-biased judge, swapped, has the fewest wrong verdicts (7) and the most ties (223). It gives 370 right verdicts, against 428 for the fair judge swapped. Count the right verdicts and the ties together with the wrong ones.
3. Swapping does not fix a length bias. The wordy judge, swapped, still has 47 wrong verdicts (416 right, 137 ties), because it prefers the long answer in both orders. Each fix addresses one bias.
4. Length control overshoots here. The raw win rate of epsilon against delta under the wordy judge is 0.550. The length-controlled rate is 0.216, and the human rate in the toy is 0.300. So the correction moved the number in the right direction and past the target. The method of the paper is richer than my one-feature logistic fit, and I do not claim that the paper's method overshoots.

The panel did best (513 right, 31 wrong, 56 ties), even though two of its four judges are biased. The toy's biases are independent and point in different directions, so they partly cancel. That is the best case for a panel. Real judges built on similar models can share their biases, and then a panel helps less. The lab cannot test that.

2.5 Pairwise or pointwise?

There are two ways to ask a judge. Pointwise (the paper's "single answer grading", and the post's "rubric-based scoring"): show one answer and ask for a score or a verdict. Pairwise: show two answers and ask which is better, or whether they tie. Zheng and colleagues list the trade-off. Pairwise comparison "may lack scalability when the number of players increases", because the number of pairs grows quadratically. Single-answer grading "may be unable to discern subtle differences between specific pairs", and its results "may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes" [2]. In their data, GPT-4 with single-answer grading matched both pairwise GPT-4 and human preferences very well, which the authors read as GPT-4 having "a relatively stable internal rubric" [2]. Prometheus 2 supports both modes, and the authors say that the two are the most common forms of assessment [10].

	Pointwise	Pairwise
Calls for k systems on m questions	k × m	k(k − 1)/2 × m, and twice that if you swap the order (my arithmetic: 20 systems and 100 questions is 2,000 against 38,000)
Good at	An absolute bar ("pass or fail"); a score that you can track over time; one call per answer	Close calls; no need for a scale; people also find it easier to say which is better
Bias to watch	Scale drift and compressed scores; length; self-preference	Position; length; self-preference. Order must be controlled
Output you need	A rubric with levels, ideally binary	A way to turn many win, loss and tie results into one ranking

From pairs to a ranking. The Chatbot Arena paper collects pairwise votes from many users (over 240K when it was written) and describes the statistical methods used to rank models from them [14]. I read the abstract only. The standard model is Bradley-Terry: each system i has a strength s_i, and the chance that i beats j is s_i / (s_i + s_j), which is the same as a logistic function of the difference of the log strengths. You fit the strengths to all the results at once. A tie counts as half a win for each side. Elo is the same idea updated online, one game at a time, and its answer depends on the order of the games. People often print Bradley-Terry strengths on the Elo scale, in which 400 points is a factor of 10 in the odds. I wrote the lab's fit from general knowledge of these methods and not from the Arena paper [14]. A system with no losses has an infinite strength, so the lab adds half a win and half a loss against every other system as a small prior (my choice).

In the lab. With the fair judge, the pointwise mean scores (3.70, 3.22, 3.10, 3.12 and 2.33 for alpha to epsilon) and the pairwise Bradley-Terry ratings (1103, 1023, 987, 994 and 893 on the Elo scale) both put the five systems in the order of their true mean quality, with a Kendall tau of 1.0 for each. That is easy here because the systems are far apart, except for gamma and delta, which differ by 0.016 in true mean quality. The bootstrap 95% intervals on the log strengths of beta, gamma and delta overlap: beta (−0.02, 0.34), gamma (−0.24, 0.18), delta (−0.18, 0.16). So pairwise gives a ranking, and the intervals tell you which parts of it to believe. With online Elo and five random orders of the same comparisons, alpha's rating ranged from 1072 to 1130 (Bradley-Terry gave 1103). Now the length-biased judge. Both its pointwise ranking and its pairwise ranking are the same wrong order (alpha, gamma, beta, epsilon, delta). Pairwise comparison is not immune to a bias in the judge. The lab swapped the order, which handles position, and that did nothing for length.

2.6 Is the judge any good? Agreement and calibration

A judge is a measuring instrument, and an instrument needs a check against something you trust. For judges the usual standard is human labels. Zheng and colleagues define agreement as the chance that two judges agree on a randomly chosen question, and report two set-ups. In S1, ties and inconsistent votes count as ties, so a random judge agrees 33% of the time. In S2, only non-tie votes count, so a random judge agrees 50% of the time [2]. On MT-bench, GPT-4 with pairwise comparison agreed with human experts 66% of the time in S1 and 85% in S2, and two humans agreed 63% in S1 and 81% in S2 [2]. That is the basis of the paper's headline: over 80% agreement, "the same level of agreement between humans" [2]. Two cautions. The S2 number leaves out the cases in which the judge or the people hesitated. And the benchmark is a set of 80 chat questions, with votes from 58 labelers whom the paper calls expert-level but says are mostly graduate students, so the number does not carry over to an agent task.

Agreement can mislead

Percent agreement is the share of examples on which the two labels match. Cohen's kappa corrects for the agreement that you would get by chance: κ = (p_o − p_e) / (1 − p_e), where p_o is the observed agreement and p_e is the agreement expected if the two labelers were independent, given how often each says pass. When failures are rare, raw agreement is high even for a useless judge. Husain's example: if an error occurs in 5% of examples, a judge that always says Pass has 95% agreement and detects no errors. He advises reporting the true positive rate and true negative rate [7]. Define the class you want to catch (a failure) as the positive class. The true positive rate is the share of real failures that the judge flags. The true negative rate is the share of real passes that it lets through. These two numbers say which way the judge errs, which is what you need to know in order to use it.

In the lab. The fair mock judge, on pass or fail labels with 49.7% failing, has agreement 0.900, kappa 0.800, and catches 87.2% of failures with a true negative rate of 92.7%. A judge that always says pass, when the threshold is moved so that about 5% fail, has agreement 0.953, kappa 0.000, and catches 0% of failures. For the 1 to 5 scores, exact agreement with the human rating is only 0.627, but the quadratic-weighted kappa, which gives credit for near misses, is 0.907, and the Spearman correlation is 0.899. A judge that is off by one level on many answers has low exact agreement and is still useful for ranking. Which of the two matters depends on whether you use the score as a threshold or as a ranking.

How many labels?

A rate from a small set has a wide interval. With 30 real failures and 26 caught, the true positive rate is 0.867, and the 95% Wilson interval is 0.70 to 0.95 (my arithmetic with the lab's wilson). That is the reason for Husain's rule of thumb of 30 to 50 examples of each label in the dev set and in the test set [7]. Keep the test set hidden until the prompt is frozen, so that the number you report is not tuned [7].

Calibration

If a judge says "80% sure that this passes", then about 80% of such answers should pass. A reliability diagram groups the judge's stated probabilities into bins and plots the share of passes in each bin against the stated probability; a perfectly calibrated judge lies on the diagonal. The expected calibration error (ECE) is the average gap between the two, weighted by bin size. Platt scaling fits a logistic curve that maps the judge's score to the human pass rate, using some labeled data. In the lab the mock judge's made-up "probability of pass" has an ECE of 0.102 on all 300 answers. A Platt fit on half of the answers lowered the error on the other half from 0.123 to 0.084. That is a modest gain from 150 labels, and the curve in Figure 7 crosses the diagonal in the middle, where there are few answers per bin. Calibration fixes how a score maps to a probability. It does not fix a bias: a judge that likes long answers is still wrong about the long answers after it is calibrated. Husain also says to repeat the human check at regular intervals and whenever something material changes, such as a model update [7].

A simpler table is often enough for a score from 1 to 5. In the lab, the judge's score of 1, 2, 3, 4 and 5 corresponds to human pass rates of 0.00, 0.07, 0.39, 0.83 and 1.00 (70, 42, 44, 78 and 66 answers). So in the toy, a 3 is a coin toss that leans to fail, and a threshold at 4 gives a clean cut. You can only make such a table by comparing with human labels.

2.7 Non-determinism: pass@k, pass^k and a noisy grader

Agent runs vary, and the post says so plainly: each task has its own success rate, "maybe 90% on one task, 50% on another", and a task that passed in one run can fail in the next [1]. The post defines two metrics. pass@k is the chance that at least one of k trials succeeds, and it rises with k. pass^k is the chance that all k trials succeed, and it falls with k. At k = 1 they are equal. The post's example: with a 75% success rate per trial, the chance of three passes in a row is 0.75³, about 42%. It says that by k = 10 the two "tell opposite stories": pass@k approaches 100% while pass^k falls to 0% [1]. It advises pass@k for tools where one success is enough, and pass^k for agents where consistency matters [1]. The pass^k metric comes from τ-bench [21], and the unbiased pass@k estimate from the Codex paper [22]. Week 7b has the formulas and the unbiased estimators (from n trials with c passes, pass@k = 1 − C(n−c, k) / C(n, k), and pass^k = C(c, k) / C(n, k)).

The grader is another source of noise. Everything above assumes that each trial is graded correctly. If the grader is a model, it can pass a failure or fail a success. Suppose that a task succeeds with probability p, the grader passes a true success with probability a, and it fails a true failure with probability b. Then the reported success rate is p·a + (1 − p)(1 − b). For an always-successful task and a grader with a = 0.92, pass^8 is 0.92⁸ = 0.513 (my arithmetic), even though the agent never fails. For a task that nearly always fails, a grader that lets 10% of failures through gives the agent "shots on goal" that it did not earn, and pass@k rises.

In the lab. The true pass^k falls from 0.739 at k = 1 to 0.289 at k = 8 and 0.183 at k = 16. The judged pass^k, from the same trials, falls from 0.700 to 0.145 at k = 8 and 0.067 at k = 16. At k = 8 the grader's noise cuts the measured pass^8 roughly in half. The judged pass@k is close to the truth (0.969 against 0.958 at k = 4, and 0.995 against 0.988 at k = 8), slightly higher. So the consistency metric is the one that a noisy grader damages most. Two limits of this toy: the grader's errors are independent from one trial to the next, whereas a real judge may repeat its mistake on the same answer, which would be a different problem; and the same fair mock judge, asked five times about the same answer, gave the same pass or fail label every time for only 71.7% of 120 answers, with a mean flip rate of 11.5% against its first call. A judge that is asked once is a noisy measurement. If you can afford it, ask several times and take the majority, and report how often the judge disagrees with itself.

2.8 Inducing metrics from human feedback

All of the above assumes that you know what to measure. For a new product, often you do not. AutoMetrics (Ryan, Zhang, Salunkhe, Chu, Xu and Yang) starts from a task description and fewer than 100 human feedback points and builds an evaluator [3]. Michael Ryan and Diyi Yang are two of the course's instructors, so this is the course's own work, and I report its claims as the authors'. The method has four steps in the paper's Figure 2: generate task-specific candidate criteria with an LLM; retrieve the most relevant ones from the generated set and from MetricBank, a collection of 48 existing metrics that the authors curated; regress, by z-scoring all the metric scores and fitting a partial least squares (PLS) model to the human labels, in two stages (fit on all candidates, keep the top n by weight, refit); and report. A last filter dropped metrics with negative coefficients in their τ-bench example. The paper's default is n = 5. It reports Kendall correlation with human ratings up to 33.4% higher than an LLM-as-a-judge baseline across five tasks with fewer than 100 feedback points, finds that performance levels off at about 80 labeled examples, and shows that the learned metric can serve as a proxy reward, to roughly the same effect as a verifiable reward in one τ-bench experiment [3]. Its own conclusion warns that scaling up synthetic data from limited sources can give metrics that reflect dataset artifacts, so data diversity matters [3]. I read the body text and not the appendices.

AutoLibra (Zhu, Cuvin, Yu, Yan, Zhang and Yang, with Diyi Yang again an author) works from the other end [4]. It takes open-ended feedback such as "if you find that the button is disabled, don't click it again", grounds each piece of feedback in a behavior in the agent's trajectory, clusters similar good and bad behaviors, and writes a metric for each cluster, with a definition and examples, which can then drive an LLM judge. It proposes two meta-metrics for a set of induced metrics: coverage (how much of the feedback the metrics account for) and redundancy (how much they repeat each other) [4]. I read the abstract only. Husain's critique shadowing and the criteria drift of Shankar and colleagues [16] are the manual version of the same loop: let the criteria come from looking at outputs.

In the lab. The toy has 12 candidate criteria that look at the text ("shows its steps", "cites a source", "hedges", "ends with a polite line", and others). Each is scored by a mock judge with its own noise level, scores are z-normalized, a ridge regression is fit on the human ratings, and the five criteria with the largest weights are kept and refit. I report Kendall tau-b with the human rating on 120 held-out answers, the mean of 20 random splits for each number of labels. The result is mixed, and I report it as it came out.

Labels used	Induced metric	Holistic judge that sees the quality	Holistic judge with its own taste
10	0.427	0.713	0.330
20	0.571	0.706	0.324
40	0.571	0.709	0.317
80	0.608	0.715	0.303
160	0.638	0.718	0.338

• The induced metric never beats the first judge, which is built to see the hidden quality plus noise. A real judge does not see the human's quality. So this row cannot say anything about the paper's claim of a gain over an LLM judge, and it is not a failure of the idea either.
• The induced metric does beat a judge whose "taste" differs from the human's (it rewards politeness and length and under-rewards shown steps). That judge scores about 0.30 to 0.34 whatever the number of labels, and the induced metric passes it at 10 labels and gains more with more. This is the situation AutoMetrics is designed for: the generic judge and the people want different things, and a few labels teach the regression the people's weights. I built that mismatch by hand, so it is the assumption, not a finding.
• In the demo's run on 80 labels with four criteria kept, the criteria were "the answer looks right" (weight 0.80), "cites a source" (0.40), "starts with Then" (0.31) and "shows its steps" (0.13). The last two are near copies of each other, because both detect step-by-step text. That is the redundancy that AutoLibra's second meta-metric is meant to catch [4]. In a separate run on the first 200 answers with 8 criteria kept, "hedges" got a negative weight (−0.30), as expected, because the humans in the toy dislike hedging; this is the case in which the AutoMetrics filter on negative coefficients would drop it, and drop it wrongly if you meant it as a penalty. A negative weight is information, so read it before you remove it.
• The curve is noisy and flat between 20 and 40 labels (0.571 at both), even with 20 splits per point. I did not tune it.

2.9 Judging agents, not just answers

An agent leaves a transcript: messages, tool calls and results. The Anthropic post says to grade the outcome when you can, and to grade the transcript with model-based graders and clear rubrics for behaviors such as how the agent calls tools or talks to the user [1]. Agent-as-a-Judge (Zhuge and colleagues) goes one step further and lets an agentic system, with tools, evaluate another agentic system, so that it can give intermediate feedback for the whole task-solving process and not only a verdict on the final answer. The authors test it on code generation with DevAI, a benchmark of 55 automated AI-development tasks with 365 hierarchical user requirements, and report that it "dramatically outperforms" LLM-as-a-Judge and is as reliable as their human evaluation baseline [17]. I read the abstract only, and the benchmark is the authors' own. The lab does not model this. It is the right place to ask the question that Week 7b asked of scorers: what does the judge need access to in order to be right, and what can it be fooled by? My own point: a judge that can read the agent's files can also be fooled by what the agent writes there.

2.10 Harness design

The harness is the code that runs the tasks, calls the agent and the graders, and records what happened. Most of its design is about being able to rerun an evaluation and get the same answer, and being able to explain a number after the fact. The Anthropic post names the first requirement: each trial should start from a clean environment, because shared state (leftover files, cached data, resource exhaustion) causes failures that look like agent failures. In one of its examples, an agent gained an unfair advantage by reading the git history left by earlier trials [1]. This section is about the other half: what the harness does around the calls.

Feature	What it does	What goes wrong without it
A seed for each (task, trial)	Each run gets its own random stream, derived from the run's identity and not from the order in which the runs happen	Results change when you parallelize or reorder; a failure cannot be replayed
A cache with a careful key	Stores a successful result under a hash of everything that could change it: version of the judge prompt and model, configuration, seed, task input, trial number	If the judge prompt changes and the key does not, you read stale results. If the trial number is missing, repeated trials all return the first result
Retries with backoff	Retries transient errors (a timeout, a rate limit) after a growing wait; other errors are raised at once	A rate limit looks like an agent failure, or a bug is hidden by a retry
Errors are not cached	Only successes are stored	A temporary outage is remembered as a result
A log, one line per attempt	Status, attempt number, key, version, in a format such as JSON lines	You cannot tell cached from fresh, or find which tasks needed retries
A budget	Stops starting new runs once a cost limit is spent and marks the rest as skipped	A loop that spends the month's money overnight

Real tools have the same features. promptfoo caches the results of API calls, caches successful responses and not errors "to allow for retry attempts", and gives each repeat index its own cache namespace so that repeated runs keep distinct outputs [19]. Inspect writes eval logs and has a viewer, inspect view, for reading the results and the individual transcripts [18]. I read the documentation pages and did not run either tool.

In the lab. The harness runs 30 tasks twice (60 runs) against a backend that fails 25% of the time with a made-up transient error. The first pass has 60 successes, 10 retries and no final errors, and the backoff would have slept 5.0 seconds (the sleep is injected, so the test does not wait). The second pass makes no new backend calls: all 60 results come from the cache and are identical to the first pass. When the version string changes, 5 of 5 tasks miss the cache and run again. With a budget of 10 units at one unit per run, 10 runs are done and 20 are logged as skipped.

2.11 Anthropic's roadmap

The schedule calls it Anthropic's "8-step roadmap" [5]. The post's section "Going from zero to one" lists nine headings, numbered Step 0 to Step 8, so I show all nine [1]. The post is written by a vendor about its own practice and has no experiments, so treat it as experienced advice, not as a finding.

Step	What the post says, in short	Where this pack touches it
0. Start early	20 to 50 simple tasks drawn from real failures is a good start. Early on, changes have big effects, so small samples are enough. Evals get harder to build the longer you wait.	2.12 (sample sizes), Week 7b
1. Start with what you test manually	Turn your manual checks and the bug tracker into tasks.	2.2 (Husain's dataset step)
2. Unambiguous tasks with reference solutions	Two domain experts should reach the same verdict. A 0% pass rate over many trials (pass@100 of 0%) usually means a broken task. Make a reference solution that passes all graders.	Week 7b
3. Balanced problem sets	Test where a behavior should happen and where it should not; one-sided evals give one-sided optimization.	2.6 (the prevalence trap)
4. A robust harness and stable environment	Isolate each trial from a clean state; avoid shared state and correlated infrastructure failures.	2.10, the lab's part 10
5. Design graders thoughtfully	Deterministic graders where possible, LLM graders where necessary, humans judiciously; grade the outcome and not the path; partial credit; calibrate model graders against experts; give them a way out ("Unknown"); make graders hard to game.	2.1 to 2.6
6. Check the transcripts	Read many trials to learn whether a failure is fair and whether graders reject valid solutions. Failures should seem fair.	2.9; Week 7b
7. Monitor for saturation	An eval at 100% tracks regressions but gives no signal for improvement; progress looks smaller as only the hard tasks remain.	Week 7b (saturation)
8. Keep suites healthy over time	A dedicated evals team owns the infrastructure, and domain experts and product teams contribute tasks. Practice eval-driven development: write the eval before the agent can pass it.	The assignment's memo

After the steps, the post compares evals with other methods: production monitoring, A/B tests, user feedback, manual transcript review and systematic human studies. Its view is a "Swiss cheese" one, in which no single layer catches everything, and it reserves systematic human studies for calibrating LLM graders and for subjective outputs where human consensus is the reference [1]. That last point is the one that this week's lab acts out: the human labels exist to check the judge.

2.12 Worked arithmetic

All the numbers in this section are my own arithmetic from the formulas above. The inputs are made up. Redo each by hand before you move on.

1. Kappa from a table. A judge and a human label 100 answers. Both say pass on 60, both say fail on 20, and they split 10 and 10. Observed agreement is 0.80. The judge says pass on 70, and the human says pass on 70. Chance agreement is 0.7 × 0.7 + 0.3 × 0.3 = 0.58. Kappa is (0.80 − 0.58) / (1 − 0.58) = 0.524. Agreement of 80% is a moderate kappa.
2. The always-pass judge. At 5% failures, agreement is 0.95. The judge says pass 100% of the time and the human 95%, so chance agreement is 1 × 0.95 + 0 × 0.05 = 0.95, and kappa is (0.95 − 0.95) / (1 − 0.95) = 0.
3. A better judge, rare failures. Of 100 answers, 5 fail. The judge flags 3 of the 5 and also flags 5 of the 95 passes. Agreement is 0.93. The true positive rate is 3/5 = 0.60 and the true negative rate is 90/95 = 0.947. Kappa is 0.43 (chance agreement 0.878). With only 5 failures, the 95% Wilson interval on the true positive rate is 0.23 to 0.88, so these labels cannot say whether the judge is good or poor at finding failures.
4. Correcting a judged rate. Say a judge passes a true pass with probability a = 0.92 and fails a true fail with probability b = 0.90, and it passes 70% of what it sees. Then the true pass rate p solves 0.70 = 0.92p + 0.10(1 − p), so p = (0.70 + 0.90 − 1) / (0.92 + 0.90 − 1) = 0.6 / 0.82 = 0.73. This is a standard correction from general statistics (not from the sources above), and it assumes that a and b are known and that the judge errs independently of the answer's quality, which is unlikely to hold exactly.
5. A noisy grader and pass^k. A task is truly passed 90% of the time. A grader with a = 0.92 and b = 0.90 reports 0.9 × 0.92 + 0.1 × 0.10 = 0.838. If the trials are independent, the true pass^8 is 0.9⁸ = 0.430 and the measured one is 0.838⁸ = 0.243. For a task that is always passed, the measured pass^8 is 0.92⁸ = 0.513.
6. Bradley-Terry and Elo. If the Elo rating of i is 100 points above j, the chance that i wins is 1 / (1 + 10^−100/400) = 0.640. At 400 points it is 0.909, which is odds of 10 to 1. If i wins 3 times as often as j, the log-strength difference is ln 3 = 1.10, which is about 191 Elo points (400 / ln 10 × 1.10).
7. Position consistency by chance. A judge that picks "A" or "B" with a coin flip on each call and never ties keeps the same winner in both orders half the time (consistency 0.5). A judge that always answers "A" names a different winner in each order, so its consistency is 0. This is why Zheng's GPT-4 number of 65.0% is above chance and still far below 100% [2].
8. The cost of pairs. 20 systems on 100 questions. Pointwise: 2,000 calls. Pairwise, one order: 190 × 100 = 19,000. Both orders: 38,000.
9. Asking a judge three times. If one call is right with probability 0.8 and the calls are independent, the majority of three is right with probability 0.8³ + 3 × 0.8² × 0.2 = 0.512 + 0.384 = 0.896. Judge errors on the same answer are usually correlated, so the real gain is smaller.

2.13 Week 8a in one page

• Three graders: code (exact, cheap, brittle), model (flexible, noisy, biased), human (the reference, slow). Use code where you can, a model where you must, and humans to check the model [1].
• A judge prompt: one criterion, reasoning before the verdict, a binary verdict with an "Unknown" exit, a few worked examples, and a reference if you have one. Plan to revise the rubric [7] [16].
• Known biases: position, verbosity, self-enhancement, weak reasoning [2]. Test each with a designed test: swap, pad, compare with human win rates, check against a reference.
• Each fix covers one bias. Swapping handles position and turns disagreement into ties. It does not handle length. Length control handles length. A panel helps when the judges' biases differ [2] [15].
• Pointwise costs k × m calls and gives an absolute bar. Pairwise costs about k²/2 × m and gives a ranking through Bradley-Terry. Neither is immune to a biased judge [2] [14].
• Validate the judge against human labels. Report kappa and the true positive and true negative rates, not agreement alone, and keep a hidden test set [7].
• Calibration maps a score to a probability. It does not remove bias.
• pass@k rises with k and pass^k falls. A noisy grader hurts pass^k the most. Ask the judge more than once and report how often it disagrees with itself [1].
• With a few human labels you can induce a metric: generate criteria, score, regress, keep a few, and read the weights [3] [4].
• A harness: a seed per run, a cache with a careful key, retries for transient errors only, a log, a budget. Test that the retry machinery does not change the answers [19] [18].
• Start early with 20 to 50 tasks, read transcripts, watch for saturation, and give the suite an owner [1].

3. Reading guide

The course lists three required readings and one additional reading for this week [5]. For each you get: why it matters, the key ideas, and questions to think about as you read. Read the questions first. The optional table at the end points to the sources I used for the notes. For most of those I read only the abstract, and the references say which. The first required reading is a vendor's post about its own practice. Michael Ryan, an author of AutoMetrics, and Diyi Yang, an author of AutoMetrics and AutoLibra, are course instructors.

Optional readings

These are my additions. For each, the third column says what I read.

Source	Why it is here	What I read
Liu and colleagues, G-Eval [8]	Chain of thought plus form filling for NLG judges; names the bias toward LLM-written text.	Abstract only.
Kim and colleagues, Prometheus [9] and Prometheus 2 [10]	Open judge models that use a rubric and a reference; Prometheus 2 handles direct and pairwise judging.	Abstracts only; the authors' own results.
Zhu, Wang and Wang, JudgeLM [11]	Fine-tuned judges; swap augmentation, reference support and drop.	Abstract only; the authors' own results.
Wang and colleagues, Large Language Models are not Fair Evaluators [12]	Order of appearance can flip a ranking; three calibration methods.	Abstract only.
Panickssery, Bowman and Feng, self-preference [13]	A test of why judges favor their own text: self-recognition.	Abstract only.
Chiang and colleagues, Chatbot Arena [14]	Crowdsourced pairwise evaluation at scale (over 240K votes).	Abstract only.
Dubois and colleagues, Length-Controlled AlpacaEval [15]	A simple debiasing of a length preference.	Abstract only; the authors' own results.
Shankar and colleagues, Who Validates the Validators? [16]	Criteria drift; assigned in the course's schedule for an earlier week and used in Week 7a.	Abstract only here.
Zhuge and colleagues, Agent-as-a-Judge [17]	Agents judging agents, with a 55-task benchmark.	Abstract only; the authors' own benchmark.
Husain, LLM-as-a-Judge guide [7]	A practical procedure for judge prompts and validation.	The whole post; practitioner advice, and the page advertises the author's paid course.
Inspect [18], promptfoo [19], OpenAI evals [20]	Open-source harnesses and how they handle scoring, caching and logs.	The pages on scorers, caching and the overview. I did not run any of them.

4. Hands-on lab: a mock judge, its biases and a small eval harness

Time: about 4.5 hours. Goal: build and test a small toolkit that does what this week's notes describe: code-based graders; a mock LLM judge with injectable biases for position, length and self-preference; tests for each bias; the standard fixes (swap, random order, length control, a panel); pointwise and pairwise rankings with a Bradley-Terry fit and online Elo; agreement, kappa, true positive and true negative rates, and calibration; pass@k and pass^k with a noisy grader; an AutoMetrics-style metric induction toy; and an eval harness with seeds, a cache, retries, a log and a budget. You do not need an API key. Nothing in it calls a model or the network.

What you will learn

• How code-based graders check what they are told to check, and what they miss.
• How to build a judge with a known bias, and how to measure the bias with a designed test: swap the order, pad the answer, compare win rates with a human.
• How swapping, length control and a panel change the verdicts, and what each costs.
• How to fit Bradley-Terry strengths from pairwise results, how Elo differs, and how to put a bootstrap interval on a ranking.
• How percent agreement, Cohen's kappa, weighted kappa, Spearman correlation and true positive and true negative rates tell different stories, and how to read a reliability diagram.
• How a noisy grader changes pass@k and pass^k.
• How to regress human ratings onto a set of criterion scores, and how to read the weights.
• How to build a harness whose results do not depend on the order of the runs, and test that its retries do not change the answers.

Map of the code

Part	Main names in the code	What it does, and what it stands for
0	mean, stdev, ranks, pearson, spearman, kendall_tau_b, pass_at_k, pass_hat_k, wilson	Shared helpers, including the unbiased pass@k and pass^k estimators from Week 7b [22] [21].
1	exact_match, contains_all, regex_match, number_close, json_fields, tool_calls_ok, run_assertions	Code-based graders (2.1) [1].
2	SYSTEMS, make_world, pad_response, human_rating, human_pass, human_prefers	The made-up world: five systems, 60 questions, 300 answers with hidden traits and quality.
3	MockJudge, standard_judges	A judge with noise and with bias knobs for position, length and self-preference (2.3).
4	position_report, verbosity_report, self_preference_report, length_correlation	Designed tests for each bias (2.3) [2].
5	swapped_verdict, random_order_verdict, panel_verdict, fit_logistic, length_controlled_winrate	The fixes (2.4) [2] [15].
6	comparisons_from, bradley_terry, to_elo, elo_online, bootstrap_bt, pointwise_system_scores	Pointwise and pairwise rankings (2.5) [14].
7	percent_agreement, cohen_kappa, weighted_kappa, tpr_tnr, reliability_bins, expected_calibration_error, platt_fit	Agreement and calibration (2.6) [7].
8	graded_runs, curves_from, judge_rerun_agreement	pass@k and pass^k with a noisy grader (2.7) [1].
9	CRITERIA, criterion_scores, ridge_fit, induce_metric, induction_curve	The metric induction toy (2.8) [3] [4].
10	EvalHarness, FlakyBackend, TransientError	The harness: seeds, cache, retries, log, budget (2.10) [19].
11	main	The ten-section demo.

Setup

1. You need Python 3.10 or later. There is nothing to install. I ran it on Python 3.13 only.
2. Save the code below as judge_infra_toolkit.py. Or download it: judge_infra_toolkit.py.
3. Save the tests as test_judge_infra_toolkit.py in the same folder. Or download: test_judge_infra_toolkit.py. The test code is not printed on this page, to keep the page shorter. It is in the download, and the names of all the tests are listed after the exercises.
4. Run python judge_infra_toolkit.py for the ten-section demo, and python test_judge_infra_toolkit.py for the tests. Both are deterministic. The demo takes a few seconds and the tests about ten seconds on my machine.

The code: judge_infra_toolkit.py

"""Week 8a lab: graders, a mock LLM judge with injectable biases, bias measurement and fixes, pointwise and
pairwise scoring with Bradley-Terry, judge-human agreement and calibration, pass@k and pass^k with a noisy
grader, an AutoMetrics-style metric induction toy, and a small eval harness with caching, retries, seeds and
a result log. Standard library only. Deterministic. No network, no model.

Everything here is made up: the questions, the "responses", the human ratings, the judges. The judges are
not language models. Each is a rule with noise and a few bias knobs. Where a method follows a paper, the
text near it says which and says what I changed.
"""
import hashlib
import json
import math
import random
import re
from collections import Counter, defaultdict

# ---------------------------------------------------------------- part 0: shared helpers


def mean(xs):
    xs = list(xs)
    return sum(xs) / len(xs) if xs else 0.0


def stdev(xs, ddof=1):
    xs = list(xs)
    n = len(xs)
    if n - ddof <= 0:
        return 0.0
    m = mean(xs)
    return math.sqrt(sum((x - m) ** 2 for x in xs) / (n - ddof))


def sigmoid(x):
    if x >= 0:
        return 1.0 / (1.0 + math.exp(-x))
    e = math.exp(x)
    return e / (1.0 + e)


def clip(x, lo, hi):
    return max(lo, min(hi, x))


def normalize(text):
    """Lower-case, drop punctuation except digits, dots and minus, collapse spaces."""
    t = re.sub(r"[^a-z0-9.\- ]+", " ", str(text).lower())
    return re.sub(r"\s+", " ", t).strip().rstrip(".").strip()


def ranks(xs):
    """Average ranks (1 is the smallest). Ties share the mean of their ranks."""
    order = sorted(range(len(xs)), key=lambda i: xs[i])
    out = [0.0] * len(xs)
    i = 0
    while i < len(order):
        j = i
        while j + 1 < len(order) and xs[order[j + 1]] == xs[order[i]]:
            j += 1
        r = (i + j) / 2.0 + 1.0
        for k in range(i, j + 1):
            out[order[k]] = r
        i = j + 1
    return out


def pearson(x, y):
    mx, my = mean(x), mean(y)
    sx = math.sqrt(sum((a - mx) ** 2 for a in x))
    sy = math.sqrt(sum((b - my) ** 2 for b in y))
    if sx == 0 or sy == 0:
        return 0.0
    return sum((a - mx) * (b - my) for a, b in zip(x, y)) / (sx * sy)


def spearman(x, y):
    """Pearson correlation of the average ranks."""
    return pearson(ranks(x), ranks(y))


def kendall_tau_b(x, y):
    """Kendall's tau-b, which corrects for ties in either list."""
    n = len(x)
    conc = disc = tx = ty = 0
    for i in range(n):
        for j in range(i + 1, n):
            dx, dy = x[i] - x[j], y[i] - y[j]
            if dx == 0 and dy == 0:
                continue
            if dx == 0:
                tx += 1
            elif dy == 0:
                ty += 1
            elif dx * dy > 0:
                conc += 1
            else:
                disc += 1
    den = math.sqrt((conc + disc + tx) * (conc + disc + ty))
    return (conc - disc) / den if den else 0.0


def comb(n, k):
    if k < 0 or k > n:
        return 0
    return math.comb(n, k)


def pass_at_k(n, c, k):
    """Unbiased pass@k from c passes in n trials (Chen et al.): 1 - C(n-c, k) / C(n, k)."""
    if k > n:
        raise ValueError("k > n")
    return 1.0 - comb(n - c, k) / comb(n, k)


def pass_hat_k(n, c, k):
    """Unbiased pass^k from c passes in n trials (tau-bench): C(c, k) / C(n, k)."""
    if k > n:
        raise ValueError("k > n")
    return comb(c, k) / comb(n, k)


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    den = 1 + z * z / n
    c = (p + z * z / (2 * n)) / den
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / den
    return (max(0.0, c - h), min(1.0, c + h))


# ---------------------------------------------------------------- part 1: code-based graders
# The first of the three grader types. Cheap, fast, repeatable, and blind to anything that was not written down.


def exact_match(answer, gold):
    return normalize(answer) == normalize(gold)


def contains_all(answer, needles):
    a = normalize(answer)
    return all(normalize(n) in a for n in needles)


def regex_match(answer, pattern):
    return re.search(pattern, str(answer)) is not None


def first_number(text):
    m = re.search(r"-?\d+(?:\.\d+)?", str(text))
    return float(m.group(0)) if m else None


def number_close(answer, gold, tol=1e-6):
    """The first number in the answer is within tol of the gold number."""
    v = first_number(answer)
    return v is not None and abs(v - gold) <= tol


def json_fields(answer, required):
    """The answer is a JSON object and every key in `required` has that value."""
    try:
        obj = json.loads(answer)
    except (ValueError, TypeError):
        return False
    return isinstance(obj, dict) and all(k in obj and obj[k] == v for k, v in required.items())


def tool_calls_ok(calls, required, forbidden=()):
    """`required` appears in `calls` as a subsequence, and no forbidden tool name is used."""
    names = [c[0] if isinstance(c, (tuple, list)) else c for c in calls]
    if any(n in forbidden for n in names):
        return False
    it = iter(names)
    return all(any(n == r for n in it) for r in required)


def run_assertions(answer, assertions):
    """assertions: list of (kind, arg). Returns (share passed, all passed, list of results)."""
    fns = {"exact": exact_match, "contains": contains_all, "regex": regex_match, "number": number_close}
    res = []
    for kind, arg in assertions:
        if kind not in fns:
            raise ValueError("unknown assertion " + str(kind))
        res.append(bool(fns[kind](answer, arg)))
    return (mean(res) if res else 0.0, all(res), res)


# ---------------------------------------------------------------- part 2: a made-up world of responses
# Five "systems" answer 60 questions. Each response has hidden traits. A hidden quality number h comes from the
# traits. Humans see h with a little noise. The mock judges see h with more noise and with bias.

SYSTEMS = {
    # p_correct, p_steps, p_source, p_hedge, padding sentences on average
    "alpha": (0.85, 0.60, 0.40, 0.10, 0.0),
    "beta": (0.75, 0.50, 0.30, 0.15, 1.0),
    "gamma": (0.65, 0.50, 0.20, 0.20, 2.0),
    "delta": (0.55, 0.30, 0.20, 0.30, 0.0),
    "epsilon": (0.45, 0.30, 0.10, 0.40, 4.0),
}
FILLER = [
    "It is worth noting that this matters in practice.",
    "There are many things to consider here.",
    "This is a common question that comes up often.",
    "In general, the details can vary from case to case.",
    "As a reminder, it helps to think about this carefully.",
]
CUTS = (0.2, 0.6, 1.0, 1.4)  # latent quality to a 1..5 rating
PASS_THRESHOLD = 0.9  # a response passes for the human when h is at least this


def rating_of(h):
    return 1 + sum(1 for c in CUTS if h >= c)


def make_world(seed=0, n_q=60):
    rng = random.Random(f"world-{seed}")
    resps = []
    questions = []
    for q in range(n_q):
        gold = rng.randint(10, 99)
        diff = rng.gauss(0, 0.5)
        questions.append({"qid": q, "gold": gold, "difficulty": diff})
        for s, (pc, ps, pso, ph, pad) in SYSTEMS.items():
            correct = rng.random() < sigmoid(math.log(pc / (1 - pc)) - diff)
            steps = rng.random() < ps
            source = rng.random() < pso
            hedge = rng.random() < ph
            polite = rng.random() < 0.5
            n_pad = max(0, int(round(rng.gauss(pad, 0.7)))) if pad > 0 else 0
            bits = []
            if hedge:
                bits.append("I think maybe")
            if steps:
                bits.append("First, read the question. Then, work it out step by step because that avoids slips.")
            ans = gold if correct else gold + rng.choice([-7, -3, 3, 7, 11])
            bits.append(f"The answer is {ans}.")
            if source:
                bits.append("According to the docs, this matches the reference.")
            for _ in range(n_pad):
                bits.append(rng.choice(FILLER))
            if polite:
                bits.append("Happy to help!")
            text = " ".join(bits)
            h = (1.0 * correct + 0.4 * steps + 0.3 * source - 0.3 * hedge) + rng.gauss(0, 0.15)
            resps.append({"id": f"q{q}-{s}", "qid": q, "system": s, "text": text, "words": len(text.split()),
                          "traits": {"correct": correct, "steps": steps, "source": source, "hedge": hedge,
                                     "polite": polite, "pad": n_pad}, "h": h})
    ws = [r["words"] for r in resps]
    m, sd = mean(ws), stdev(ws)
    for r in resps:
        r["zlen"] = (r["words"] - m) / sd
    return {"seed": seed, "questions": questions, "responses": resps, "len_mean": m, "len_sd": sd,
            "by_q": _group(resps, "qid"), "systems": list(SYSTEMS)}


def _group(resps, key):
    out = defaultdict(dict)
    for r in resps:
        out[r[key]][r["system"]] = r
    return dict(out)


def pad_response(world, r, n_sentences=4):
    """The same response with filler added. Same quality, longer. Used to test for a taste for length."""
    extra = " ".join(FILLER[i % len(FILLER)] for i in range(n_sentences))
    words = r["words"] + len(extra.split())
    q = dict(r)
    q.update({"id": r["id"] + "+pad", "text": r["text"] + " " + extra, "words": words,
              "zlen": (words - world["len_mean"]) / world["len_sd"]})
    return q


def human_rating(r):
    return rating_of(r["h"])


def human_pass(r, thr=PASS_THRESHOLD):
    return r["h"] >= thr


def human_prefers(a, b, noise=0.12, seed=0):
    """A human's pairwise choice: 'A', 'B' or 'tie'. A little noise, and a tie when they are very close."""
    rng = random.Random(f"human-{seed}-{a['id']}-{b['id']}")
    d = (a["h"] - b["h"]) + rng.gauss(0, noise)
    if abs(d) < 0.05:
        return "tie"
    return "A" if d > 0 else "B"


def true_system_quality(world):
    by = defaultdict(list)
    for r in world["responses"]:
        by[r["system"]].append(r["h"])
    return {s: mean(v) for s, v in by.items()}


# ---------------------------------------------------------------- part 3: a mock judge with injectable biases


class MockJudge:
    """What the judge "sees" of a response is h plus a persistent error for this judge and response, plus a fresh
    error for each call, plus bias terms. It never reads the text. Bias knobs:
      pos_bias  added to the response shown first in a pairwise call (position bias)
      len_bias  added per standard deviation of length (verbosity bias)
      self_bias added when the response comes from `self_system` (self-preference)
    `k` is the sample index: the same k gives the same answer, a different k draws a new call error."""

    def __init__(self, name="judge", sigma=0.25, item_noise=0.15, pos_bias=0.0, len_bias=0.0, self_bias=0.0,
                 self_system=None, tie_margin=0.1, seed=0, taste=None):
        self.taste = taste
        self.name, self.sigma, self.item_noise = name, sigma, item_noise
        self.pos_bias, self.len_bias, self.self_bias = pos_bias, len_bias, self_bias
        self.self_system, self.tie_margin, self.seed = self_system, tie_margin, seed

    def _gauss(self, tag, sd):
        if sd <= 0:
            return 0.0
        return random.Random(f"{self.name}|{self.seed}|{tag}").gauss(0, sd)

    def _taste_value(self, r):
        """A judge that does not see h. It scores the traits with its own weights, which differ from the human's."""
        t = r["traits"]
        return sum(w * (r["zlen"] if k == "zlen" else float(t[k])) for k, w in self.taste.items())

    def perceived(self, r, k=0, call=""):
        base = r["h"] if self.taste is None else self._taste_value(r)
        v = base + self._gauss("item|" + r["id"], self.item_noise) + self._gauss(f"call|{r['id']}|{call}|{k}", self.sigma)
        v += self.len_bias * r["zlen"]
        if self.self_system is not None and r["system"] == self.self_system:
            v += self.self_bias
        return v

    def pointwise(self, r, k=0):
        """A 1 to 5 score against a fixed rubric (the cut points are the human's)."""
        return rating_of(self.perceived(r, k, "pw"))

    def pairwise(self, a, b, k=0):
        """'A', 'B' or 'tie'. a is shown first."""
        call = f"pair|{a['id']}|{b['id']}"
        d = self.perceived(a, k, call) - self.perceived(b, k, call) + self.pos_bias
        if abs(d) < self.tie_margin:
            return "tie"
        return "A" if d > 0 else "B"

    def pass_confidence(self, r, thr=PASS_THRESHOLD, scale=0.35, k=0):
        """A made-up "probability that this passes": a sigmoid of the distance from the threshold."""
        return sigmoid((self.perceived(r, k, "pc") - thr) / scale)

    def pass_label(self, r, thr=PASS_THRESHOLD, k=0):
        return self.perceived(r, k, "pl") >= thr


# ---------------------------------------------------------------- part 4: measuring the biases


def pair_list(world, n_pairs=None, seed=0):
    """All pairs of different systems on the same question, as (a, b) response pairs."""
    pairs = []
    for q, d in sorted(world["by_q"].items()):
        names = sorted(d)
        for i in range(len(names)):
            for j in range(i + 1, len(names)):
                pairs.append((d[names[i]], d[names[j]]))
    if n_pairs is not None:
        rng = random.Random(f"pairs-{seed}")
        pairs = rng.sample(pairs, min(n_pairs, len(pairs)))
    return pairs


def position_report(judge, pairs, k=0):
    """Swap the order of each pair. Consistent means the same response wins (or a tie) both times.
    first_rate is the share of non-tie verdicts, over both orders, that pick the response shown first."""
    cons = first = nontie = 0
    for a, b in pairs:
        v1 = judge.pairwise(a, b, k)
        v2 = judge.pairwise(b, a, k)
        w1 = {"A": a["id"], "B": b["id"], "tie": None}[v1]
        w2 = {"A": b["id"], "B": a["id"], "tie": None}[v2]
        cons += int(w1 == w2)
        for v in (v1, v2):
            if v != "tie":
                nontie += 1
                first += int(v == "A")
    n = len(pairs)
    return {"consistency": cons / n if n else 0.0, "first_rate": first / nontie if nontie else 0.0, "pairs": n}


def verbosity_report(judge, world, responses, n_sentences=4, k=0):
    """Compare each response with a padded copy of itself, in both orders. Equal quality, so an unbiased judge
    gives the padded copy half of the credit (a tie counts as half). Returns the padded copy's share."""
    credit = 0.0
    n = 0
    for r in responses:
        p = pad_response(world, r, n_sentences)
        for a, b in ((p, r), (r, p)):
            v = judge.pairwise(a, b, k)
            if v == "tie":
                credit += 0.5
            elif (v == "A") == (a is p):
                credit += 1.0
            n += 1
    return {"padded_share": credit / n if n else 0.0, "n": n}


def self_preference_report(judge, world, pairs=None, k=0):
    """For pairs that include the judge's own system: the judge's win rate for the own side (both orders, a tie is
    half) minus the human win rate (from h, no noise, a tie is half). Positive means the judge likes its own."""
    s = judge.self_system
    pairs = pairs if pairs is not None else pair_list(world)
    jw = hw = n = 0.0
    for a, b in pairs:
        if s not in (a["system"], b["system"]):
            continue
        own, other = (a, b) if a["system"] == s else (b, a)
        for x, y in ((own, other), (other, own)):
            v = judge.pairwise(x, y, k)
            if v == "tie":
                jw += 0.5
            elif (v == "A") == (x is own):
                jw += 1.0
            n += 1
        hw += 1.0 if own["h"] > other["h"] else 0.0
    hn = n / 2.0
    return {"judge_winrate": jw / n if n else 0.0, "human_winrate": hw / hn if hn else 0.0,
            "gap": (jw / n - hw / hn) if n else 0.0, "n": int(n)}


def length_correlation(judge, responses, k=0):
    """Correlation between the judge's pointwise score and the number of words, minus the same for humans."""
    j = [judge.pointwise(r, k) for r in responses]
    h = [human_rating(r) for r in responses]
    w = [r["words"] for r in responses]
    return {"judge": pearson(j, w), "human": pearson(h, w)}


# ---------------------------------------------------------------- part 5: mitigations


def swapped_verdict(judge, a, b, k=0):
    """Ask in both orders. Declare a winner only if both orders agree. Else a tie. (Zheng et al. call this the
    conservative approach to position bias.) Returns 'a', 'b' or 'tie' for the responses, not for the slots."""
    v1 = judge.pairwise(a, b, k)
    v2 = judge.pairwise(b, a, k)
    w1 = "a" if v1 == "A" else "b" if v1 == "B" else "tie"
    w2 = "b" if v2 == "A" else "a" if v2 == "B" else "tie"
    return w1 if w1 == w2 else "tie"


def random_order_verdict(judge, a, b, rng, k=0):
    """Randomize which response is shown first. Unbiased on average, noisier for any one pair."""
    if rng.random() < 0.5:
        v = judge.pairwise(a, b, k)
        return {"A": "a", "B": "b", "tie": "tie"}[v]
    v = judge.pairwise(b, a, k)
    return {"A": "b", "B": "a", "tie": "tie"}[v]


def panel_verdict(judges, a, b, k=0, swap=True, exclude_own=True):
    """A panel of judges. Each votes (in both orders if swap). The sign of the vote total decides. With exclude_own,
    a judge does not vote on a pair that includes its own system."""
    total = 0
    voters = 0
    for j in judges:
        if exclude_own and j.self_system in (a["system"], b["system"]):
            continue
        voters += 1
        v = swapped_verdict(j, a, b, k) if swap else {"A": "a", "B": "b", "tie": "tie"}[j.pairwise(a, b, k)]
        total += {"a": 1, "b": -1, "tie": 0}[v]
    if voters == 0 or total == 0:
        return "tie"
    return "a" if total > 0 else "b"


def fit_logistic(X, y, l2=1e-3, iters=200, lr=0.5):
    """Logistic regression by gradient descent, with an intercept. X is a list of feature lists, y in [0, 1]
    (half credit allowed). Returns [intercept, coef...]."""
    n, d = len(X), len(X[0]) if X else 0
    w = [0.0] * (d + 1)
    for _ in range(iters):
        g = [0.0] * (d + 1)
        for xi, yi in zip(X, y):
            p = sigmoid(w[0] + sum(w[j + 1] * xi[j] for j in range(d)))
            e = p - yi
            g[0] += e
            for j in range(d):
                g[j + 1] += e * xi[j]
        for j in range(d + 1):
            reg = l2 * w[j] if j else 0.0
            w[j] -= lr * (g[j] / n + reg)
    return w


def length_controlled_winrate(rows, **kw):
    """rows: dicts with 'len_diff' (standardized length of the first minus the second) and 'pref' (1, 0.5 or 0 for
    the first). Fit pref ~ sigmoid(b0 + b1 * len_diff) and report the prediction at len_diff = 0. This is a
    simplified version of the idea in Length-Controlled AlpacaEval, which uses a richer model."""
    X = [[r["len_diff"]] for r in rows]
    y = [r["pref"] for r in rows]
    w = fit_logistic(X, y, **kw)
    raw = mean(y)
    return {"raw": raw, "controlled": sigmoid(w[0]), "length_coef": w[1]}


def system_pair_rows(world, judge, system, baseline, k=0, both_orders=True):
    """Pairwise rows for `system` against `baseline` on every question, with the length difference."""
    rows = []
    for q, d in sorted(world["by_q"].items()):
        a, b = d[system], d[baseline]
        orders = ((a, b), (b, a)) if both_orders else ((a, b),)
        for x, y in orders:
            v = judge.pairwise(x, y, k)
            first = 1.0 if v == "A" else 0.0 if v == "B" else 0.5
            mine = first if x is a else 1.0 - first
            rows.append({"len_diff": a["zlen"] - b["zlen"], "pref": mine})
    return rows


# ---------------------------------------------------------------- part 6: pointwise and pairwise rankings


def comparisons_from(world, verdict_fn):
    """verdict_fn(a, b) returns 'a', 'b' or 'tie'. Returns (system_a, system_b, outcome for a: 1, 0.5, 0)."""
    out = []
    for a, b in pair_list(world):
        v = verdict_fn(a, b)
        out.append((a["system"], b["system"], {"a": 1.0, "tie": 0.5, "b": 0.0}[v]))
    return out


def bradley_terry(comps, systems, iters=500, tol=1e-9, prior=0.5):
    """Bradley-Terry strengths by the standard minorization-maximization update. A tie is half a win for each side.
    `prior` adds that many wins of each side against each other side, so that a system with no losses stays finite.
    Returns log-strengths with mean 0."""
    w = {s: 0.0 for s in systems}
    n = defaultdict(float)
    for a, b, o in comps:
        w[a] += o
        w[b] += 1 - o
        n[(a, b)] += 1
        n[(b, a)] += 1
    for s in systems:
        for t in systems:
            if s != t:
                w[s] += prior
                n[(s, t)] += 2 * prior
    p = {s: 1.0 for s in systems}
    for _ in range(iters):
        new = {}
        for s in systems:
            den = sum(n[(s, t)] / (p[s] + p[t]) for t in systems if t != s and n[(s, t)] > 0)
            new[s] = w[s] / den if den else p[s]
        g = math.exp(mean(math.log(v) for v in new.values()))
        new = {s: v / g for s, v in new.items()}
        delta = max(abs(math.log(new[s]) - math.log(p[s])) for s in systems)
        p = new
        if delta < tol:
            break
    return {s: math.log(v) for s, v in p.items()}


def to_elo(log_strengths, base=1000.0):
    """Put log-strengths on the familiar scale: 400 points per factor of 10 in odds."""
    c = 400.0 / math.log(10)
    return {s: base + c * v for s, v in log_strengths.items()}


def elo_online(comps, k=32.0, seed=0, base=1000.0):
    """Online Elo over the comparisons in a random order. The answer depends on the order, which is one reason
    to fit all the comparisons at once, as bradley_terry does."""
    rng = random.Random(f"elo-{seed}")
    order = list(comps)
    rng.shuffle(order)
    r = defaultdict(lambda: base)
    for a, b, o in order:
        ea = 1.0 / (1.0 + 10 ** ((r[b] - r[a]) / 400.0))
        r[a] += k * (o - ea)
        r[b] += k * ((1 - o) - (1 - ea))
    return dict(r)


def bootstrap_bt(comps, systems, n_boot=100, seed=0):
    """Resample the comparisons with replacement and refit. Returns per-system (low, high) of the 95% interval on
    the log-strength, and how often each system lands at each rank."""
    rng = random.Random(f"boot-{seed}")
    fits = []
    for _ in range(n_boot):
        samp = [comps[rng.randrange(len(comps))] for _ in comps]
        fits.append(bradley_terry(samp, systems, iters=100))
    ci = {}
    rank_counts = {s: Counter() for s in systems}
    for f in fits:
        order = sorted(systems, key=lambda s: -f[s])
        for pos, s in enumerate(order):
            rank_counts[s][pos + 1] += 1
    for s in systems:
        v = sorted(f[s] for f in fits)
        ci[s] = (v[int(0.025 * (n_boot - 1))], v[int(math.ceil(0.975 * (n_boot - 1)))])
    return {"ci": ci, "rank_counts": rank_counts}


def pointwise_system_scores(world, judge, k=0):
    by = defaultdict(list)
    for r in world["responses"]:
        by[r["system"]].append(judge.pointwise(r, k))
    return {s: mean(v) for s, v in by.items()}


def ranking_agreement(scores, truth):
    keys = sorted(truth)
    a = [scores[s] for s in keys]
    b = [truth[s] for s in keys]
    return {"kendall": kendall_tau_b(a, b), "spearman": spearman(a, b)}


# ---------------------------------------------------------------- part 7: judge and human agreement, calibration


def percent_agreement(a, b):
    return mean(1.0 if x == y else 0.0 for x, y in zip(a, b))


def cohen_kappa(a, b):
    n = len(a)
    po = percent_agreement(a, b)
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[c] / n) * (cb[c] / n) for c in set(ca) | set(cb))
    return (po - pe) / (1 - pe) if pe < 1 else 0.0


def weighted_kappa(a, b, levels=(1, 2, 3, 4, 5)):
    """Quadratic-weighted kappa for ordinal ratings."""
    n = len(a)
    k = len(levels)
    idx = {v: i for i, v in enumerate(levels)}
    obs = [[0.0] * k for _ in range(k)]
    for x, y in zip(a, b):
        obs[idx[x]][idx[y]] += 1.0 / n
    ra = [sum(obs[i]) for i in range(k)]
    rb = [sum(obs[i][j] for i in range(k)) for j in range(k)]
    num = den = 0.0
    for i in range(k):
        for j in range(k):
            w = ((i - j) / (k - 1)) ** 2
            num += w * obs[i][j]
            den += w * ra[i] * rb[j]
    return 1 - num / den if den else 0.0


def confusion(judge_labels, human_labels, positive=False):
    """Treat `positive` as the class you want to catch (default: a fail, which is False). Returns tp, fp, tn, fn."""
    tp = fp = tn = fn = 0
    for j, h in zip(judge_labels, human_labels):
        jp, hp = (j == positive), (h == positive)
        if jp and hp:
            tp += 1
        elif jp and not hp:
            fp += 1
        elif not jp and not hp:
            tn += 1
        else:
            fn += 1
    return tp, fp, tn, fn


def tpr_tnr(judge_labels, human_labels, positive=False):
    tp, fp, tn, fn = confusion(judge_labels, human_labels, positive)
    return {"tpr": tp / (tp + fn) if tp + fn else 0.0, "tnr": tn / (tn + fp) if tn + fp else 0.0}


def reliability_bins(confs, labels, n_bins=10):
    """For each confidence bin: count, mean confidence, share of positives."""
    bins = [[] for _ in range(n_bins)]
    for c, y in zip(confs, labels):
        bins[min(n_bins - 1, int(c * n_bins))].append((c, 1.0 if y else 0.0))
    return [{"n": len(b), "conf": mean(c for c, _ in b), "freq": mean(y for _, y in b)} if b else
            {"n": 0, "conf": 0.0, "freq": 0.0} for b in bins]


def expected_calibration_error(confs, labels, n_bins=10):
    n = len(confs)
    return sum(b["n"] / n * abs(b["conf"] - b["freq"]) for b in reliability_bins(confs, labels, n_bins) if b["n"])


def platt_fit(scores, labels, **kw):
    """Fit sigmoid(a + b * score) to the human labels. Returns (a, b)."""
    w = fit_logistic([[s] for s in scores], [1.0 if y else 0.0 for y in labels], **kw)
    return w[0], w[1]


def score_to_pass_rate(scores, labels):
    """For each judge score, the share of responses that the human passes."""
    d = defaultdict(list)
    for s, y in zip(scores, labels):
        d[s].append(1.0 if y else 0.0)
    return {s: (mean(v), len(v)) for s, v in sorted(d.items())}


def imbalanced_labels(world, fail_rate):
    """Human pass labels, but with the threshold moved so that about `fail_rate` of responses fail."""
    hs = sorted(r["h"] for r in world["responses"])
    thr = hs[int(fail_rate * (len(hs) - 1))]
    return thr


# ---------------------------------------------------------------- part 8: pass@k and pass^k with a noisy grader


def graded_runs(task_p, trials, tpr=1.0, tnr=1.0, seed=0):
    """Each task has a true success rate. Run `trials` per task. A grader has a hit rate `tpr` (it passes a true
    success) and a correct-reject rate `tnr` (it fails a true failure). Returns (true matrix, judged matrix)."""
    truth, judged = [], []
    for i, p in enumerate(task_p):
        rng = random.Random(f"gr-{seed}-{i}")
        t_row, j_row = [], []
        for _ in range(trials):
            ok = rng.random() < p
            seen = (rng.random() < tpr) if ok else (rng.random() >= tnr)
            t_row.append(ok)
            j_row.append(seen)
        truth.append(t_row)
        judged.append(j_row)
    return truth, judged


def curves_from(mat, ks):
    n = len(mat[0])
    out = {"pass_at_k": [], "pass_hat_k": []}
    for k in ks:
        out["pass_at_k"].append(mean(pass_at_k(n, sum(row), k) for row in mat))
        out["pass_hat_k"].append(mean(pass_hat_k(n, sum(row), k) for row in mat))
    return out


def judge_rerun_agreement(judge, responses, runs=5, thr=PASS_THRESHOLD):
    """Ask the same judge the same question `runs` times (different call errors). Returns the share of responses on
    which every run gave the same pass/fail label, and the mean label flip rate against the first run."""
    same = 0
    flips = []
    for r in responses:
        labs = [judge.pass_label(r, thr, k) for k in range(runs)]
        same += int(len(set(labs)) == 1)
        flips.append(mean(1.0 if l != labs[0] else 0.0 for l in labs[1:]))
    return {"all_same": same / len(responses), "flip_rate": mean(flips)}


# ---------------------------------------------------------------- part 9: an AutoMetrics-style metric induction toy
# Generate candidate criteria, score every response on each (a noisy "LLM judge for one criterion"), z-normalize,
# regress onto a few human labels, keep the best few. This copies the shape of the AutoMetrics pipeline
# (generate, retrieve, regress, report). It does not call a model, it has no MetricBank, and it uses ridge
# regression, not the partial least squares of the paper.

CRITERIA = {
    # name: (what it looks at in the text, noise sd of the mock criterion judge)
    "shows_steps": (lambda r: 1.0 if "step by step" in r["text"] else 0.0, 0.20),
    "cites_a_source": (lambda r: 1.0 if "According to the docs" in r["text"] else 0.0, 0.20),
    "hedges": (lambda r: 1.0 if "I think maybe" in r["text"] else 0.0, 0.20),
    "polite_closing": (lambda r: 1.0 if "Happy to help" in r["text"] else 0.0, 0.20),
    "is_long": (lambda r: clip(r["words"] / 80.0, 0, 1), 0.10),
    "has_filler": (lambda r: 1.0 if any(f in r["text"] for f in FILLER) else 0.0, 0.20),
    "answer_looks_right": (lambda r: 1.0 if r["traits"]["correct"] else 0.0, 0.45),
    "mentions_answer": (lambda r: 1.0 if "The answer is" in r["text"] else 0.0, 0.05),
    "uses_because": (lambda r: 1.0 if "because" in r["text"] else 0.0, 0.20),
    "exclamation": (lambda r: 1.0 if "!" in r["text"] else 0.0, 0.10),
    "sentence_count": (lambda r: clip(r["text"].count(".") / 8.0, 0, 1), 0.10),
    "starts_with_then": (lambda r: 1.0 if "Then," in r["text"] else 0.0, 0.20),
}


TASTE = {"correct": 0.6, "steps": 0.1, "source": 0.0, "hedge": 0.0, "polite": 0.4, "zlen": 0.3}  # a judge whose taste differs from the human's


def criterion_scores(responses, names=None, seed=0):
    """A matrix (responses x criteria) of mock criterion-judge scores."""
    names = names or list(CRITERIA)
    X = []
    for r in responses:
        row = []
        for nm in names:
            fn, sd = CRITERIA[nm]
            noise = random.Random(f"crit|{seed}|{nm}|{r['id']}").gauss(0, sd) if sd else 0.0
            row.append(fn(r) + noise)
        X.append(row)
    return X


def zscore_cols(X, means=None, sds=None):
    d = len(X[0])
    means = means or [mean(row[j] for row in X) for j in range(d)]
    sds = sds or [stdev([row[j] for row in X]) or 1.0 for j in range(d)]
    return [[(row[j] - means[j]) / sds[j] for j in range(d)] for row in X], means, sds


def solve_linear(A, b):
    """Gaussian elimination with partial pivoting."""
    n = len(A)
    M = [list(A[i]) + [b[i]] for i in range(n)]
    for c in range(n):
        p = max(range(c, n), key=lambda i: abs(M[i][c]))
        if abs(M[p][c]) < 1e-12:
            raise ValueError("singular")
        M[c], M[p] = M[p], M[c]
        for i in range(c + 1, n):
            f = M[i][c] / M[c][c]
            for j in range(c, n + 1):
                M[i][j] -= f * M[c][j]
    x = [0.0] * n
    for i in range(n - 1, -1, -1):
        x[i] = (M[i][n] - sum(M[i][j] * x[j] for j in range(i + 1, n))) / M[i][i]
    return x


def ridge_fit(Z, y, lam=1.0):
    """Ridge regression on standardized columns with a centered target. Returns (coefs, y mean)."""
    d = len(Z[0])
    my = mean(y)
    A = [[sum(row[i] * row[j] for row in Z) + (lam if i == j else 0.0) for j in range(d)] for i in range(d)]
    b = [sum(row[i] * (yy - my) for row, yy in zip(Z, y)) for i in range(d)]
    return solve_linear(A, b), my


def induce_metric(responses, y, names=None, n_keep=5, lam=5.0, drop_negative=False, seed=0):
    """Fit on all candidate criteria, keep the n_keep with the largest |coefficient|, refit on those. With
    drop_negative, drop kept criteria whose refit coefficient is negative and refit once more.
    Returns a dict with names, weights, and a predict(responses) function."""
    names = names or list(CRITERIA)
    X = criterion_scores(responses, names, seed)
    Z, mu, sd = zscore_cols(X)
    coef, _ = ridge_fit(Z, y, lam)
    order = sorted(range(len(names)), key=lambda j: -abs(coef[j]))[:n_keep]
    keep = [names[j] for j in order]
    cols = order

    def refit(cols):
        Zs = [[row[j] for j in cols] for row in Z]
        return ridge_fit(Zs, y, lam)

    c2, my = refit(cols)
    if drop_negative and any(c < 0 for c in c2):
        cols = [j for j, c in zip(cols, c2) if c >= 0] or cols
        c2, my = refit(cols)
    keep = [names[j] for j in cols]

    def predict(rs):
        Xn = criterion_scores(rs, names, seed)
        out = []
        for row in Xn:
            out.append(my + sum(c * (row[j] - mu[j]) / sd[j] for c, j in zip(c2, cols)))
        return out

    return {"names": keep, "weights": c2, "predict": predict}


def induction_curve(world, sizes=(10, 20, 40, 80, 160), repeats=20, n_keep=5, seed=0):
    """Mean Kendall tau-b on held-out responses between the induced metric and the human rating, against the number
    of human labels. Also a baseline: one holistic mock judge's pointwise score on the same held-out set."""
    rs = world["responses"]
    base = MockJudge("holistic", sigma=0.25, item_noise=0.30, seed=seed)
    taste = MockJudge("holistic-taste", sigma=0.25, item_noise=0.30, seed=seed, taste=TASTE)
    rows = []
    for n in sizes:
        taus, btaus, wtaus = [], [], []
        for t in range(repeats):
            rng = random.Random(f"ind-{seed}-{n}-{t}")
            idx = list(range(len(rs)))
            rng.shuffle(idx)
            train = [rs[i] for i in idx[:n]]
            test = [rs[i] for i in idx[n:n + 120]]
            m = induce_metric(train, [human_rating(r) for r in train], n_keep=n_keep, seed=seed)
            taus.append(kendall_tau_b(m["predict"](test), [human_rating(r) for r in test]))
            btaus.append(kendall_tau_b([base.pointwise(r) for r in test], [human_rating(r) for r in test]))
            wtaus.append(kendall_tau_b([taste.pointwise(r) for r in test], [human_rating(r) for r in test]))
        rows.append({"n": n, "induced": mean(taus), "baseline": mean(btaus), "taste": mean(wtaus)})
    return rows


# ---------------------------------------------------------------- part 10: a small eval harness


class TransientError(Exception):
    """An error worth retrying: a timeout, a rate limit."""


class FlakyBackend:
    """Wraps a function so that some calls raise TransientError. Whether an attempt fails depends only on the key
    and the attempt number, so runs are repeatable."""

    def __init__(self, fn, fail_rate=0.2, seed=0):
        self.fn, self.fail_rate, self.seed = fn, fail_rate, seed
        self.calls = 0
        self.attempts = defaultdict(int)

    def __call__(self, task, rng):
        self.calls += 1
        # identify the call by the task and the generator state, without drawing from the generator
        key = task["id"] + "|" + hashlib.md5(repr(rng.getstate()).encode()).hexdigest()
        a = self.attempts[key] = self.attempts[key] + 1
        if random.Random(f"flaky|{self.seed}|{task['id']}|{a}").random() < self.fail_rate:
            raise TransientError("rate limit")
        return self.fn(task, rng)


class EvalHarness:
    """Runs `fn(task, rng)` for every (task, trial). Features:
    - a seed per (task, trial), so results do not depend on the order of the runs
    - a cache keyed by a hash of (version, config, seed, task, trial); only successes are stored
    - retries on TransientError with exponential backoff (the sleep function is injected, so tests do not wait)
    - a result log, one dict per attempt (JSON lines if log_path is given)
    - a cost budget: once spent, remaining runs are marked skipped
    `version` must change when the judge prompt, the model or the rubric changes, or old results will be reused."""

    def __init__(self, fn, version="v1", config=None, cache=None, max_retries=3, backoff=0.5, sleep=None,
                 seed=0, log_path=None, budget=None, cost_fn=None):
        self.fn, self.version, self.config = fn, version, dict(config or {})
        self.cache = cache if cache is not None else {}
        self.max_retries, self.backoff = max_retries, backoff
        self.sleep = sleep or (lambda s: None)
        self.seed, self.log_path, self.budget = seed, log_path, budget
        self.cost_fn = cost_fn or (lambda task, result: 1.0)
        self.log = []
        self.slept = 0.0
        self.spent = 0.0
        self._clock = 0

    def key(self, task, trial):
        blob = json.dumps({"v": self.version, "c": self.config, "s": self.seed, "t": task["id"],
                           "x": task.get("input"), "k": trial}, sort_keys=True, default=str)
        return hashlib.sha256(blob.encode()).hexdigest()[:16]

    def _record(self, **kw):
        self._clock += 1
        kw["step"] = self._clock
        self.log.append(kw)
        if self.log_path:
            with open(self.log_path, "a") as f:
                f.write(json.dumps(kw, sort_keys=True, default=str) + "\n")

    def run_one(self, task, trial=0):
        key = self.key(task, trial)
        base = {"task": task["id"], "trial": trial, "key": key, "version": self.version}
        if key in self.cache:
            self._record(status="cached", attempt=0, **base)
            return {"task": task["id"], "trial": trial, "status": "cached", "result": self.cache[key]}
        if self.budget is not None and self.spent >= self.budget:
            self._record(status="skipped", attempt=0, **base)
            return {"task": task["id"], "trial": trial, "status": "skipped", "result": None}
        for attempt in range(1, self.max_retries + 2):
            rng = random.Random(f"{self.seed}-{task['id']}-{trial}")
            try:
                res = self.fn(task, rng)
            except TransientError as e:
                self._record(status="retry" if attempt <= self.max_retries else "error", attempt=attempt,
                             error=str(e), **base)
                if attempt <= self.max_retries:
                    wait = self.backoff * 2 ** (attempt - 1)
                    self.slept += wait
                    self.sleep(wait)
                continue
            self.cache[key] = res
            self.spent += self.cost_fn(task, res)
            self._record(status="ok", attempt=attempt, **base)
            return {"task": task["id"], "trial": trial, "status": "ok", "result": res}
        return {"task": task["id"], "trial": trial, "status": "error", "result": None}

    def run(self, tasks, trials=1):
        return [self.run_one(t, k) for t in tasks for k in range(trials)]

    def summary(self):
        c = Counter(r["status"] for r in self.log)
        return {"ok": c["ok"], "cached": c["cached"], "retries": c["retry"], "errors": c["error"],
                "skipped": c["skipped"], "slept": self.slept, "spent": self.spent}


# ---------------------------------------------------------------- part 11: the demo


def _f(x, d=3):
    return f"{x:.{d}f}"


def standard_judges():
    """A panel of three judges, each with a different taste, and one judge for each single bias."""
    return {
        "fair": MockJudge("fair", seed=1),
        "first": MockJudge("first", pos_bias=0.40, seed=2),
        "wordy": MockJudge("wordy", len_bias=0.25, seed=3),
        "selfish": MockJudge("selfish", self_bias=0.35, self_system="epsilon", seed=4),
    }


def main():
    W = make_world(0)
    R = W["responses"]
    pairs = pair_list(W)
    J = standard_judges()
    print("== 1. code-based graders on the toy answers (the first of three grader types)")
    golds = {q["qid"]: q["gold"] for q in W["questions"]}
    right = [number_close(r["text"].split("The answer is ")[1], golds[r["qid"]]) for r in R]
    agree = percent_agreement(right, [r["traits"]["correct"] for r in R])
    print(f"number_close finds the right answer in {sum(right)} of {len(R)} responses; it agrees with the hidden trait on {_f(agree)}")
    hp = [human_pass(r) for r in R]
    print(f"but a human passes {sum(hp)} of {len(R)}; code-based pass (correct only) vs human pass: agreement {_f(percent_agreement(right, hp))}, kappa {_f(cohen_kappa(right, hp))}")
    print(f"\n== 2. position bias ({len(pairs)} pairs of different systems, each asked in both orders)")
    for nm in ("fair", "first"):
        r = position_report(J[nm], pairs)
        print(f"{nm:8s} consistency {_f(r['consistency'])}  first-slot rate {_f(r['first_rate'])}")
    print("\n== 3. verbosity bias (a response against itself plus 4 filler sentences)")
    sample = R[::6]
    for nm in ("fair", "wordy"):
        print(f"{nm:8s} padded copy gets {_f(verbosity_report(J[nm], W, sample)['padded_share'])} of the credit (0.5 is fair)")
    print("\n== 4. self-preference (the judge's own system is epsilon, which is weak and wordy)")
    neutral = MockJudge("selfish", self_bias=0.0, self_system="epsilon", seed=4)  # same judge, bias switched off
    for nm, jd in (("no bias", neutral), ("selfish", J["selfish"])):
        r = self_preference_report(jd, W)
        print(f"{nm:8s} judge win rate {_f(r['judge_winrate'])}  human win rate {_f(r['human_winrate'])}  gap {r['gap']:+.3f}")
    print("\n== 5. mitigations: swap, length control, a panel")
    fair, first, wordy = J["fair"], J["first"], J["wordy"]
    single = position_report(first, pairs)["consistency"]
    print(f"first-biased judge: consistency {_f(single)}; after swapping, verdicts are consistent by construction, and a disagreement becomes a tie")
    truth = [("a" if a["h"] > b["h"] else "b") for a, b in pairs]
    raw = [{"A": "a", "B": "b", "tie": "tie"}[first.pairwise(a, b)] for a, b in pairs]
    sw = [swapped_verdict(first, a, b) for a, b in pairs]
    cnt = lambda v: (sum(x == t for x, t in zip(v, truth)), sum(x not in (t, "tie") for x, t in zip(v, truth)), sum(x == "tie" for x in v))
    print(f"verdicts out of {len(pairs)} pairs against the hidden quality (right, wrong, tie):")
    print(f"  first-biased judge, raw order  : {cnt(raw)}")
    print(f"  first-biased judge, swapped    : {cnt(sw)}")
    rows = system_pair_rows(W, wordy, "epsilon", "delta")
    lc = length_controlled_winrate(rows)
    hum = mean(1.0 if W["by_q"][q]["epsilon"]["h"] > W["by_q"][q]["delta"]["h"] else 0.0 for q in W["by_q"])
    print(f"epsilon vs delta with a wordy judge: raw win rate {_f(lc['raw'])}, length-controlled {_f(lc['controlled'])}, human (from h) {_f(hum)}")
    panel = [J["fair"], J["first"], J["wordy"], J["selfish"]]
    for nm in ("fair", "first", "wordy", "selfish"):
        v = [swapped_verdict(J[nm], a, b) for a, b in pairs]
        print(f"  {nm:8s} alone, swapped: {cnt(v)}")
    pv = [panel_verdict(panel, a, b) for a, b in pairs]
    print(f"  panel of 4, swapped, no own-system votes: {cnt(pv)}")
    print("\n== 6. pointwise against pairwise: five systems ranked")
    tq = true_system_quality(W)
    print("true quality (mean h):", {s: round(v, 3) for s, v in tq.items()})
    pw = pointwise_system_scores(W, fair)
    comps = comparisons_from(W, lambda a, b: swapped_verdict(fair, a, b))
    bt = bradley_terry(comps, W["systems"])
    elo = to_elo(bt)
    print("pointwise mean score:", {s: round(v, 2) for s, v in pw.items()})
    print("Bradley-Terry Elo   :", {s: round(v) for s, v in elo.items()})
    print(f"judge calls: pointwise {len(R)}, pairwise {len(pairs)} pairs x 2 orders = {2 * len(pairs)}")
    print("agreement with truth: pointwise", {k: round(v, 3) for k, v in ranking_agreement(pw, tq).items()}, "BT", {k: round(v, 3) for k, v in ranking_agreement(bt, tq).items()})
    pw_w = pointwise_system_scores(W, wordy)
    bt_w = bradley_terry(comparisons_from(W, lambda a, b: swapped_verdict(wordy, a, b)), W["systems"])
    print("wordy judge, rank order: pointwise", sorted(pw_w, key=lambda s: -pw_w[s]), "BT", sorted(bt_w, key=lambda s: -bt_w[s]), "truth", sorted(tq, key=lambda s: -tq[s]))
    elos = [elo_online(comps, seed=i) for i in range(5)]
    print("online Elo, 5 shuffles, alpha's rating:", [round(e["alpha"]) for e in elos])
    boot = bootstrap_bt(comps, W["systems"], n_boot=60)
    print("bootstrap 95% interval of log-strength:", {s: (round(lo, 2), round(hi, 2)) for s, (lo, hi) in boot["ci"].items()})
    print("\n== 7. judge and human agreement")
    hl = [human_pass(r) for r in R]
    jl = [fair.pass_label(r) for r in R]
    print(f"fair judge pass/fail: agreement {_f(percent_agreement(jl, hl))}, kappa {_f(cohen_kappa(jl, hl))}, fail TPR/TNR {tpr_tnr(jl, hl)}")
    jr = [fair.pointwise(r) for r in R]
    hr = [human_rating(r) for r in R]
    print(f"fair judge 1 to 5: exact agreement {_f(percent_agreement(jr, hr))}, weighted kappa {_f(weighted_kappa(jr, hr))}, Spearman {_f(spearman(jr, hr))}")
    thr = imbalanced_labels(W, 0.05)
    hl2 = [r["h"] >= thr for r in R]
    jl2 = [True] * len(R)
    print(f"rare failures (5%): a judge that always says pass has agreement {_f(percent_agreement(jl2, hl2))}, kappa {_f(cohen_kappa(jl2, hl2))}, fail TPR {tpr_tnr(jl2, hl2)['tpr']:.2f}")
    confs = [fair.pass_confidence(r) for r in R]
    print(f"calibration: expected calibration error {_f(expected_calibration_error(confs, hl))}")
    a, b = platt_fit([c for c in confs[::2]], hl[::2], iters=3000, lr=2.0)
    cal = [sigmoid(a + b * c) for c in confs[1::2]]
    print(f"after a Platt fit on half: error on the other half {_f(expected_calibration_error(cal, hl[1::2]))} (before, same half: {_f(expected_calibration_error(confs[1::2], hl[1::2]))})")
    print("score to human pass rate:", {s: (round(p, 2), n) for s, (p, n) in score_to_pass_rate(jr, hl).items()})
    print("\n== 8. pass@k and pass^k with a noisy grader (the agent is the same in both rows)")
    rng = random.Random(5)
    task_p = [rng.choice([0.95, 0.9, 0.8, 0.6, 0.3]) for _ in range(60)]
    t, j = graded_runs(task_p, 16, tpr=0.92, tnr=0.90, seed=1)
    ks = [1, 2, 4, 8, 16]
    ct, cj = curves_from(t, ks), curves_from(j, ks)
    print("k            ", ks)
    print("true pass^k  ", [round(v, 3) for v in ct["pass_hat_k"]])
    print("judged pass^k", [round(v, 3) for v in cj["pass_hat_k"]])
    print("true pass@k  ", [round(v, 3) for v in ct["pass_at_k"]])
    print("judged pass@k", [round(v, 3) for v in cj["pass_at_k"]])
    rr = judge_rerun_agreement(fair, R[:120])
    print(f"the fair judge asked 5 times on 120 responses: all runs equal on {_f(rr['all_same'])}, mean flip rate {_f(rr['flip_rate'])}")
    print("\n== 9. an AutoMetrics-style metric induction toy")
    for row in induction_curve(W):
        print(f"labels {row['n']:4d}  induced metric tau {row['induced']:.3f}   holistic judge {row['baseline']:.3f}   holistic judge with its own taste {row['taste']:.3f}")
    tr = R[::2][:80]
    m = induce_metric(tr, [human_rating(r) for r in tr], n_keep=4)
    print("criteria kept from 80 labels:", {n: round(w, 2) for n, w in zip(m["names"], m["weights"])})
    print("\n== 10. the harness: seeds, cache, retries, log")
    tasks = [{"id": f"t{i}", "input": i} for i in range(30)]
    flaky = FlakyBackend(lambda task, rng: round(rng.random() + task["input"], 6), fail_rate=0.25, seed=3)
    h = EvalHarness(flaky, version="judge-v1", seed=7)
    first_pass = h.run(tasks, trials=2)
    s1 = h.summary()
    again = h.run(tasks, trials=2)
    s2 = h.summary()
    print("first run :", {k: v for k, v in s1.items() if k in ("ok", "cached", "retries", "errors", "slept")})
    print("second run:", {k: v for k, v in s2.items() if k in ("ok", "cached", "retries", "errors", "slept")}, "(all hits, no new backend calls)")
    h2 = EvalHarness(flaky, version="judge-v2", cache=h.cache, seed=7)
    h2.run(tasks[:5], trials=1)
    print("after changing the version, 5 tasks:", {k: v for k, v in h2.summary().items() if k in ("ok", "cached")})
    same = [r["result"] for r in first_pass if r["status"] in ("ok", "cached")] == [r["result"] for r in again if r["status"] in ("ok", "cached")]
    print("results identical on rerun:", same)
    hb = EvalHarness(lambda task, rng: 1, budget=10.0, seed=0)
    hb.run(tasks, 1)
    print("with a budget of 10 units at 1 per run:", {k: v for k, v in hb.summary().items() if k in ("ok", "skipped", "spent")})


if __name__ == "__main__":
    main()

Run the demo

$ python judge_infra_toolkit.py
== 1. code-based graders on the toy answers (the first of three grader types)
number_close finds the right answer in 189 of 300 responses; it agrees with the hidden trait on 1.000
but a human passes 151 of 300; code-based pass (correct only) vs human pass: agreement 0.860, kappa 0.720

== 2. position bias (600 pairs of different systems, each asked in both orders)
fair     consistency 0.772  first-slot rate 0.501
first    consistency 0.635  first-slot rate 0.652

== 3. verbosity bias (a response against itself plus 4 filler sentences)
fair     padded copy gets 0.440 of the credit (0.5 is fair)
wordy    padded copy gets 0.845 of the credit (0.5 is fair)

== 4. self-preference (the judge's own system is epsilon, which is weak and wordy)
no bias  judge win rate 0.301  human win rate 0.267  gap +0.034
selfish  judge win rate 0.428  human win rate 0.267  gap +0.161

== 5. mitigations: swap, length control, a panel
first-biased judge: consistency 0.635; after swapping, verdicts are consistent by construction, and a disagreement becomes a tie
verdicts out of 600 pairs against the hidden quality (right, wrong, tie):
  first-biased judge, raw order  : (473, 82, 45)
  first-biased judge, swapped    : (370, 7, 223)
epsilon vs delta with a wordy judge: raw win rate 0.550, length-controlled 0.216, human (from h) 0.300
  fair     alone, swapped: (428, 28, 144)
  first    alone, swapped: (370, 7, 223)
  wordy    alone, swapped: (416, 47, 137)
  selfish  alone, swapped: (421, 47, 132)
  panel of 4, swapped, no own-system votes: (513, 31, 56)

== 6. pointwise against pairwise: five systems ranked
true quality (mean h): {'alpha': 1.178, 'beta': 0.851, 'gamma': 0.786, 'delta': 0.801, 'epsilon': 0.409}
pointwise mean score: {'alpha': 3.7, 'beta': 3.22, 'gamma': 3.1, 'delta': 3.12, 'epsilon': 2.33}
Bradley-Terry Elo   : {'alpha': 1103, 'beta': 1023, 'gamma': 987, 'delta': 994, 'epsilon': 893}
judge calls: pointwise 300, pairwise 600 pairs x 2 orders = 1200
agreement with truth: pointwise {'kendall': 1.0, 'spearman': 1.0} BT {'kendall': 1.0, 'spearman': 1.0}
wordy judge, rank order: pointwise ['alpha', 'gamma', 'beta', 'epsilon', 'delta'] BT ['alpha', 'gamma', 'beta', 'epsilon', 'delta'] truth ['alpha', 'beta', 'delta', 'gamma', 'epsilon']
online Elo, 5 shuffles, alpha's rating: [1116, 1082, 1130, 1072, 1109]
bootstrap 95% interval of log-strength: {'alpha': (0.41, 0.78), 'beta': (-0.02, 0.34), 'gamma': (-0.24, 0.18), 'delta': (-0.18, 0.16), 'epsilon': (-0.79, -0.46)}

== 7. judge and human agreement
fair judge pass/fail: agreement 0.900, kappa 0.800, fail TPR/TNR {'tpr': 0.87248322147651, 'tnr': 0.9271523178807947}
fair judge 1 to 5: exact agreement 0.627, weighted kappa 0.907, Spearman 0.899
rare failures (5%): a judge that always says pass has agreement 0.953, kappa 0.000, fail TPR 0.00
calibration: expected calibration error 0.102
after a Platt fit on half: error on the other half 0.084 (before, same half: 0.123)
score to human pass rate: {1: (0.0, 70), 2: (0.07, 42), 3: (0.39, 44), 4: (0.83, 78), 5: (1.0, 66)}

== 8. pass@k and pass^k with a noisy grader (the agent is the same in both rows)
k             [1, 2, 4, 8, 16]
true pass^k   [0.739, 0.594, 0.435, 0.289, 0.183]
judged pass^k [0.7, 0.518, 0.313, 0.145, 0.067]
true pass@k   [0.739, 0.883, 0.958, 0.988, 1.0]
judged pass@k [0.7, 0.882, 0.969, 0.995, 1.0]
the fair judge asked 5 times on 120 responses: all runs equal on 0.717, mean flip rate 0.115

== 9. an AutoMetrics-style metric induction toy
labels   10  induced metric tau 0.427   holistic judge 0.713   holistic judge with its own taste 0.330
labels   20  induced metric tau 0.571   holistic judge 0.706   holistic judge with its own taste 0.324
labels   40  induced metric tau 0.571   holistic judge 0.709   holistic judge with its own taste 0.317
labels   80  induced metric tau 0.608   holistic judge 0.715   holistic judge with its own taste 0.303
labels  160  induced metric tau 0.638   holistic judge 0.718   holistic judge with its own taste 0.338
criteria kept from 80 labels: {'answer_looks_right': 0.8, 'cites_a_source': 0.4, 'shows_steps': 0.13, 'starts_with_then': 0.31}

== 10. the harness: seeds, cache, retries, log
first run : {'ok': 60, 'cached': 0, 'retries': 10, 'errors': 0, 'slept': 5.0}
second run: {'ok': 60, 'cached': 60, 'retries': 10, 'errors': 0, 'slept': 5.0} (all hits, no new backend calls)
after changing the version, 5 tasks: {'ok': 5, 'cached': 0}
results identical on rerun: True
with a budget of 10 units at 1 per run: {'ok': 10, 'skipped': 20, 'spent': 10.0}

Exercises

1. A new code grader. Add a grader that passes an answer only if it contains a number and a source phrase. How does its agreement with the human pass label change? What does it still not see [1]?
2. Find the bias from the outside. Build a judge with pos_bias of 0.2, 0.4 and 0.6. Plot consistency and the first-slot rate against the bias. At what size can you tell it from the fair judge with 600 pairs? Use the standard error of a proportion [2].
3. Noise or bias? Raise the fair judge's sigma from 0.25 to 0.5. What happens to consistency and to the first-slot rate? Explain why one of them moves and the other does not.
4. Padding length. Call verbosity_report with 1, 2, 4 and 8 filler sentences for the fair and the wordy judges. How does the credit grow with the length [2]?
5. Hide the self-preference. Change self_system to "alpha", the strongest system. Does the self-preference gap grow or shrink with the same self_bias? Why would the gap be harder to see for a strong system [13]?
6. Does swapping fix length? Build a judge with both pos_bias and len_bias. Count right, wrong and tie verdicts in one order and swapped. Which bias remains [2]?
7. A length-controlled win rate with more features. Add the difference in the number of numbered steps as a second feature in length_controlled_winrate. Does the estimate for the wordy judge move toward the human rate of 0.300 [15]?
8. A better panel. Use three judges that have the same bias (all with len_bias 0.25, different seeds). Does the panel help? Compare with the lab's panel, whose judges have different biases. What does this say about using the same model as three judges [2]?
9. Pairwise on a budget. Rank the five systems from a random 10%, 25% and 50% of the pairs. When does Kendall tau against the truth fall below 1? Compare with pointwise scores on the same number of calls [14].
10. Elo against Bradley-Terry. Run elo_online with k of 8, 32 and 64 over 20 shuffles. How does the spread of alpha's rating change? Which k is closest to the Bradley-Terry fit?
11. Rare failures. Use imbalanced_labels to make 2%, 5% and 20% of answers fail. For the fair judge, report agreement, kappa, the true positive rate with its Wilson interval, and the true negative rate [7].
12. A better calibration. Split the labels 50 against 250 and 150 against 150. Fit a Platt curve on the first part and test on the second. How much does the error depend on the size of the fitting set? Do five random splits and report the range.
13. Correlated grader errors. Change graded_runs so that the grader makes the same mistake on the same task every time with probability 0.5 and an independent mistake otherwise. How does the judged pass^8 change? Which assumption in 2.7 does this break [1]?
14. Induction with a hidden criterion. Add a criterion for "mentions a number from the question" that the toy world does not use. Does induce_metric ignore it? Add a criterion that is a copy of "shows its steps" and see what happens to the weights [3] [4].
15. Harness invalidation. Change the judge prompt in a function that you run under EvalHarness but leave version the same. Show that stale results are served. Then add a hash of the prompt to config and show that they are not [19].
16. Resume after a crash. Run the harness with log_path set, make the function raise a non-transient error on task 17, and catch it. Start a new harness with the old cache. How many tasks run again [18]?

The tests: test_judge_infra_toolkit.py

The 192 tests check the shared helpers, including pass@k, pass^k, the rank correlations and the Wilson interval on known values; every code grader; the toy world (size, determinism, that the answer text carries the hidden correctness trait, the padding helper, the pair sampler); the mock judge (determinism, each bias on its own, and that a judge with zero noise matches the human); the three bias tests, including a case with no judge bias and an empty case; the fixes (swapping is a mirror, a judge with only a position bias gives only ties after a swap, the panel drops a judge's vote on its own system, and the length-controlled estimate removes a planted length effect); Bradley-Terry on known odds, on ties and on a system with no losses, online Elo, and the bootstrap; kappa on a hand-worked table, weighted kappa, the true positive and true negative rates, calibration error and the Platt fit; the noisy-grader runs and their curves; ridge regression, the criteria, the induced metric and its exact prediction formula; the harness (seeds, order independence, cache keys, retries and backoff, errors not cached, the budget, the log file, and that a flaky backend gives the same answers as a clean one); a set of cases added after the mutation run; and the demo itself. Writing the lab found real faults in my first design, which I fixed: my flaky backend drew from the run's random generator and so changed the answers (the test that compares flaky and clean results found it); my first normalizer kept a trailing full stop, so "42." did not match "42"; the demo showed a self-preference gap of exactly zero for the fair judge because I had not given it an own system, which I fixed with a neutral twin of the selfish judge; the heading said 400 pairs when there are 600; the Platt fit had not converged; a "swap is symmetric" printout that compared the wrong things, which I replaced with counts of right, wrong and tied verdicts; and a few wrong expectations in my own tests. I also ran a mutation check, in which I made small deliberate changes to the code, one at a time, and ran the tests on each. This was a light, sampled run, not an exhaustive one (the toolkit has about 690 places where such a change can be made, and I sampled 70 and then 110 of them at random, with a fixed seed). The first sample of 70 was caught 44 times, which is a weak result. For the 26 misses I added 17 tests (defaults and empty cases, k equal to n, the noise levels and taste weights of the criteria, the exact prediction formula of the induced metric, dropping exactly the negative weights, the exact flip rate of a judge that is asked again, a pinned human preference, and the signature of the rare-failure helper). Then I drew a second sample of 110 changes, and the tests caught 93. Of the 17 that survived, I judge 12 to be harmless or equivalent: the text of an error message "k > n" in two places (the mutation changed the words, not the logic), a "greater than" against "greater than or equal" where the equal case is handled earlier or adds zero (in the Bradley-Terry update and in the panel vote), a "less than" against "less than or equal" or "greater than or equal" against "greater than" on a continuous random draw (three places), a check that dropped weights below 1 and not below 0 (the refit gives the same result when no weight is negative), a "greater than or equal to zero" against "greater than zero" that differs only for a weight of exactly zero, a perfect-grader default of 1.0 changed to 1.1 (a random draw is always below 1), and the unreachable "else" value of a criterion that every response satisfies. The other 5 were default arguments or boundaries that no test pinned: the number of iterations in the logistic fit, the inclusive edge of the number tolerance, the default number of criteria kept, the default number of bootstrap fits, and the first-slot rate when every verdict is a tie. After this second run I added one test for each of them (192 tests in all). I did not run the mutation sample again after that, so those 5 are covered by new tests that were not themselves mutation-checked. That says the tests are sensitive to many changes. It does not say the code is correct.

$ python test_judge_infra_toolkit.py -v
test_code_grader_vs_human (__main__.TestAgreement.test_code_grader_vs_human) ... ok
test_confusion (__main__.TestAgreement.test_confusion) ... ok
test_ece_overconfident (__main__.TestAgreement.test_ece_overconfident) ... ok
test_ece_perfect (__main__.TestAgreement.test_ece_perfect) ... ok
test_fair_judge_has_decent_agreement (__main__.TestAgreement.test_fair_judge_has_decent_agreement) ... ok
test_imbalanced_threshold (__main__.TestAgreement.test_imbalanced_threshold) ... ok
test_kappa_always_pass_is_zero (__main__.TestAgreement.test_kappa_always_pass_is_zero) ... ok
test_kappa_constant_both (__main__.TestAgreement.test_kappa_constant_both) ... ok
test_kappa_known_value (__main__.TestAgreement.test_kappa_known_value) ... ok
test_kappa_perfect_and_chance (__main__.TestAgreement.test_kappa_perfect_and_chance) ... ok
test_percent_agreement (__main__.TestAgreement.test_percent_agreement) ... ok
test_platt_improves (__main__.TestAgreement.test_platt_improves) ... ok
test_reliability_bins_sum (__main__.TestAgreement.test_reliability_bins_sum) ... ok
test_score_to_pass_rate (__main__.TestAgreement.test_score_to_pass_rate) ... ok
test_tpr_tnr (__main__.TestAgreement.test_tpr_tnr) ... ok
test_tpr_tnr_empty_class (__main__.TestAgreement.test_tpr_tnr_empty_class) ... ok
test_weighted_kappa_near_miss_beats_far_miss (__main__.TestAgreement.test_weighted_kappa_near_miss_beats_far_miss) ... ok
test_weighted_kappa_perfect (__main__.TestAgreement.test_weighted_kappa_perfect) ... ok
test_length_correlation_sign (__main__.TestBiasMeasurement.test_length_correlation_sign) ... ok
test_perfect_judge_consistent (__main__.TestBiasMeasurement.test_perfect_judge_consistent) ... ok
test_position_bias_lowers_consistency (__main__.TestBiasMeasurement.test_position_bias_lowers_consistency) ... ok
test_position_empty (__main__.TestBiasMeasurement.test_position_empty) ... ok
test_self_pref_bias_raises_gap (__main__.TestBiasMeasurement.test_self_pref_bias_raises_gap) ... ok
test_self_pref_n (__main__.TestBiasMeasurement.test_self_pref_n) ... ok
test_self_pref_none_without_system (__main__.TestBiasMeasurement.test_self_pref_none_without_system) ... ok
test_verbosity_bias_detected (__main__.TestBiasMeasurement.test_verbosity_bias_detected) ... ok
test_verbosity_n (__main__.TestBiasMeasurement.test_verbosity_n) ... ok
test_verbosity_unbiased_perfect_judge (__main__.TestBiasMeasurement.test_verbosity_unbiased_perfect_judge) ... ok
test_contains_all (__main__.TestCodeGraders.test_contains_all) ... ok
test_contains_all_empty (__main__.TestCodeGraders.test_contains_all_empty) ... ok
test_exact_match (__main__.TestCodeGraders.test_exact_match) ... ok
test_first_number (__main__.TestCodeGraders.test_first_number) ... ok
test_json_fields (__main__.TestCodeGraders.test_json_fields) ... ok
test_number_close (__main__.TestCodeGraders.test_number_close) ... ok
test_regex (__main__.TestCodeGraders.test_regex) ... ok
test_run_assertions (__main__.TestCodeGraders.test_run_assertions) ... ok
test_run_assertions_empty (__main__.TestCodeGraders.test_run_assertions_empty) ... ok
test_run_assertions_unknown (__main__.TestCodeGraders.test_run_assertions_unknown) ... ok
test_tool_calls_forbidden (__main__.TestCodeGraders.test_tool_calls_forbidden) ... ok
test_tool_calls_subsequence (__main__.TestCodeGraders.test_tool_calls_subsequence) ... ok
test_tool_calls_tuples (__main__.TestCodeGraders.test_tool_calls_tuples) ... ok
test_bootstrap_default_count (__main__.TestDefaultsAndEdges.test_bootstrap_default_count) ... ok
test_bootstrap_default_seed (__main__.TestDefaultsAndEdges.test_bootstrap_default_seed) ... ok
test_criteria_values_and_noise_levels (__main__.TestDefaultsAndEdges.test_criteria_values_and_noise_levels) ... ok
test_drop_negative_off_when_all_positive (__main__.TestDefaultsAndEdges.test_drop_negative_off_when_all_positive) ... ok
test_drop_negative_removes_exactly_the_negatives (__main__.TestDefaultsAndEdges.test_drop_negative_removes_exactly_the_negatives) ... ok
test_fit_logistic_default_pinned (__main__.TestDefaultsAndEdges.test_fit_logistic_default_pinned) ... ok
test_graded_runs_default_is_perfect (__main__.TestDefaultsAndEdges.test_graded_runs_default_is_perfect) ... ok
test_human_prefers_pinned (__main__.TestDefaultsAndEdges.test_human_prefers_pinned) ... ok
test_human_prefers_tie_zone (__main__.TestDefaultsAndEdges.test_human_prefers_tie_zone) ... ok
test_imbalanced_labels_signature (__main__.TestDefaultsAndEdges.test_imbalanced_labels_signature) ... ok
test_induce_default_keeps_five (__main__.TestDefaultsAndEdges.test_induce_default_keeps_five) ... ok
test_judge_defaults_equal_k0 (__main__.TestDefaultsAndEdges.test_judge_defaults_equal_k0) ... ok
test_judge_rerun_flip_rate_exact (__main__.TestDefaultsAndEdges.test_judge_rerun_flip_rate_exact) ... ok
test_kendall_joint_ties_are_skipped (__main__.TestDefaultsAndEdges.test_kendall_joint_ties_are_skipped) ... ok
test_number_close_boundary_is_inclusive (__main__.TestDefaultsAndEdges.test_number_close_boundary_is_inclusive) ... ok
test_pass_k_with_k_equal_n (__main__.TestDefaultsAndEdges.test_pass_k_with_k_equal_n) ... ok
test_position_report_all_ties (__main__.TestDefaultsAndEdges.test_position_report_all_ties) ... ok
test_predict_matches_manual_formula (__main__.TestDefaultsAndEdges.test_predict_matches_manual_formula) ... ok
test_random_order_default_k0 (__main__.TestDefaultsAndEdges.test_random_order_default_k0) ... ok
test_reliability_bins_empty_bin (__main__.TestDefaultsAndEdges.test_reliability_bins_empty_bin) ... ok
test_self_pref_empty_values (__main__.TestDefaultsAndEdges.test_self_pref_empty_values) ... ok
test_taste_pinned (__main__.TestDefaultsAndEdges.test_taste_pinned) ... ok
test_budget (__main__.TestHarness.test_budget) ... ok
test_cache_hit (__main__.TestHarness.test_cache_hit) ... ok
test_cached_runs_cost_nothing (__main__.TestHarness.test_cached_runs_cost_nothing) ... ok
test_config_busts_cache (__main__.TestHarness.test_config_busts_cache) ... ok
test_cost_fn (__main__.TestHarness.test_cost_fn) ... ok
test_errors_not_cached_and_rerun_succeeds (__main__.TestHarness.test_errors_not_cached_and_rerun_succeeds) ... ok
test_flaky_deterministic (__main__.TestHarness.test_flaky_deterministic) ... ok
test_flaky_does_not_consume_the_generator (__main__.TestHarness.test_flaky_does_not_consume_the_generator) ... ok
test_flaky_results_equal_clean_results (__main__.TestHarness.test_flaky_results_equal_clean_results) ... ok
test_key_includes_input_and_trial (__main__.TestHarness.test_key_includes_input_and_trial) ... ok
test_log_file_jsonl (__main__.TestHarness.test_log_file_jsonl) ... ok
test_log_records (__main__.TestHarness.test_log_records) ... ok
test_order_independence (__main__.TestHarness.test_order_independence) ... ok
test_other_errors_propagate (__main__.TestHarness.test_other_errors_propagate) ... ok
test_retry_gives_up (__main__.TestHarness.test_retry_gives_up) ... ok
test_retry_then_success (__main__.TestHarness.test_retry_then_success) ... ok
test_run_counts (__main__.TestHarness.test_run_counts) ... ok
test_seed_changes_results (__main__.TestHarness.test_seed_changes_results) ... ok
test_sleep_injected (__main__.TestHarness.test_sleep_injected) ... ok
test_summary_zero (__main__.TestHarness.test_summary_zero) ... ok
test_trials_differ_seeds_same (__main__.TestHarness.test_trials_differ_seeds_same) ... ok
test_version_busts_cache (__main__.TestHarness.test_version_busts_cache) ... ok
test_clip (__main__.TestHelpers.test_clip) ... ok
test_comb_edges (__main__.TestHelpers.test_comb_edges) ... ok
test_k_larger_than_n_raises (__main__.TestHelpers.test_k_larger_than_n_raises) ... ok
test_kendall_all_tied (__main__.TestHelpers.test_kendall_all_tied) ... ok
test_kendall_perfect_and_reverse (__main__.TestHelpers.test_kendall_perfect_and_reverse) ... ok
test_kendall_ties_known_value (__main__.TestHelpers.test_kendall_ties_known_value) ... ok
test_mean (__main__.TestHelpers.test_mean) ... ok
test_mean_empty (__main__.TestHelpers.test_mean_empty) ... ok
test_normalize (__main__.TestHelpers.test_normalize) ... ok
test_pass_at_k_known (__main__.TestHelpers.test_pass_at_k_known) ... ok
test_pass_hat_k_known (__main__.TestHelpers.test_pass_hat_k_known) ... ok
test_pass_hat_never_above_pass_at (__main__.TestHelpers.test_pass_hat_never_above_pass_at) ... ok
test_pearson_constant (__main__.TestHelpers.test_pearson_constant) ... ok
test_pearson_perfect (__main__.TestHelpers.test_pearson_perfect) ... ok
test_ranks_ties (__main__.TestHelpers.test_ranks_ties) ... ok
test_sigmoid_extremes (__main__.TestHelpers.test_sigmoid_extremes) ... ok
test_sigmoid_symmetry (__main__.TestHelpers.test_sigmoid_symmetry) ... ok
test_spearman_monotone (__main__.TestHelpers.test_spearman_monotone) ... ok
test_stdev (__main__.TestHelpers.test_stdev) ... ok
test_stdev_short (__main__.TestHelpers.test_stdev_short) ... ok
test_wilson (__main__.TestHelpers.test_wilson) ... ok
test_criteria_look_at_text (__main__.TestInduction.test_criteria_look_at_text) ... ok
test_criterion_scores_shape_and_determinism (__main__.TestInduction.test_criterion_scores_shape_and_determinism) ... ok
test_drop_negative (__main__.TestInduction.test_drop_negative) ... ok
test_induce_beats_random (__main__.TestInduction.test_induce_beats_random) ... ok
test_induce_predict_length (__main__.TestInduction.test_induce_predict_length) ... ok
test_induce_selects_known_drivers (__main__.TestInduction.test_induce_selects_known_drivers) ... ok
test_induced_beats_mismatched_judge_with_labels (__main__.TestInduction.test_induced_beats_mismatched_judge_with_labels) ... ok
test_induction_curve_shape (__main__.TestInduction.test_induction_curve_shape) ... ok
test_more_labels_help (__main__.TestInduction.test_more_labels_help) ... ok
test_ridge_recovers (__main__.TestInduction.test_ridge_recovers) ... ok
test_ridge_shrinks (__main__.TestInduction.test_ridge_shrinks) ... ok
test_solve_linear (__main__.TestInduction.test_solve_linear) ... ok
test_solve_singular (__main__.TestInduction.test_solve_singular) ... ok
test_zscore (__main__.TestInduction.test_zscore) ... ok
test_deterministic (__main__.TestMain.test_deterministic) ... ok
test_no_dashes_in_output (__main__.TestMain.test_no_dashes_in_output) ... ok
test_pinned_lines (__main__.TestMain.test_pinned_lines) ... ok
test_ten_sections (__main__.TestMain.test_ten_sections) ... ok
test_fit_logistic_recovers (__main__.TestMitigations.test_fit_logistic_recovers) ... ok
test_lc_no_length_effect (__main__.TestMitigations.test_lc_no_length_effect) ... ok
test_lc_removes_length_effect (__main__.TestMitigations.test_lc_removes_length_effect) ... ok
test_lc_wordy_judge_demo (__main__.TestMitigations.test_lc_wordy_judge_demo) ... ok
test_panel_excludes_own (__main__.TestMitigations.test_panel_excludes_own) ... ok
test_panel_includes_own_when_asked (__main__.TestMitigations.test_panel_includes_own_when_asked) ... ok
test_panel_tie_on_split (__main__.TestMitigations.test_panel_tie_on_split) ... ok
test_random_order_deterministic_with_seed (__main__.TestMitigations.test_random_order_deterministic_with_seed) ... ok
test_random_order_unbiased_on_average (__main__.TestMitigations.test_random_order_unbiased_on_average) ... ok
test_swapped_fewer_wrong (__main__.TestMitigations.test_swapped_fewer_wrong) ... ok
test_swapped_mirror (__main__.TestMitigations.test_swapped_mirror) ... ok
test_swapped_pos_bias_only_ties (__main__.TestMitigations.test_swapped_pos_bias_only_ties) ... ok
test_system_pair_rows_count (__main__.TestMitigations.test_system_pair_rows_count) ... ok
test_deterministic (__main__.TestMockJudge.test_deterministic) ... ok
test_different_names_differ (__main__.TestMockJudge.test_different_names_differ) ... ok
test_k_changes_call_noise (__main__.TestMockJudge.test_k_changes_call_noise) ... ok
test_len_bias_raises_long (__main__.TestMockJudge.test_len_bias_raises_long) ... ok
test_pass_confidence_range_and_order (__main__.TestMockJudge.test_pass_confidence_range_and_order) ... ok
test_pointwise_range (__main__.TestMockJudge.test_pointwise_range) ... ok
test_pos_bias_flips_equal_pair (__main__.TestMockJudge.test_pos_bias_flips_equal_pair) ... ok
test_pos_bias_is_added_to_first_only (__main__.TestMockJudge.test_pos_bias_is_added_to_first_only) ... ok
test_self_bias_only_for_own_system (__main__.TestMockJudge.test_self_bias_only_for_own_system) ... ok
test_taste_judge_ignores_h (__main__.TestMockJudge.test_taste_judge_ignores_h) ... ok
test_tie_margin (__main__.TestMockJudge.test_tie_margin) ... ok
test_zero_noise_matches_human (__main__.TestMockJudge.test_zero_noise_matches_human) ... ok
test_zero_noise_pairwise (__main__.TestMockJudge.test_zero_noise_pairwise) ... ok
test_curve_k1_equal (__main__.TestPassK.test_curve_k1_equal) ... ok
test_curves_monotone (__main__.TestPassK.test_curves_monotone) ... ok
test_deterministic (__main__.TestPassK.test_deterministic) ... ok
test_graded_runs_shape (__main__.TestPassK.test_graded_runs_shape) ... ok
test_judge_rerun_noisy (__main__.TestPassK.test_judge_rerun_noisy) ... ok
test_judge_rerun_zero_noise (__main__.TestPassK.test_judge_rerun_zero_noise) ... ok
test_noisy_grader_lowers_pass_hat (__main__.TestPassK.test_noisy_grader_lowers_pass_hat) ... ok
test_noisy_grader_raises_pass_at_for_weak_tasks (__main__.TestPassK.test_noisy_grader_raises_pass_at_for_weak_tasks) ... ok
test_perfect_grader_equal (__main__.TestPassK.test_perfect_grader_equal) ... ok
test_bootstrap_interval_contains_point (__main__.TestRanking.test_bootstrap_interval_contains_point) ... ok
test_bt_finite_with_no_losses (__main__.TestRanking.test_bt_finite_with_no_losses) ... ok
test_bt_known_odds (__main__.TestRanking.test_bt_known_odds) ... ok
test_bt_mean_zero (__main__.TestRanking.test_bt_mean_zero) ... ok
test_bt_on_world_matches_truth_with_good_judge (__main__.TestRanking.test_bt_on_world_matches_truth_with_good_judge) ... ok
test_bt_order (__main__.TestRanking.test_bt_order) ... ok
test_bt_ties_count_half (__main__.TestRanking.test_bt_ties_count_half) ... ok
test_bt_two_equal_systems (__main__.TestRanking.test_bt_two_equal_systems) ... ok
test_comparisons_from (__main__.TestRanking.test_comparisons_from) ... ok
test_elo_order_dependence (__main__.TestRanking.test_elo_order_dependence) ... ok
test_elo_seed_repeatable (__main__.TestRanking.test_elo_seed_repeatable) ... ok
test_elo_updates_zero_sum (__main__.TestRanking.test_elo_updates_zero_sum) ... ok
test_pointwise_system_scores (__main__.TestRanking.test_pointwise_system_scores) ... ok
test_ranking_agreement (__main__.TestRanking.test_ranking_agreement) ... ok
test_to_elo_scale (__main__.TestRanking.test_to_elo_scale) ... ok
test_wordy_judge_flips_close_systems (__main__.TestRanking.test_wordy_judge_flips_close_systems) ... ok
test_deterministic (__main__.TestWorld.test_deterministic) ... ok
test_epsilon_is_longest (__main__.TestWorld.test_epsilon_is_longest) ... ok
test_human_pass_threshold (__main__.TestWorld.test_human_pass_threshold) ... ok
test_human_prefers_big_gap (__main__.TestWorld.test_human_prefers_big_gap) ... ok
test_human_prefers_deterministic (__main__.TestWorld.test_human_prefers_deterministic) ... ok
test_other_seed_differs (__main__.TestWorld.test_other_seed_differs) ... ok
test_pad_does_not_change_original (__main__.TestWorld.test_pad_does_not_change_original) ... ok
test_pad_response (__main__.TestWorld.test_pad_response) ... ok
test_pair_list_sampling (__main__.TestWorld.test_pair_list_sampling) ... ok
test_pairs_same_question_different_system (__main__.TestWorld.test_pairs_same_question_different_system) ... ok
test_rating_of (__main__.TestWorld.test_rating_of) ... ok
test_size (__main__.TestWorld.test_size) ... ok
test_system_quality_order (__main__.TestWorld.test_system_quality_order) ... ok
test_text_encodes_the_correct_trait (__main__.TestWorld.test_text_encodes_the_correct_trait) ... ok
test_unique_ids (__main__.TestWorld.test_unique_ids) ... ok
test_words_count (__main__.TestWorld.test_words_count) ... ok
test_zlen_standardized (__main__.TestWorld.test_zlen_standardized) ... ok

----------------------------------------------------------------------
Ran 192 tests in 9.297s

OK

5. Discussion questions

Use these in a study group or a class. Try to answer with an example and a source. Give each about 5 to 10 minutes.

1. Pick an agent task and split it into the things you could check. Which parts would you give to a code-based grader, which to a model-based grader, and which to a person? What would each choice miss [1]?
2. After you add order swapping to a judge, wrong verdicts fall from 82 to 7. In the lab the fair judge also fell, from 81 to 28. What else besides a position bias could explain a fall, and what would you measure first to tell them apart [2]?
3. Zheng and colleagues report agreement with the ties left out (S2) as well as with them (S1). Which number would you give to a product manager, and what would you say about the other [2]?
4. Few-shot examples raised a judge's consistency from 65.0% to 77.5%, and the paper warns that consistency need not mean accuracy. Design a check that tells the two apart with 100 human labels [2].
5. You can only afford one test for self-preference. You have outputs from two models and a judge that is one of them. What would you compare, and what would still be uncertain [2] [13]?
6. A length-controlled win rate fits the judge's preference against the length difference. Name a case in which longer answers really are better, so that the correction would be wrong. How would you notice [15]?
7. You must rank 20 candidate prompts on 100 questions, with a budget of 5,000 judge calls. Would you use pointwise scores, pairwise comparisons with swapping, or a mix? Show the arithmetic [2] [14].
8. A judge for a rare failure (3% of answers) has 97% agreement with people. List what you would ask for before you trust it, and how many labeled failures you would collect [7].
9. An internal tool that runs overnight and a customer-facing agent both use the same model. Choose pass@k or pass^k for each, say what k is, and say what a noisy grader would do to your number [1].
10. An induced metric gives "hedges" a negative weight, and the AutoMetrics filter would drop negative weights. Who should decide, and what would you read first [3] [4]?
11. You run 1,000 tasks every night on a flaky API. Rank the six harness features of 2.10 by what you would build first, and say what each protects you from [18] [19].
12. Two of the three required readings are by the course's instructors or by a vendor about its own practice. Which claims would you want checked by someone else, and how would you check them [1] [3]?

6. Quiz (10 questions)

Pick one answer for each question, then press Check answers. Check answers 0 of 10 answered.

1. What does the Anthropic post advise about the three grader types [1]?
   • Use model-based graders first because they are the most flexible, and add code checks only where a model grader turns out to be too slow
   • Use people for every task, because only a person can tell whether an agent did well, and treat code and models as a way to save time
   • Choose deterministic graders where you can, model graders where you must, and use people mainly to validate the other two kinds
   • Pick the grader that is cheapest for each task, and avoid mixing types in one suite because scores cannot be compared
2. In Zheng and colleagues' conservative approach to position bias, when does an answer win a pair [2]?
   • When the same answer wins in both orders, and when the two orders disagree the pair is called a tie with no winner
   • When it wins in the first order that is tried, and the second order is used only to break a tie between two close answers
   • When it wins more of a set of random orders than the other answer, with the number of orders chosen by the judge
   • When the judge's average score for it, over both orders, is higher than the score for the other answer by at least one point
3. A judge and a person label 100 answers. Both pass 50, both fail 30, the judge passes and the person fails 12, and the judge fails and the person passes 8. What is Cohen's kappa (my arithmetic)?
   • 0.80, because kappa equals the observed agreement when the two raters have similar pass rates
   • 0.20, because that is the share of answers on which the judge and the person disagree with each other
   • 0.52, because kappa is the share of agreement that is left after the two raters are compared
   • 0.58, because chance agreement is 0.52 and so (0.80 − 0.52) / (1 − 0.52) is about 0.58
4. An agent passes each trial with probability 0.8, and trials are independent. What is pass^4 (my arithmetic)?
   • 0.80, because pass^k stays at the per-trial rate however many trials you ask for
   • 0.41, because all four trials must pass, which is 0.8 × 0.8 × 0.8 × 0.8
   • 0.998, because at least one of four trials passes almost always when each passes 80% of the time
   • 0.20, because that is the chance that a single trial fails, which sets the ceiling for four trials
5. In the lab the true pass^8 was 0.289 and the pass^8 from the noisy grader was 0.145. What does that show [1]?
   • That the agent got worse when it was graded by a model, so the grader must be changing how the agent behaves
   • That a noisy grader helps an agent, since it makes more of the failed trials look like they have passed
   • That pass@k and pass^k are the same measure, because both of them were computed from the same set of trials
   • That grader noise roughly halves a consistency metric, while pass@k from the same trials moves very little
6. Which number is the cleaner sign of a position bias in a judge [2]?
   • Consistency across the two orders, since a perfect judge always gives 100% and any lower value is bias
   • The share of ties, since a judge that gives many ties must be favoring one of the two slots
   • The first-slot rate among decided verdicts, since noise alone gives about 50% and a bias moves it
   • The judge's mean score, because a position bias always raises or lowers the scores of every answer
7. In the lab, the wordy judge still had 47 wrong verdicts after swapping. Why [2]?
   • It prefers the longer answer in both orders, so the two calls agree and swapping cannot remove the bias
   • The swap was applied wrongly, because the second call should have been given the same order as the first
   • Swapping adds noise to every judge, so a judge that is swapped has more wrong verdicts than before
   • The human labels are wrong for long answers, so the verdicts of a good judge look wrong in the toy
8. You compare 8 systems on 50 questions, pairwise, with every pair judged in both orders. How many judge calls (my arithmetic)?
   • 400, which is one call for each answer, as in a pointwise run of 8 systems on 50 questions
   • 2,800, because there are 28 pairs, 50 questions and 2 orders, and 28 × 50 × 2 is 2,800
   • 1,400, because there are 28 pairs and 50 questions, and the second order is free when the call is cached
   • 5,600, because each of the 8 systems is compared with the 7 others on 50 questions in 2 orders
9. What goes wrong in a harness if the cache key leaves out the trial number [19]?
   • Nothing, because trials of the same task are meant to give the same result, so a cache hit is correct for all of them
   • The cache grows without limit, because each trial then stores one more entry than it would have stored before
   • Repeated trials of one task return the first stored result, so pass^k looks perfect and the variance is hidden
   • Retries stop working, since a retry must have a different trial number to be allowed to call the backend again
10. In the lab's induction toy, how did the induced metric compare with the two holistic judges (made-up data) [3]?
    • It stayed below the judge that sees quality, and above the judge with its own taste, at every size
    • It beat both judges once it had 80 labels, which matches the paper's claim for real tasks
    • It was below both judges at every number of labels, and more labels never made any difference to it
    • It matched the judge that sees the quality from 20 labels on, which shows the method needs few labels

Answer key

7. Graded assignment and rubric

Title: Build, test and report on a small judge and eval harness on a mock.
Time: 5 to 6 hours. Total: 100 points. Work: solo.

The task

Extend the lab code and write a short memo (three pages at most). Everything is mock: no real model, no real data, and no network. Build your own domain, such as a travel-itinerary assistant, a code-review bot or a recipe helper. Do not reuse the lab's world of numeric answers, its five systems or its ten sections. Label every made-up number as made up, and every calculation of your own as your own. Say what a reader must not conclude from a mock judge.

Part A. A world, graders and a judge (25 points)

• Build a made-up set of at least 40 tasks and at least 4 systems, with a hidden quality for each answer and a human rating on a scale from 1 to 5. State how you made the human rating.
• Write at least three code-based graders for your domain, and say what each one can and cannot see. Report each one's agreement and kappa with the human pass label.
• Write a mock judge with a noise setting and at least three bias settings: position, length and self-preference, or others that fit your domain. Show that with every bias off it matches the human rating when its noise is zero.
• State three limits of your world and your judge.

Part B. Measure and reduce the biases (25 points)

• For each bias, write a designed test, as in 2.3, and show the number for a fair judge and for a biased one, with at least 300 pairs or items for the position test.
• Apply at least three fixes: swapping, a length control or a length-matched test, and a panel or another fix of your choice. Report right, wrong and tied verdicts against the hidden quality, for each judge and each fix.
• Find a case in which a fix does not help, or helps for a reason that you did not expect (as swapping helped the fair judge in the lab). Explain it with a number.
• State three limits of your tests.

Part C. Rankings, agreement and calibration (25 points)

• Rank your systems with pointwise scores and with pairwise comparisons fitted by Bradley-Terry. Report the number of judge calls for each, the rank agreement with the truth, and bootstrap intervals for the strengths. Say which pairs of systems the data cannot separate.
• Make a pass or fail label set in which 5% of answers fail. Report agreement, kappa, the true positive and true negative rates with a Wilson interval, and show that an always-pass judge looks good on agreement. Work one kappa by hand.
• Report a reliability table for your judge's scores, fit a Platt curve on one half of the labels, and report the error on the other half for five random splits.
• State three limits of your agreement study.

Part D. Reliability, harness and memo (25 points)

• Run at least 12 trials for each of at least 40 tasks. Report pass@k and pass^k at k = 1, 2, 4 and 8, from the truth and from a noisy grader. Say what happens if the grader repeats its mistakes on the same task, and test it.
• Put your judge calls behind a harness with per-run seeds, a cache, retries, a log and a budget. Show with tests that the results do not depend on the order of the runs, that a changed prompt version does not read stale results, and that a flaky backend gives the same answers as a clean one.
• Write a memo of two pages at most to a team that wants to use an LLM judge for a real product. It must say what you would check before trusting the judge, how many human labels you would collect and for which labels, which bias tests you would run, whether you would report pass@k or pass^k, and what the toy cannot tell them.
• State three limits of your harness and your memo.

What to hand in

• Your judge_infra_toolkit.py (or your own file names), your mock world, and the tests. All tests must pass.
• A memo in PDF or Markdown with the tables from Parts A to D and the memo from Part D.
• A sources list for any reading you cite.

Rubric

Criterion	Full marks	Partial	Points
A. A world, graders and a judge	40 or more tasks and 4 or more systems in a new domain, a stated human rating, three code graders with what each can and cannot see and their agreement, a mock judge with noise and three biases that matches the human at zero noise, three limits.	Fewer tasks or systems, or two graders, or no statement of what the graders miss, or fewer than three biases, or no zero-noise check.	25
B. Measure and reduce the biases	A designed test for each bias with numbers for a fair and a biased judge, 300 or more pairs for position, three fixes with right, wrong and tied counts, a case in which a fix does not help or helps for another reason, explained with a number, three limits.	Fewer tests or fixes, or only accuracy among decided verdicts, or no surprising case, or no counts of ties.	25
C. Rankings, agreement and calibration	Pointwise and Bradley-Terry rankings with call counts, rank agreement and bootstrap intervals, a 5% failure label set with kappa and true positive and negative rates and a Wilson interval, a hand-worked kappa, a reliability table and a Platt fit over five splits, three limits.	No call counts, or no intervals, or no rare-failure set, or no hand calculation, or one split only.	25
D. Reliability, harness and memo	pass@k and pass^k from the truth and from a noisy grader, a test of repeated grader mistakes, a harness with seeds, cache, retries, log and budget and the three tests, a two-page memo that answers every question in the list and says what the toy cannot show, three limits.	Fewer trials, or no repeated-mistake test, or a harness without one of the three tests, or a memo that overclaims or leaves out the label counts.	25
Total			100

Grade bands: 90 to 100 excellent. 75 to 89 good. 60 to 74 needs work. Under 60: redo and resubmit. Plain writing is rewarded: short sentences, no filler. Label your own arithmetic as your own.

8. Glossary

Term	Meaning
Agent-as-a-Judge	An agentic system, with tools, that evaluates another agentic system and gives feedback on the whole process [17] (2.9).
Bradley-Terry model	A model in which the chance that A beats B depends on the difference of their strengths. It is fitted to pairwise results [14] (2.5).
Calibration	A judge's stated confidence matches how often it is right. A reliability diagram and the expected calibration error (ECE) check it (2.6).
Cohen's kappa	Agreement between two raters after removing the agreement expected by chance: (po − pe) / (1 − pe) (2.6).
Consistency (position)	The share of pairs on which the same answer wins in both orders [2]. It mixes bias with noise (2.3).
Criteria drift	People need criteria to grade outputs, and grading outputs changes their criteria [16].
Critique shadowing	Husain's procedure: an expert gives a pass or fail and a written critique, and the critiques are used to write and test the judge prompt [7].
Elo rating	A rating updated after each comparison. It depends on the order of the comparisons, unlike a Bradley-Terry fit to all of them at once (2.5).
First-slot rate	The share of decided verdicts that pick the answer shown first. Noise alone gives about 50% (2.3).
Grader (code, model, human)	The three kinds of grader in the Anthropic post [1] (2.1).
Harness (evaluation)	The code that runs the tasks, calls the agent and the graders, and records what happened [1] (2.10).
Length-controlled win rate	A win rate predicted at zero length difference from a model of the judge's preference [15] (2.4).
LLM-as-a-judge	Using a language model to grade or compare answers [2].
Metric induction	Building a set of criteria and their weights from human feedback, for example AutoMetrics [3] and AutoLibra [4] (2.8).
pass@k	The chance that at least one of k trials passes [22].
pass^k	The chance that all k trials pass [21].
Pairwise and pointwise	Pairwise shows two answers and asks which is better. Pointwise shows one and asks for a score or a verdict [2] (2.5).
Panel of judges	Several judges whose verdicts are combined. It helps most when their biases differ (2.4).
Platt scaling	A logistic curve that maps a judge's score to the share of answers that humans pass (2.6).
Position bias	The verdict depends on which answer is shown first [2] (2.3).
Prevalence trap	When failures are rare, raw agreement is high even for a judge that finds none of them [7] (2.6).
Reference-guided judging	The judge is given a reference answer, which helped in the paper's math tests [2].
Self-enhancement or self-preference	A judge favors answers from its own model [2] [13] (2.3).
Swap (conservative)	Ask in both orders and call a win only if the same answer wins both times; otherwise a tie [2] (2.4).
True positive rate, true negative rate	The share of real failures that the judge flags, and the share of real passes that it lets through (2.6).
Verbosity bias	The judge prefers the longer answer even when it adds nothing [2] (2.3).
Weighted kappa	Kappa that gives partial credit for near misses on an ordered scale (2.6).
Wilson interval	A confidence interval for a rate that stays sensible near 0 and 1 and for small n (my own arithmetic in the lab, 2.6).

9. References

Entries marked (required) are the course's required reading, and (course, additional) is the course's additional reading. Entries marked (added) are my own additions. The course site and the slide test, [5] and [6], are the course's own pages. "Abstract level" means that I read the abstract and not the whole paper. Where an entry names more, that is what I read. Web pages were read in October 2026.

• [1] Grace, M., Hadfield, J., Olivares, R., and De Jonghe, J. "Demystifying evals for AI agents." Anthropic engineering blog, 9 January 2026. Vendor-authored. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents (I read the whole post; it describes the authors' own practice and its customers' examples, it has no experiments, and its section "Going from zero to one" has nine headings, Step 0 to Step 8) (required)
• [2] Zheng, L., Chiang, W.-L., Sheng, Y., and 10 others, including J. E. Gonzalez and I. Stoica. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685 (2023). Zheng, Chiang and Stoica are also authors of [14]. https://arxiv.org/abs/2306.05685 (I read the whole paper text, including the appendix tables; the numbers are the authors' own, for the models and prompts of 2023, and I did not rerun anything) (required)
• [3] Ryan, M. J., Zhang, Y., Salunkhe, A., Chu, Y., Xu, D., and Yang, D. "AutoMetrics: Approximate Human Judgements with Automatically Generated Evaluators." ICLR 2026. arXiv:2512.17267 (2025). Ryan and Yang are instructors of the course. https://arxiv.org/abs/2512.17267 (I read the arXiv HTML body text and not the appendices; the authors report their own results and I did not rerun anything) (required)
• [4] Zhu, H., Cuvin, P., Yu, X., Yan, C. K. Y., Zhang, J., and Yang, D. "AutoLibra: Agent Metric Induction from Open-Ended Human Feedback." ICLR 2026. arXiv:2505.02820 (2025). Yang is an instructor of the course. https://arxiv.org/abs/2505.02820 (abstract level only; the authors' own claims) (course, additional)
• [5] Stanford CS329Z: Engineering AI Agents, course site and schedule (Diyi Yang, Michael Ryan, John Yang). https://cs329z.stanford.edu/ (the course's own page, read on 29 October 2026; it lists "LLM-as-Judge & Evaluation Infrastructure" and its readings under Wednesday, 11 November 2026, lists Monday, 9 November 2026 as a fireside chat with Andrew Ng, and says the schedule is tentative)
• [6] CS329Z slide addresses lecture14.pdf and lecture15.pdf, each tried once on 29 October 2026. https://web.stanford.edu/class/cs329z/slides/lecture15.pdf (both addresses returned "not found" (HTTP 404), and the slide folder's index listed only lecture01.pdf to lecture11.pdf, so I could not use slides for this week)
• [7] Husain, H. "Using LLM-as-a-Judge For Evaluation: A Complete Guide." Blog post, published 29 October 2024, shown as modified 1 September 2026. https://hamel.dev/blog/posts/llm-judge/ (I read the whole post as it stood on 29 October 2026; it is practitioner advice with no experiments, and the page advertises the author's paid course) (added)
• [8] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment." arXiv:2303.16634 (2023). https://arxiv.org/abs/2303.16634 (abstract level only; the authors' own claims) (added)
• [9] Kim, S., Shin, J., Cho, Y., Jang, J., and 7 others. "Prometheus: Inducing Fine-grained Evaluation Capability in Language Models." arXiv:2310.08491 (2023). https://arxiv.org/abs/2310.08491 (abstract level only; the authors' own claims) (added)
• [10] Kim, S., Suk, J., Longpre, S., Lin, B. Y., and 6 others. "Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models." arXiv:2405.01535 (2024). https://arxiv.org/abs/2405.01535 (abstract level only; the authors' own claims) (added)
• [11] Zhu, L., Wang, X., and Wang, X. "JudgeLM: Fine-tuned Large Language Models are Scalable Judges." arXiv:2310.17631 (2023). https://arxiv.org/abs/2310.17631 (abstract level only; the authors' own claims) (added)
• [12] Wang, P., Li, L., Chen, L., Cai, Z., and 6 others. "Large Language Models are not Fair Evaluators." arXiv:2305.17926 (2023). https://arxiv.org/abs/2305.17926 (abstract level only; the authors' own claims) (added)
• [13] Panickssery, A., Bowman, S. R., and Feng, S. "LLM Evaluators Recognize and Favor Their Own Generations." arXiv:2404.13076 (2024). https://arxiv.org/abs/2404.13076 (abstract level only; the authors' own claims) (added)
• [14] Chiang, W.-L., Zheng, L., Sheng, Y., and 8 others, including I. Stoica. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." arXiv:2403.04132 (2024). Chiang, Zheng and Stoica are also authors of [2]. https://arxiv.org/abs/2403.04132 (abstract level only; the authors' own claims) (added)
• [15] Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. "Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators." arXiv:2404.04475 (2024). https://arxiv.org/abs/2404.04475 (abstract level only; the authors' own claims) (added)
• [16] Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., and Arawjo, I. "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences." arXiv:2404.12272 (2024). https://arxiv.org/abs/2404.12272 (abstract level only for this week; the authors' own claims; I used it in Week 7a) (added)
• [17] Zhuge, M., Zhao, C., Ashley, D., Wang, W., and 9 others, including J. Schmidhuber. "Agent-as-a-Judge: Evaluate Agents with Agents." arXiv:2410.10934 (2024). https://arxiv.org/abs/2410.10934 (abstract level only; the authors' own claims and benchmark) (added)
• [18] UK AI Security Institute. Inspect AI documentation: overview and the page on scorers. https://inspect.aisi.org.uk/scorers.html (I read the overview and the scorers page and did not run the tool) (added)
• [19] promptfoo documentation: "Caching Configuration." https://www.promptfoo.dev/docs/configuration/caching/ (I read this page and did not run the tool; it is the vendor's own description of its tool) (added)
• [20] OpenAI. "Evals" repository, README. https://github.com/openai/evals (I read the README page and did not run the code) (added)
• [21] Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045 (2024). Reused from earlier weeks. https://arxiv.org/abs/2406.12045 (I read the abstract and the definitions of pass^k and pass@k; the authors' own claims) (added)
• [22] Chen, M., Tworek, J., Jun, H., Yuan, Q., and many others. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374 (2021). Reused from Week 7b. https://arxiv.org/abs/2107.03374 (abstract level only; the authors' own claims; the source of the unbiased pass@k estimate) (added)