TwinEdgeAI

↑ Course home

← Previous: Week 6b, Data for Agents  ·  Next: Week 7b, Evaluation Fundamentals & Benchmark Design →

Engineering AI Agents · Week 7a

Data Selection & Quality

Which data is worth keeping? Finding the most informative examples; filters and selection rules; tiny but targeted benchmarks; how people annotate; how to tell if a dataset is good; and how to build a training set from agent traces.

Author: Grok Bot Version: 1.0, October 29, 2026 Study time: about 15 hours

The slides were not availableThis lecture is on Monday, November 2, and I wrote this pack on Thursday, October 29, 2026. I checked the course's usual slide addresses once, lecture12.pdf and lecture13.pdf, and both returned "not found" (HTTP 404) [5]. The schedule page has no slides link for this lecture [4]. So I have not seen the slides. This pack is built from the schedule's description of the lecture, the two required readings, the additional reading, and sources I found myself. The real lecture may stress other things. Where I guess what the lecture covers, I say so. Check the slides when they appear.

CreditThis coursework is adapted from, and inspired by, Stanford CS329Z: Engineering AI Agents (Diyi Yang, Michael Ryan, John Yang), Week 7, Lecture "Data Selection & Quality" (Monday, November 2). The topic list and the readings come from the course's public schedule [4], which says that it is tentative. The schedule describes the lecture as: finding maximally informative data, filtering and selection strategies, tiny-but-targeted benchmarks, annotation practices, quality assessment, and building datasets from agent traces. The notes, lab, quiz and assignment are original writing, and the lab uses its own examples and code. This pack does not reproduce any slides. It is not an official Stanford document. Each source I added is marked "added" in the references.

1. Week overview and learning objectives

Week 6b asked which data an agent needs and where it comes from. This week asks a harder question: out of everything you could train or test on, which examples are worth keeping? The schedule lists six topics [4]. I read them as three jobs and three tools.

The three jobs are choosing data (informative, filtered, selected), checking data (annotation, quality assessment) and making data (a dataset built from agent traces). The tools are a score for each example, a set of rules that keep or drop it, and a small audited sample that tells you if the first two worked. The required readings fit this frame. SWE-smith is a large case of making and filtering data from agent runs [1]. "Who Validates the Validators?" is about checking, and about why the criteria for "good" move while you grade [2]. LIMA is the argument that a thousand well-chosen examples can be enough for one kind of goal [3].

The lab uses the standard library only. It does not call any model, any API or the network. Everything it prints comes from small made-up worlds: made-up pass probabilities, made-up points, a made-up linear model, made-up annotators, made-up model scores, and made-up traces. It shows how the methods work: uncertainty scores, farthest-first and MinHash selection, a gradient-similarity proxy, confident learning, agreement coefficients, item response theory, an EvalGen-style alignment score, and a trace pipeline with quality gates. It says nothing about how real data or real models behave. In several places the toy gives a weak or even opposite result to a paper, and I report that.

Learning objectives

After this week you should be able to:

1. Say what "good data" can mean (correct, informative, diverse, on target, allowed), and why a rule that raises one can hurt another. [2.1]
2. Score examples by uncertainty, committee disagreement and pass-rate variance, and say when each one fails. [2.2]
3. Compare quality filters, loss-based and gradient-based selection, and importance resampling, with one example from the literature each. [2.3]
4. Run farthest-first selection and MinHash de-duplication, and compute the chance that a pair becomes a candidate. [2.4]
5. Explain confident learning and a data map, and say why a model-based label-noise finder can only find what the model finds. [2.5]
6. Compute percent agreement, Cohen's kappa, Fleiss's multi-pi and Krippendorff's alpha on a small table, and explain the prevalence problem. [2.6]
7. Describe how the OpenAI team made SWE-bench Verified, and say what "take the most severe of three labels" trades away. [2.6]
8. Fit a one- or two-parameter item response model, choose a tiny benchmark by Fisher information, and measure rank agreement with Kendall's tau. [2.7]
9. Describe the SWE-smith pipeline, and design quality gates for a dataset built from agent traces, with an audit that has a known limit. [2.8]
10. Define coverage, false failure rate and alignment as EvalGen does, and explain criteria drift. [2.9]
11. Build and test a small selection toolkit and use it on made-up data. [Lab]

How to use this pack

Time	What to do
3.5 hours	Read the notes (section 2). Skim the figures first.
3.5 hours	Read the two required readings (SWE-smith and EvalGen) and skim LIMA with the reading guide questions. Skim the optional ones that match your project.
4.5 hours	Do the lab and at least five exercises.
1.5 hours	Discussion and the quiz.
2 hours	The graded assignment.

2. Lecture notes

The slides were not available (see the top of the page), so these notes follow the schedule's six topics [4] and the readings. The section titles are mine.

2.1 Why selection, and what "good data" means

Last week's question was where agent data comes from. Most pools of data are far bigger than what you can afford to train on, label, or run as a test. So you choose. The choice is not neutral. Every rule for choosing is a bet about what the model will learn, and the rule's blind spots become the model's blind spots.

The required reading on LIMA is the best-known argument that choosing can beat adding. The authors fine-tuned a 65B model on only 1,000 prompt and response pairs: 750 chosen from community forums such as Stack Exchange and wikiHow, and 250 that the authors wrote themselves [3]. They report that in a human study, LIMA's answers were equal to or preferred over GPT-4's in 43% of cases [3]. They state a "Superficial Alignment Hypothesis": a model learns its knowledge during pretraining, and alignment mostly teaches which format to use [3]. They then ran three ablations on a 7B model, which I read in the paper: more varied prompts helped, filtering for quality helped (a gap of 0.5 points on their 1 to 6 scale), and doubling the amount of data did not help [3]. Treat this as the authors' result on one goal, which is a helpful chat style. They say themselves that building such examples takes a lot of effort that is hard to scale, and that LIMA is less robust than product models [3].

Now compare SWE-smith, the other required reading. Its authors made 50 thousand task instances and found that performance went up with more training trajectories, and went up roughly with the logarithm of the number of repositories [1]. So "less is more" is not a law. It depends on the goal. A format can be taught with a few examples. A skill that needs many kinds of code probably needs many kinds of examples. That is my reading of the two papers, and neither paper says it in these words.

Five questions to ask about every example

I use five questions as a checklist. The framing is mine, not the schedule's or any paper's. Each question maps to methods in this week's notes.

1. Is it correct? Labels, tests and answers can be wrong (2.5, 2.6).
2. Is it informative? Would training or testing on it change anything (2.2)?
3. Is it different? Is it a copy, or close to one, of something already chosen (2.4)?
4. Is it on target? Does it teach the skill you care about (2.3, 2.7)?
5. Is it allowed? Consent, privacy, and not a leak from your test set (2.8)?

Some questions remove examples (correct, allowed). Others rank them (informative, different, on target). Rules interact. If you keep the most "informative" examples by a score, you may keep the noisiest ones, because a wrong label looks surprising to a model. If you keep only the best-rated, you may lose variety. The Deita authors and the QuRating authors both report that quality and diversity must be balanced [9] [10].

2.2 Finding maximally informative data

An example is informative if learning from it, or testing on it, tells you something you did not know. Active learning has asked this for decades: it tries to get the largest gain from the fewest labeled examples, and a recent survey covers its use with deep networks [16]. I read only that survey's abstract. The lab uses the standard textbook scores, which I wrote from general knowledge.

Scores from a model's output

• Entropy of the predicted class probabilities. High when the model is unsure among many classes.
• Least confidence: one minus the top probability.
• Margin: the top probability minus the second. A small margin means a close call.
• Committee disagreement: several models vote, and the score is the entropy of the votes (vote entropy).

Scores from training

• EL2N: the size of the error vector (the prediction minus the one-hot label), averaged over a few training runs, early in training. Paul and colleagues report that it can find important examples early, and that pruning half of CIFAR-10 by it slightly improved test accuracy [12]. That is their claim on vision data.
• Forgetting events: how often an example goes from correct to wrong during training. Toneva and colleagues report that some examples are forgotten often and some never, and that many can be dropped without hurting accuracy [13].
• Confidence and variability across epochs, the two axes of a data map (2.5) [15].

A score for agent tasks: the pass rate

For an agent task with a verifier you can run the agent k times and count passes. If the true pass probability is p, one attempt has variance p(1 − p). It is largest at p = 0.5 and zero at 0 and 1. A task that is always solved, or never solved, gives the same result every time, so its outcome carries no news about the agent. The "0 < passes < k" filter keeps tasks where the runs differ. Week 6b's synthetic-data filter was the same idea. It is a common heuristic, and I treat it as one.

Two cautions. First, the pass rate must be estimated, and with few attempts it is a poor estimate. In the lab's made-up pool of 300 tasks, ranking by the estimated variance from one attempt finds none of the truly most informative tasks better than chance: it recovers 16% of the true top 50, and a random pick of 50 from 300 would recover about 17% (my arithmetic). With 4, 16 and 64 attempts the overlap is 32%, 56% and 80%. Cost grows with every attempt, so the filter is cheap to state and expensive to do well.

Second, a high variance means the current agent's outcome varies. It does not mean the task teaches a lot, and it does not mean the task is a good test. A task whose tests are flaky, or whose description is vague, also has outcomes that vary. The SWE-smith authors looked at this directly. They rated difficulty with a model and trained on subsets of different difficulty. The expert's resolve rates fell from 58.6% to 41.0% to 17.0% across easy, medium and hard tasks. Yet student models trained on 500 trajectories from mixes of difficulty scored 12.4%, 10.8%, 13.6% and 12.2%, and the authors say they saw no strong link between difficulty and downstream performance [1]. Difficulty predicts whether the task gets solved. It did not predict how well the trajectory taught.

2.3 Filtering and selection strategies

Here are the main families, with one well-known example each. All numbers in this section are the authors' own claims, from the abstracts unless I say otherwise.

Filter with a rater

AlpaGasus asked ChatGPT to score each (instruction, input, response) tuple and kept the ones with a score of at least 4.5. From the 52,002 examples of Alpaca this kept 9,229 (I read this in the paper's text). The authors report that the model trained on the filtered set beat the one trained on all of it, and that training time for the 7B model fell from 80 to 14 minutes [8]. Deita scores complexity and quality with models, then picks by "score first, diversity aware": take examples in order of score, and add one only if it is not too close to those already chosen. The authors report that 6K samples gave results better than or on par with open models trained on over 10 times more data [9]. QuRating asked models for pairwise judgments of four qualities, trained a rater, and sampled with the ratings as logits, which kept diversity. The authors report that it is important to balance quality and diversity [10]. LIMA's curation was by hand [3].

Select with the training signal

EL2N, forgetting and data maps (2.2) are scores from training [12] [13] [15]. CRAIG goes further. It picks a weighted subset whose gradient is close to the full gradient, and proves a convergence result for convex problems. The authors report speed-ups of up to 6 times for logistic regression and 3 times for deep networks [11]. A benchmark of ten pruning metrics on ImageNet found that most high-performing metrics scale poorly, and that the best were costly and need labels [14].

Select for a target: LESS

LESS is for targeted tuning. You have a pool of instruction data and a few examples of the skill you want. LESS trains briefly on a random 5% of the pool for four epochs. It then computes each example's gradient (with LoRA), projects it down to 8,192 dimensions, and scores each pool example by the cosine similarity of its gradient with the gradients of the target examples. It keeps the top 5% [6]. The authors report that this 5% often does better than training on the full pool, and that the selected data transfers from small to large models [6]. Two details matter for the lab. First, the warmup run is there because the choice would otherwise be circular: the gradients depend on a model that was trained on the data you are choosing [6]. Second, the authors use cosine and not the dot product because the gradient norm of a sequence falls as its completion gets longer, so the dot product upweights short sequences, and they report that this makes results markedly worse [6].

Select to match a distribution: DSIR

Data Selection with Importance Resampling (DSIR) maps each document to hashed n-gram counts, fits one distribution to the target and one to the raw pool, weights each document by the ratio, and samples without replacement by those weights. The authors report that this selected 100M documents from the Pile in 4.5 hours, and that a measure they call KL reduction correlated with downstream accuracy (r = 0.82 across 8 methods) [7]. The lab's version is a toy of this idea.

Remove duplicates

Lee and colleagues found many near-duplicate examples and long repeated passages in standard language model datasets. They report that de-duplicated training gave models that repeat training text about ten times less often, and that train and test overlap affected over 4% of the validation set of some datasets [18]. Section 2.4 covers the method the lab uses.

Family	Needs	Tends to favor	Known or likely failure (my reading)
Rater filter [8] [9]	A rater model and a threshold	Examples the rater likes	Rater bias becomes data bias. Add diversity, or it narrows.
Training signal [12] [13]	A model trained a bit	Hard or surprising examples	Picks up label noise. Metrics may scale poorly [14].
Gradient similarity [6]	A warmup model, a few target examples	Examples that push the model like the target does	Costly. Dot product favors large-norm examples. Needs good target examples.
Importance resampling [7]	Target text, a feature map	Examples that look like the target	Matches surface features only.
Pass-rate filter	Several runs per task	Tasks the agent sometimes solves	Noisy with few runs. Weak link to training value [1].
De-duplication [18]	A similarity measure	Variety	Order matters when combined with other gates (lab).

The lab's gradient-similarity toy has three made-up tasks that share one linear model. Task A is the target, task B is unrelated, and task C partly contradicts A. In the demo, training on the 20% chosen by gradient cosine scored 0.695 on A, while a random 20% scored 0.528 and the full pool scored 0.540. The full pool does badly because C pulls the model away. The dot-product choice scored 0.645, and it took 24.7 large-norm examples on average against 18.1 for cosine and 17.7 for random (10 seeds). That matches the paper's reason for cosine in direction, but the toy's accuracy gap between cosine and dot product is small, and I would not lean on it. I made the toy, and I chose its numbers.

2.4 Diversity, duplicates and targets

Ranking by a score picks the top of the list. The top of a list is often one cluster, repeated. Diversity methods fix that by spreading the picks.

Farthest-first (k-center)

Start with one point. Then keep adding the point that is farthest from every point already chosen. After k picks, the covering radius is the largest distance from any point in the pool to its nearest pick. This greedy rule is a classic method for the k-center problem. I recall that its radius is within a factor of two of the best possible, and I did not check a source for that. Farthest-first has one known weakness: it loves outliers, because stray points are far from everything. That is good for coverage and bad if the strays are errors.

In Figure 3 the covering radius is 2.02 for farthest-first and 3.11 for the shown random draw, which has the mean radius of 200 draws. In the lab demo, with 10 picks from 150 points in one big clump plus four small far clumps of 10, farthest-first gets a radius of 2.07 and random picks average 7.45 over 20 draws. Random picks usually miss the small clumps. Those numbers depend on how I built the points. A fairer reading is: when the data has small, far-away groups, random sampling underserves them.

Near-duplicate removal with MinHash

Two texts are near-duplicates if their word sets overlap a lot. The Jaccard similarity of two sets is the size of their intersection divided by the size of their union. Comparing every pair is too slow at scale. MinHash gives each text a short signature so that, for each signature position, the chance that two texts agree equals their Jaccard similarity. Banding then cuts the signature into b bands of r rows. Two texts become a candidate pair if any band matches in all its rows. The chance is 1 − (1 − s^r)^b for Jaccard s. The lab checks each candidate with the exact Jaccard value, so banding only has to be a cheap first step. This is the standard textbook method, and I wrote it from general knowledge. Lee and colleagues built two de-duplication tools for real corpora. I read only their abstract, so I did not check which methods they use [18].

The lab uses 64 signature values in 16 bands of 4 rows. Then a pair with Jaccard 0.3 is a candidate with probability 0.12, one at 0.5 with 0.64, one at 0.6 with 0.89, and one at 0.8 with 0.9998 (my arithmetic, section 2.10). The S-shaped curve is the point: pairs well above the threshold are found, pairs well below are mostly skipped, and pairs near it are found by chance.

Importance resampling toward a target

DSIR [7] weights each pool document by how much more likely it is under the target's feature distribution than under the pool's. The lab's toy uses hashed word and word-pair counts, a smoothed bucket distribution for each side, and the log ratio as the weight. It then draws without replacement with the Gumbel top-k trick (add Gumbel noise to each log weight and keep the top k). With three made-up topics that use different words, 60 picks aimed at "code" were all code (random would give about one third). That is easy by design, because the topics share almost no words. The paper's claim is about real text where topics overlap. The toy only shows the mechanics. Sampling, and not just taking the top weights, keeps some variety, which is also the QuRating authors' point about using ratings as logits [10].

2.5 Label quality and data maps

Labels, tests and answers are made by people and by models, so some are wrong. Finding them is a selection problem too.

Confident learning

Confident learning (CL) finds label errors from a model's predicted probabilities [17]. I read the paper's abstract, its introduction and its definition of the confident joint. The steps are:

1. Get out-of-sample predicted probabilities, for example by cross-validation, so that the model has not memorized each example's own label.
2. For each class j, set a threshold t_j equal to the average predicted probability of class j among the examples labeled j. The paper calls this the expected self-confidence of the class [17].
3. For each example, find the classes whose probability is at least their threshold, and take the one with the highest probability. If that class differs from the given label, count the example in the off-diagonal cell of the confident joint, a table of given labels against estimated true labels.
4. Prune the flagged examples, or rank them by how low the probability of their own label is.

The class thresholds make the method less sensitive to a model that is over-confident about one class, or to class imbalance [17]. The authors report finding label errors in MNIST, which was presumed to be error-free, and they estimate that 645 "missile" images in ImageNet are labeled with the parent class "projectile" [17].

The lab's toy has three Gaussian classes with unequal spread, and it flips a share of labels. A centroid model with cross-validation gives the probabilities. Averaged over 8 seeds, an average of 31.4 labels were flipped. CL flagged 36.3, with precision 0.79 and recall 0.92. A baseline that flags the same number of lowest self-confidence examples had precision 0.77 and recall 0.89. So CL was a little better in the toy, and the gap is small. I had to choose unequal spreads to see any gap at all, and with equal spreads there was none. I report this as a weak result. It says nothing about CL on real data.

Data maps

A data map plots each training example by two numbers from one training run: the confidence (the average probability the model gives to the example's label across epochs) and the variability (how much that probability changes) [15]. The authors of the data maps paper report three regions: "ambiguous" examples with high variability, which they say help out-of-distribution generalization most; "easy to learn" examples, which are the most common; and "hard to learn" examples, which often correspond to labeling errors [15]. In the lab's toy with flipped labels, the share of flipped labels was 0.50 in the hard region, 0.08 in the ambiguous region and 0.00 in the easy region, against a base rate of 0.09 (8 seeds).

2.6 Annotation practices

How SWE-bench Verified was made

OpenAI's post on SWE-bench Verified is the clearest public description I found of a large annotation campaign for an agent benchmark. I read the whole page [23]. It is vendor-authored, and its author list includes John Yang, a course instructor. It describes this process:

• The team worked with 93 software developers experienced in Python, who screened 1,699 random samples of the SWE-bench test set [23].
• Each sample got labels on two criteria: whether the issue text was underspecified, and whether the FAIL_TO_PASS tests would reject valid solutions. Each label is on four ordinal levels, 0 to 3. Levels 0 and 1 are minor, and 2 and 3 are severe [23]. The post says the four levels were chosen to capture more detail than a yes or no.
• Annotators also estimated how long a fix would take, and could flag other major issues in free text, for example tests that are easy to game [23].
• The team hand-labeled 50 samples with high confidence, used them as an onboarding test, and gave feedback to each annotator during onboarding [23].
• Each sample was labeled three times by different annotators. The team took the highest severity among the three. The post says that this is the same as dropping a sample if any one of the three raised an issue, and that it gives more false positives but more confidence in the kept samples [23].
• The post reports that 38.3% of samples were flagged as underspecified, 61.1% for tests that may be unfair, and 68.3% were filtered out overall. It says the filtering is probably overzealous. Verified has 500 samples, chosen to include as many of the harder 1 to 4 hour and over 4 hour samples as possible, with the rest sampled at random [23].

I looked for an agreement coefficient between the annotators in the post and did not see one. That does not mean none was computed. LIMA's human study reports a tie-discounted percent agreement for 50 shared examples: 82% between two crowd workers, 81% between crowd and author, and 78% between the two authors [3]. Those are raw agreement numbers, with no correction for chance.

Agreement coefficients

Percent agreement is easy but misleading when one label is common. Chance-corrected coefficients compare observed agreement A_o with the agreement expected by chance A_e: (A_o − A_e) / (1 − A_e). They differ in how they model chance. I read the survey by Artstein and Poesio in full for this section [22].

• Scott's pi: chance comes from the pooled label shares of all coders.
• Cohen's kappa: chance comes from each coder's own label shares. The two differ most when coders use the labels at very different rates [22].
• Fleiss's multi-pi is the generalization of pi to more than two coders. Artstein and Poesio note that a coefficient often called "kappa" in the literature is really this generalization of pi and not of kappa [22]. I call it multi-pi for that reason.
• Krippendorff's alpha is built on observed and expected disagreement. It handles any number of coders, missing labels, and a distance between labels. With all disagreements equal it is nearly the same as multi-pi, with a small-sample correction [22].
• Weighted versions let a near miss count less than a far miss, which suits ordinal labels such as severity 0 to 3.

The lab reproduces the two-coder example in Artstein and Poesio: 100 utterances, three labels, agreement 0.88, pi 0.7995, kappa 0.8013, weighted kappa 0.8163, and alpha 0.8156 with their weights [22]. The tests check these against the numbers printed in the paper.

The prevalence problem [22] is that when most items carry one label, agreement coefficients can be low or even negative while percent agreement is high. In the lab's example, 100 items, 90 are "yes" for both coders, and in the other 10 the coders split into 5 with one "no" and 5 with the other. Agreement is 0.90, kappa is −0.053 and alpha is −0.047. There is no rule that fixes this. You must look at the table, not only at one number. The survey also argues that there is no single threshold for "good enough". It is a decision for each task.

Keep rules after annotation

The SWE-bench Verified rule (keep only if no annotator flags it) trades recall for precision. The lab's simulation has 2,000 items, three annotators each reporting the true severity with probability 0.7 and otherwise one step off, and two criteria. An item is "truly fine" if both true severities are below 2. About 48.5% of items are fine. With the max-of-3 rule, 35.1% of items are kept, nearly all of them truly fine (precision 0.999), but only 72.3% of the fine items are kept (recall). With a median-of-3 rule, 47.7% are kept, with precision 0.970 and recall 0.954. A single annotator keeps 46.9% with precision 0.925 and recall 0.895. The right rule depends on cost. If a bad item in a benchmark is very costly and items are plentiful, max is a good choice. If items are scarce, it is expensive. The toy's error model is mine.

Weak supervision

When people are too slow, one can write labeling functions, small programs or heuristics that label some examples, and combine their noisy and conflicting votes with a model. Data programming proposed this, and the Snorkel system built it [19] [20]. I read both abstracts. The Snorkel authors report that in a user study, experts built models 2.8 times faster, with an average 45.5% better performance, than with seven hours of hand labeling [20]. The lab does not implement weak supervision, but the EvalGen assertions in 2.9 are labeling functions of a kind.

2.7 Tiny but targeted benchmarks

A benchmark is data you select for a different purpose: to rank models, or to see where they fail, at low cost. Full evaluations are expensive, and for agents they are slower and costlier still (every item is a multi-step run). A tiny benchmark is a small subset chosen so that it gives about the same answer as the full one. This lecture touches the idea, and the next lecture's additional reading is on it (the schedule lists tinyBenchmarks for November 4 [4]), so this section is short.

What the papers report

• tinyBenchmarks. The authors report that for several benchmarks, the best strategies estimate a model's score to within 2% using 100 examples or fewer per scenario. For MMLU that is 100 of 14,000 examples. They found IRT-based methods performed consistently well, and they released tiny versions of several benchmarks [25]. I read the abstract and the sections on key findings and limitations. They warn that errors are larger for models whose pattern of right and wrong answers differs from the models used to fit the method, for example models that fail easy questions and pass hard ones, and they advise refreshing the chosen examples over time [25].
• Anchor points. The authors group items by how model confidence moves together, and report that 1 to 30 "anchor points" rank 87 model and prompt pairs better than random sampling [26]. I read the abstract.
• Efficient benchmarking. The authors report that the leader of a benchmark can change when you remove one low-ranked model, and that a correct ranking can be obtained from only a fraction of the examples [27]. I read the abstract.

Item response theory in one paragraph

Item response theory (IRT) models the chance that model m answers item i correctly as p = σ(α_i θ_m − β_i), where σ is the logistic function. The model has an ability θ. The item has a difficulty β and a discrimination α. The one-parameter (1PL) version fixes α = 1 for all items. The two-parameter (2PL) version fits α too. This is the form I used in the lab. The Fisher information of an item at ability θ is α² p(1 − p). It is largest where p = 0.5, that is at θ = β / α, and it is bigger for items with a larger α. So an item tells you most about models whose ability is near its difficulty. Choose items so that this information is high across the range of abilities you care about, and you get a tiny benchmark.

The lab's tiny benchmark study

The lab makes a made-up world of 90 models and 200 items (60 models to fit, 30 held out), with true abilities and difficulties that I drew. It fits IRT on the 60 models by plain gradient steps with weak priors, chooses 24 items, and then predicts each held-out model's full 200-item accuracy from its answers on the 24. I compare six recipes by the Kendall tau between the predicted accuracies and the full accuracies, and by the mean absolute error of the predicted accuracy. The numbers below average 20 such worlds, in one ability dimension.

Recipe	Kendall tau	Mean abs error
24 random items, plain mean	0.761	0.072
24 items stratified by difficulty, plain mean	0.776	0.061
24 anchor points, weighted mean (my simplified version)	0.626	0.102
24 random items, 2PL ability	0.735	0.062
24 IRT-chosen items, 2PL ability	0.793	0.051
24 IRT-chosen items, 1PL ability	0.778	0.056

How to read this honestly. The IRT-chosen items with a 2PL ability are best, by 0.03 in tau over random items with the plain mean, and the standard deviation across worlds is 0.04 to 0.07. That is a modest gain that I would not call decisive. Stratifying by difficulty gets most of it for free. My "anchor points" recipe is worst, but it is a simplification of the paper's method (farthest-first on the columns of the response table, weighted by cluster size), so it says nothing about the real method [26]. A tau of about 0.79 is not close to 1, so with 24 items the ranking of 30 models is still often wrong in the middle. In a second run, in which the models' abilities have two correlated dimensions but the IRT fit still has one, the IRT-chosen 2PL recipe got a tau of 0.822, which is higher than in the one-dimension run. I did not expect that, and I did not investigate it, so I do not trust the comparison between the two settings.

Targeted, and what can go wrong

"Targeted" means you choose items for a purpose: to separate the models you will actually compare, or to cover a skill. Selection can fail in three ways. The held-out models can differ from the fit models, which the tinyBenchmarks authors flag [25]. The items can become known, and then models are tuned to them. And a ranking is only as good as the noise in each item: the lesson of section 2.6 on bad items applies here too, because a benchmark made of a few items is hurt badly by one wrong test.

2.8 Building datasets from agent traces

This is the sixth topic on the schedule, and SWE-smith, the first required reading, is the case study [1]. Here is the pipeline the authors describe. I read the paper's text, with the most attention to sections 2 to 4 and the appendix on difficulty. All numbers are theirs.

1. Make an environment first. For each repository, SWE-agent tries for at most 100 steps to install the code and run the tests. A person checks the instructions, a repository is kept if more than 80% of its tests pass, and a Docker image is built. The authors choose popular Python packages with at least 1,000 stars, and leave out the 12 repositories of the SWE-bench test set [1].
2. Make candidate tasks. Four strategies: an LM edits a function to break it ("LM Modify"), an LM rewrites a function from its header and docstring ("LM Rewrite"), random changes to the syntax tree ("Procedural"), and undoing a past pull request ("PR Mirror"). A fifth, "Combine", merges several bugs [1].
3. Validate by running. Apply each candidate and run the tests. Keep it only if it breaks at least one test that passed before. Throw it away if the tests take more than two minutes [1]. The yield differs by strategy: 96.9% for Combine, 56.0% for LM Modify, 40.2% for Procedural, 35.0% for LM Rewrite and 33.8% for PR Mirror [1].
4. Write the issue text. An LM writes an issue from the bug patch, one failing test and the test output [1].
5. Get expert trajectories and filter. They ran SWE-agent with Claude 3.7 Sonnet, with at most 75 steps and a $2 cost limit, on 8,686 task instances, and kept the resolved runs. The resolve rate was 36% of 17,906 attempts. They then capped each task instance at 3 trajectories, because (they say, citing the SWE-Gym authors) easier trajectories that are solved repeatedly hurt the model. That left 5,016 trajectories for fine-tuning a 32B model [1]. I read only SWE-Gym's abstract, and it does not say this [28].
6. Result and cost. The authors report 40.2% on SWE-bench Verified, single attempt, for the 32B model fine-tuned on those trajectories, the best open-weight result they know of at the time of writing. They report $1,360 to make the 50 thousand tasks and about 20 hours of one author's time [1]. These are their claims, from authors that include a course instructor.

The ablations are the best part for data selection. PR Mirror tasks gave the best students (9.2% resolved at equal size on a 7B model), Procedural and LM Rewrite were close (8.6% and 8.8%), and LM Modify was clearly worse (5.7%). LM-written issue text was about as good as the original issue (7.7% against 7.8%). Fixed issue templates gave 31% fewer distinct actions in the trajectories. Putting the failing test in the issue made the student skip writing a reproduction script, 127 of 500 runs against 379 [1]. The last result shows that what you put in the data changes the behavior, not only how much of it you keep. And as noted in 2.2, difficulty did not predict training value [1].

The authors also found that their fine-tuned model often gets stuck repeating an action. More than 25% of its runs had a repeat of at least length 10, against under 4% for Claude 3.7 Sonnet, and a repeat of length 10 went with an 89% failure rate [1]. That is a reason to filter loops from traces before training, and the lab does.

Gates for a trace dataset, and what they cannot see

The lab's pipeline has ten gates. The list is my own design. The notes say which ones have a source behind them.

Gate	What it does	Basis
consent	Drop traces whose owner did not allow training.	Week 6b's consent flags. My choice here.
resolved	Keep only traces whose verifier passed.	Rejection sampling, as in SWE-smith [1].
complete	Drop truncated traces that do not end in a submit.	My own rule.
loops	Drop traces with a long run of the same action.	Repetition analysis in SWE-smith [1]. The cut-off is mine.
tamper	Drop traces that edit the tests or exit early.	My own rule. The Verified post flags tests that are "easily gamed" as an issue [23].
pii	Drop traces with an email or a phone number in the prompt.	My own regexes. Not a compliance tool.
leak	Drop traces whose prompt is a near copy of an eval prompt.	Train and test overlap is real [18].
dedupe tasks	Keep one prompt of each near-duplicate group.	[18]. The order is my choice.
not trivial	Drop tasks that the agent nearly always solves.	My own rule, in the spirit of 2.2.
cap per task	Keep at most 3 traces of one task.	SWE-smith's cap of 3 [1].

In the demo the pool shrinks from 1,008 to 201 traces. The two biggest drops are "resolved" (402 traces) and "not trivial" (94). Because I planted the defects, I can count them. All 50 traces that leak an eval prompt, 62 with loops, 32 with PII, 51 with tampering and 34 truncated traces were removed. Without the "complete" and "tamper" gates, 24 of 51 tampered and 11 of 34 truncated traces survive the others. Of 59 traces that are near copies of another task, 10 stayed (the original was removed by an earlier gate, so the copy is the only one left). Of 13 traces with weak tests (tests that pass whatever the agent does), 3 stayed, and no gate looks for them. The gates catch what they were written for, and nothing else. That is the main point of the exercise.

The audit is the second line of defense. I read 40 kept traces at random, and none had weak tests. The 95% Wilson interval for 0 of 40 is 0 to 0.088. The true rate in the kept set is 3 of 201, or 1.5%, which an audit of 40 cannot see: the chance of seeing no case in 40 draws is 0.985⁴⁰ = 0.55, and you would need about 198 draws to have a 95% chance of seeing at least one (my arithmetic, 2.10). Audits find common defects. They do not certify that rare ones are absent.

One more limit. SWE-smith's validity check shows that a task breaks tests. It does not show that the issue text is enough to find the fix, or that the tests only accept correct fixes. Those are exactly the two criteria that the SWE-bench Verified annotators rated, and they flagged 38.3% and 61.1% of the original samples on them [23]. That is my reading of the two works side by side, and I have not seen the SWE-smith authors discuss it in these terms.

A data card

The lab ends by writing a small data card in JSON: how many traces and tasks, the repositories, the gates in order, and the known gaps (weak tests are not detected; gates only catch what they were written for; everything here is made up). It follows the idea of datasheets for datasets, which propose to document how a dataset was made and what it should be used for [21]. I read the datasheets paper's abstract only. The lab's card is my own simplification.

2.9 Validating the validators

Every filter above is a validator: a rule, a score or a model that says "keep" or "drop". The second required reading asks who checks the validators [2]. Its authors (Shankar and colleagues, UIST 2024) argue that an LLM-written evaluator inherits the problems of the LLM, so a person must validate it. Their tool, EvalGen, works like this [2]:

1. An LLM suggests criteria in natural language. The user edits, adds or removes them.
2. An LLM writes several candidate assertions for each criterion, as Python functions or LLM grader prompts.
3. The user grades a sample of outputs as good or bad.
4. For each criterion, EvalGen picks the candidate with the highest alignment with the grades, subject to a cap on its false failure rate.

Definitions. Let an assertion fail an output if it says the output is bad. For a set of assertions, coverage is the share of truly bad outputs that at least one assertion fails. The false failure rate (FFR) is the share of truly good outputs that at least one assertion fails. Alignment is the harmonic mean of coverage and 1 − FFR [2]. The paper's equations and its prose disagree about which grade value means "bad". The lab follows the prose and treats 1 as good. That is my choice.

The lab recomputes the paper's Table 1 from its coverage and FFR values. The medical pipeline (coverage 0.33, FFR 0.10) gives 48.29%. The product pipeline gives 66.46% for EvalGen (0.73, 0.39) and 54.35% for the SPADE baseline (0.49, 0.39) [2]. For example, 2 × 0.73 × 0.61 / (0.73 + 0.61) = 0.6646 (my arithmetic). The authors report that with user input at the criteria stage, EvalGen gets the same or better alignment with fewer assertions [2]. The two pipelines are small (84 and 100 outputs) and two of the authors graded all outputs [2].

Which outputs to grade is a selection problem

Users will not grade everything, so the tool must choose. The paper scores each output by the selectivity of the assertions that fail it. It compares four policies for picking outputs to grade: random, highest score, lowest score, and alternating. In the authors' offline test with 16 graded outputs and 10 trials, random sampling had a large variance in alignment, and the other policies gave higher alignment. The authors say they do not claim the best policy, and they used alternating in their study [2]. The lab's toy gives the opposite order for the mean. Over 20 worlds with 16 grades each, random got a mean alignment of 0.878, lowest 0.712, alternating 0.574 and highest 0.551. In the toy, the other policies pick extreme examples, which tilts the sample away from the typical output. The toy's spread is across worlds and the paper's is across trials on one pipeline, so the two spreads are not comparable. I report this as a case where my toy does not reproduce the paper, and I do not claim the paper is wrong. My outputs, my score and my graders differ from theirs.

Criteria drift

The study with nine industry practitioners found what the authors call criteria drift: people need criteria to grade outputs, but grading outputs helps them define the criteria [2]. The authors saw two kinds. Users added new criteria when they saw a new type of bad output. And users reinterpreted existing criteria as they graded more. In the paper's example, two participants first graded an output as bad if any extracted entity was not a proper noun, and then changed the criterion to "most entities" once they saw outputs with a mixed number of proper nouns. One participant said, "I think it's hard to know until you see it" [2]. The authors conclude that it is not possible to fix evaluation criteria entirely before people look at outputs [2].

The lab simulates a small version. The made-up outputs have the share of extracted entities that are proper nouns, a word count and a flag for a copied hashtag. A grader first uses a strict rule ("every entity is a proper noun, and at most 60 words"). After 20 graded outputs, the grader switches to a relaxed rule ("at least half of the entities are proper nouns, and at most 60 words"). EvalGen-style selection is run on the early grades and again on the later ones. I made up the rules and the outputs.

Seed	Early set vs strict grades	Early set vs relaxed grades	Re-selected set vs relaxed grades
0	0.777	0.627	0.744
1	1.000	0.672	0.893
2	0.656	0.388	0.863

The set picked early is well aligned with the early grades and worse against the new grades, and re-selection recovers part of the loss. The numbers are alignment scores. This is a made-up picture of the effect and does not measure it. Practical rules that follow from the paper: grade in rounds, keep each grade with its date and the criteria text in force when it was given, and re-grade a sample after the criteria change [2].

2.10 Worked arithmetic

Everything in this section is my own arithmetic.

A. The difficulty filter with k = 8 attempts

A task is kept if it has at least one pass and at least one fail. The chance is 1 − p⁸ − (1 − p)⁸.

True pass probability p	All 8 pass	All 8 fail	Kept
0.50	0.004	0.004	0.992
0.90 (or 0.10)	0.430	0.000	0.570
0.05 (hard)	0.000	0.663	0.337
0.01 (very hard)	0.000	0.923	0.077

The filter prefers middle-difficulty tasks and removes most very hard ones, which may be the valuable ones if they are solvable.

B. The banding curve, 16 bands of 4 rows

Jaccard s	s4 (one band)	1 − (1 − s4)16 (candidate)
0.3	0.0081	0.122
0.5	0.0625	0.644
0.6	0.1296	0.891
0.8	0.4096	0.9998

C. Agreement by hand

The prevalence example has 100 items. Observed agreement is 0.90. Each coder says "yes" for 95 items, so the chance agreement is 0.95² + 0.05² = 0.905. Kappa = (0.90 − 0.905) / (1 − 0.905) = −0.053. For the Artstein and Poesio example, A_o = 0.88 and pi's chance term is 0.4014, so pi = (0.88 − 0.4014) / (1 − 0.4014) = 0.7995, which is the paper's value [22].

D. Fisher information

Information is α² p(1 − p). At its peak, p = 0.5, an item with α = 1 gives 0.25. An item with α = 2.2 gives 2.2² × 0.25 = 1.21, which is the peak of the blue curve, as Figure 6 shows. An item with α = 1 and β = 1 peaks at θ = β / α = 1.

E. Kendall's tau on four models

Say the full benchmark ranks models A, B, C, D from best to worst, and a tiny benchmark ranks them A, C, B, D. Of the 6 pairs, 5 are in the same order and 1 (B and C) is swapped. Tau = (5 − 1) / 6 = 0.667. The lab uses tau-b, which also handles ties.

F. What an audit of 40 can see

The Wilson interval for 0 successes in n = 40 at 95% has the upper end z² / (n + z²) = 3.84 / 43.84 = 0.088. If 1.5% of kept traces are defective, the chance of seeing none in 40 random draws is 0.985⁴⁰ = 0.55. To see at least one with probability 0.95 you need n with 0.985ⁿ = 0.05, which gives n = ln 0.05 / ln 0.985 = 198.

G. EvalGen alignment

Alignment = 2 × coverage × (1 − FFR) / (coverage + (1 − FFR)). With coverage 0.73 and FFR 0.39: 2 × 0.73 × 0.61 / 1.34 = 0.665.

2.11 Week 7a in one page

1. Selection is a bet. A rule picks what it can see. Ask five questions: correct, informative, different, on target, allowed (my framing). Keep a test set the selection never saw [3] [9].
2. Informative is not the same as useful. Uncertainty, disagreement and pass-rate variance find surprising examples, including bad labels. Difficulty did not predict training value in SWE-smith [1] [15].
3. Many ways to select. Rater filters, training signals, gradient similarity (LESS), importance resampling (DSIR), duplicate removal. Each has a blind spot [6] [7] [8] [12] [18].
4. Spread your picks. Farthest-first covers a space. MinHash finds near copies. Order of steps changes the result.
5. Labels are noisy. Confident learning and data maps flag suspects, and a model-based finder misses systematic errors [15] [17].
6. Annotation is a process. Rubric, onboarding with gold labels, several annotators, an ensemble rule. SWE-bench Verified took the most severe of three labels [23].
7. One agreement number is not enough. Look at the table. Prevalence can make kappa low at 90% agreement [22].
8. Tiny benchmarks can work, with limits. IRT picks items by information. Gains were modest in the toy, and the paper warns about shifts [25].
9. A trace dataset needs gates and an audit. Gates catch what they were written for. A small audit cannot see rare defects [1] [23].
10. Validate the validators. Criteria drift: you learn what you mean by grading. Grade in rounds and date the grades [2].

3. Reading guide

The course lists two required readings and one additional reading for this week [4]. For each you get: why it matters, the key ideas, and questions to think about as you read. Read the questions first. The optional table at the end points to the sources that I used for the notes. For most of those I read only the abstract, and the references say which. Two of the three readings come from authors who built what they describe: SWE-smith's first author teaches the course, and LIMA's authors report their own results. EvalGen's authors report a user study of nine people, which is small.

Optional readings

Source	Why it is worth your time
LESS [6]; DSIR [7]	Two ways to select for a target: gradient similarity and importance resampling. Read how each handles the target examples.
AlpaGasus [8]; Deita [9]; QuRating [10]	Rater-based filters, and how each handles diversity. The authors report their own gains.
CRAIG [11]; EL2N [12]; forgetting events [13]; data pruning [14]	Selection by training signals, with the caveat in [14] that metrics scale poorly.
Data maps [15]; confident learning [17]	Finding label errors from training dynamics and from probabilities.
Deep active learning survey [16]	The label-efficient learning view of "informative".
Lee et al. on de-duplication [18]	Duplicates and train-test overlap in real corpora.
Data programming [19]; Snorkel [20]	Labeling functions and weak supervision.
Datasheets for datasets [21]	A documentation template for datasets.
Artstein and Poesio [22]	The clearest survey of agreement coefficients that I found. I read all of it.
OpenAI on SWE-bench Verified [23] and the later post [24]	A large annotation campaign for a benchmark, and a later vendor view. Vendor-authored.
tinyBenchmarks [25]; anchor points [26]; efficient benchmarking [27]	Tiny benchmarks. tinyBenchmarks is also next lecture's additional reading.
SWE-Gym [28]; Shankar's flywheel post [29]	An earlier SWE environment for training, and the loop that uses the labels from this week.

4. Hands-on lab: a data selection and quality toolkit

Time: about 4.5 hours. Goal: build and test a small toolkit for choosing and checking data: informativeness scores; farthest-first selection, MinHash de-duplication and importance resampling; a gradient-similarity proxy for LESS; confident-learning label-noise detection and a data map; agreement coefficients; item response theory for a tiny benchmark; an EvalGen-style alignment score with a criteria-drift demo; and a trace-to-dataset pipeline with quality gates and an audit. You do not need an API key, a model or any package. The lab never touches the network. Every data set in it is made up by a seeded random generator.

What you will learn

• How to score informativeness by entropy, margin, vote entropy and pass-rate variance, and how the estimate improves with more attempts.
• How farthest-first, MinHash with banding, and importance resampling with a Gumbel top-k draw each work.
• How a gradient-similarity score picks data for a target task, and why cosine and dot product differ.
• How out-of-sample probabilities, class thresholds and a confident joint flag label errors, and how a data map splits examples into regions.
• How to compute percent agreement, pi, kappa, weighted kappa, multi-pi and alpha, and how prevalence breaks them.
• How to fit a 1PL and a 2PL model, pick items by Fisher information, and measure rank agreement with Kendall's tau-b.
• How coverage, false failure rate and alignment are computed, and how criteria drift changes the best assertion set.
• How a pipeline of gates turns traces into a dataset, how planted defects show what each gate catches, and what an audit can see.

Map of the code

Part	Main names in the code	What it does, and what it stands for
1	entropy, margin, vote_entropy, pass_variance, sometimes_passes, rank_ids, cap_per_group, make_task_pool, informativeness_demo	Informativeness scores and a made-up task pool with pass probabilities (2.2) [16] [1].
2	k_center_greedy, covering_radius, score_first_diverse, minhash, lsh_candidates, near_duplicates, duplicate_groups, dsir_log_weights, gumbel_top_k	Diversity, near-duplicate search and importance resampling (2.3, 2.4) [7] [9] [18].
3	make_less_world, example_grad, train_logistic, make_projection, less_scores, less_demo	A gradient-similarity selection proxy on a toy logistic model (2.3) [6].
4	make_blobs, flip_labels, cross_val_probs, class_thresholds, confident_joint, find_label_issues, training_dynamics, data_map_regions	Confident learning and a data map (2.5) [15] [17].
5	percent_agreement, cohen_kappa, scott_pi, weighted_kappa, fleiss_multi_pi, krippendorff_alpha, ensemble_max, simulate_annotation, prevalence_demo	Agreement coefficients and keep rules (2.6) [22] [23].
6	make_irt_world, fit_irt, fit_theta, estimate_pirt, fisher_information, select_irt_grid, select_anchors, kendall_tau_b, tiny_benchmark_study	IRT and tiny benchmarks (2.7) [25] [26].
7	coverage, false_failure_rate, alignment, selectivity, failure_score, grading_order, select_per_criterion, criteria_drift_demo, sampling_study, paper_table_alignment	EvalGen-style alignment, grading policies and criteria drift (2.9) [2].
8	Trace, make_trace_pool, gate_* functions, run_pipeline, defect_report, wilson_interval, audit_sample, dataset_card	A trace-to-dataset pipeline with gates, planted defects, an audit and a data card (2.8) [1] [21].
9	main	The nine-section demo.

Setup

1. You need Python 3.10 or later. There is nothing to install. I ran it on Python 3.13 only.
2. Save the code below as selection_quality_toolkit.py. Or download it: selection_quality_toolkit.py.
3. Save the tests as test_selection_quality_toolkit.py in the same folder. Or download: test_selection_quality_toolkit.py. The test code is not printed on this page, to keep the page shorter. It is in the download, and the names of all the tests are listed after the exercises.
4. Run python selection_quality_toolkit.py for the nine-section demo, and python test_selection_quality_toolkit.py for the tests. Both are deterministic. The demo takes about 30 seconds and the tests about 10 seconds on my machine.

The code: selection_quality_toolkit.py

"""Week 7a toolkit: data selection and quality, standard library only.

Nothing here calls a model, an API or the network. Every world is made up: the tasks, the "models",
the annotators, the traces. The code shows how the pieces work, not how real data behaves.

Parts:  1 informativeness   2 diversity and dedupe   3 gradient similarity (LESS-like) proxy
        4 label noise and data maps   5 agreement   6 IRT and tiny benchmarks
        7 EvalGen-style alignment   8 trace-to-dataset pipeline   9 demo
"""
import hashlib
import json
import math
import random
import re
from dataclasses import dataclass, field

# ---------------------------------------------------------------- shared helpers
_WORD = re.compile(r"[a-z0-9_]+")


def words(text):
    return _WORD.findall(str(text).lower())


def stable_hash(s, bits=64):
    h = hashlib.blake2b(str(s).encode("utf-8"), digest_size=8).hexdigest()
    return int(h[: bits // 4], 16)


def mean(xs):
    xs = list(xs)
    return sum(xs) / len(xs) if xs else 0.0


def stdev(xs):
    xs = list(xs)
    if not xs:
        return 0.0
    m = mean(xs)
    return math.sqrt(sum((x - m) ** 2 for x in xs) / len(xs))


def sigmoid(x):
    if x >= 0:
        return 1.0 / (1.0 + math.exp(-x))
    e = math.exp(x)
    return e / (1.0 + e)


def logit(p, eps=1e-6):
    p = min(max(p, eps), 1 - eps)
    return math.log(p / (1 - p))


def dot(a, b):
    return sum(x * y for x, y in zip(a, b))


def norm(a):
    return math.sqrt(dot(a, a))


def cosine(a, b):
    na, nb = norm(a), norm(b)
    return 0.0 if na == 0 or nb == 0 else dot(a, b) / (na * nb)


def euclid(a, b):
    return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))


def overlap(a, b):
    """Share of the first selection that the second selection also has."""
    a, b = set(a), set(b)
    return len(a & b) / len(a) if a else 0.0


# ---------------------------------------------------------------- 1. informativeness
def entropy(probs):
    """Shannon entropy in bits of a probability list. High means the model is unsure."""
    return -sum(p * math.log2(p) for p in probs if p > 0)


def least_confidence(probs):
    return 1.0 - max(probs)


def margin(probs):
    """Top probability minus the second. Small means a close call."""
    s = sorted(probs, reverse=True)
    return s[0] - (s[1] if len(s) > 1 else 0.0)


def vote_entropy(votes):
    """Entropy (bits) of the labels that a committee of models voted."""
    counts = {}
    for v in votes:
        counts[v] = counts.get(v, 0) + 1
    n = len(votes)
    return entropy([c / n for c in counts.values()]) if n else 0.0


def disagreement(votes):
    """0 when all votes agree. Otherwise one minus the share of the most common vote."""
    if not votes:
        return 0.0
    top = max(votes.count(v) for v in set(votes))
    return 1.0 - top / len(votes)


def pass_rate(outcomes):
    return mean(1.0 if o else 0.0 for o in outcomes)


def pass_variance(p):
    """p(1-p). Largest at p = 0.5. A task that is always passed or always failed carries no signal."""
    return p * (1.0 - p)


def difficulty_from_passes(passes, k):
    return 1.0 - passes / k


def sometimes_passes(passes, k):
    """The week 6b difficulty filter: keep a task when 0 < passes < k."""
    return 0 < passes < k


def rank_ids(scores, top=None, largest=True):
    """Ids sorted by score, ties broken by id. Pass top to keep only the first top ids."""
    ordered = sorted(scores, key=lambda i: ((-scores[i]) if largest else scores[i], i))
    return ordered if top is None else ordered[:top]


def cap_per_group(items, key, cap):
    """Keep at most cap items per group, in the order given. SWE-smith caps one task at 3 trajectories."""
    seen, out = {}, []
    for it in items:
        g = key(it)
        if seen.get(g, 0) < cap:
            seen[g] = seen.get(g, 0) + 1
            out.append(it)
    return out


def make_task_pool(n_tasks, ability=0.0, seed=0):
    """Made-up tasks. Task i has a hidden difficulty d. The current model passes it with p = sigmoid(ability - d)."""
    rng = random.Random(seed)
    return {"ability": ability, "difficulty": {i: rng.gauss(0.0, 1.8) for i in range(n_tasks)}}


def pass_probability(tasks, i):
    return sigmoid(tasks["ability"] - tasks["difficulty"][i])


def sample_passes(tasks, k, seed=0):
    """Attempt every task k times. Returns {task id: passes}. tasks comes from make_task_pool."""
    rng = random.Random(seed)
    out = {}
    for i in sorted(tasks["difficulty"]):
        p = pass_probability(tasks, i)
        out[i] = sum(rng.random() < p for _ in range(k))
    return out


def true_signal(tasks):
    """The hidden learning signal of each task in this toy: p(1-p) at the true pass probability."""
    return {i: pass_variance(pass_probability(tasks, i)) for i in tasks["difficulty"]}


def informativeness_demo(n_tasks=300, top=50, ks=(1, 4, 16, 64), seed=0):
    """How well does the pass-rate score from k attempts find the tasks with the most true signal?"""
    tasks = make_task_pool(n_tasks, ability=0.0, seed=seed)
    truth = rank_ids(true_signal(tasks), top)
    rows = []
    for k in ks:
        passes = sample_passes(tasks, k, seed=seed + 1)
        est = {i: pass_variance(passes[i] / k) for i in passes}
        # ties are broken by id, so break them by a seeded random key instead for a fair comparison
        rng = random.Random(seed + 2)
        jitter = {i: est[i] + rng.random() * 1e-9 for i in est}
        sel = rank_ids(jitter, top)
        kept = sum(sometimes_passes(passes[i], k) for i in passes)
        rows.append({"k": k, "overlap_with_truth": round(overlap(truth, sel), 3),
                     "share_0_lt_passes_lt_k": round(kept / n_tasks, 3)})
    return rows


# ---------------------------------------------------------------- 2. diversity, dedupe, importance resampling
def k_center_greedy(points, k, start=None):
    """Farthest-first traversal. points is {id: vector}. Each new center is the point farthest from all centers."""
    ids = sorted(points)
    if k >= len(ids):
        return ids
    if k <= 0:
        return []
    centers = [min(ids) if start is None else start]
    dmin = {i: euclid(points[i], points[centers[0]]) for i in ids}
    while len(centers) < k:
        nxt = min(ids, key=lambda i: (-dmin[i], i))
        centers.append(nxt)
        for i in ids:
            d = euclid(points[i], points[nxt])
            if d < dmin[i]:
                dmin[i] = d
    return centers


def covering_radius(points, centers):
    """The largest distance from any point to its nearest center. Smaller means the centers cover the data better."""
    return max(min(euclid(points[i], points[c]) for c in centers) for i in points)


def score_first_diverse(items, budget, min_dist):
    """Take items best score first. Skip one that is closer than min_dist to something already taken.
    items is a list of (id, score, vector). This is the shape of a score-first, diversity-aware filter."""
    chosen = []
    vecs = {}
    for iid, _score, vec in sorted(items, key=lambda t: (-t[1], t[0])):
        if len(chosen) >= budget:
            break
        if all(euclid(vec, v) >= min_dist for v in vecs.values()):
            chosen.append(iid)
            vecs[iid] = vec
    return chosen


def shingles(text, k=3):
    w = words(text)
    if len(w) < k:
        return {" ".join(w)} if w else set()
    return {" ".join(w[i:i + k]) for i in range(len(w) - k + 1)}


def jaccard(a, b):
    if not a and not b:
        return 1.0
    return len(a & b) / len(a | b)


_PRIME = (1 << 61) - 1


def minhash(sh, num_perm=64, seed=0):
    rng = random.Random(seed)
    params = [(rng.randrange(1, _PRIME), rng.randrange(0, _PRIME)) for _ in range(num_perm)]
    hs = [stable_hash(s) for s in sh]
    if not hs:
        return tuple([_PRIME] * num_perm)
    return tuple(min((a * h + b) % _PRIME for h in hs) for a, b in params)


def lsh_candidates(texts, k=3, num_perm=64, bands=16, seed=0):
    """Pairs whose MinHash signatures agree on every row of at least one band. Cheap. A pair with Jaccard similarity s
    is found with probability 1 - (1 - s^rows)^bands, so close pairs are usually found and some are missed."""
    if num_perm % bands:
        raise ValueError("bands must divide num_perm")
    rows = num_perm // bands
    sig = {i: minhash(shingles(t, k), num_perm, seed) for i, t in texts.items()}
    buckets = {}
    for i in sorted(texts):
        for b in range(bands):
            buckets.setdefault((b, sig[i][b * rows:(b + 1) * rows]), []).append(i)
    cands = set()
    for ids in buckets.values():
        for x in range(len(ids)):
            for y in range(x + 1, len(ids)):
                cands.add((ids[x], ids[y]))
    return sorted(cands)


def near_duplicates(texts, threshold=0.8, k=3, num_perm=64, bands=16, seed=0):
    """Pairs with shingle Jaccard at least threshold. MinHash bands pick candidates and exact Jaccard decides."""
    sh = {i: shingles(t, k) for i, t in texts.items()}
    out = []
    for a, b in lsh_candidates(texts, k, num_perm, bands, seed):
        j = jaccard(sh[a], sh[b])
        if j >= threshold:
            out.append((a, b, round(j, 4)))
    return out


def duplicate_groups(texts, threshold=0.8, **kw):
    """Groups of ids linked by near-duplicate pairs (union-find). Only groups of two or more are returned."""
    parent = {i: i for i in texts}

    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    for a, b, _j in near_duplicates(texts, threshold, **kw):
        ra, rb = find(a), find(b)
        if ra != rb:
            parent[max(ra, rb)] = min(ra, rb)
    groups = {}
    for i in texts:
        groups.setdefault(find(i), []).append(i)
    return sorted(sorted(g) for g in groups.values() if len(g) > 1)


def dedupe_keep_first(texts, threshold=0.8, **kw):
    """Keep the smallest id of every near-duplicate group. Returns (kept ids, {dropped id: kept id})."""
    dropped = {}
    for g in duplicate_groups(texts, threshold, **kw):
        for i in g[1:]:
            dropped[i] = g[0]
    return [i for i in sorted(texts) if i not in dropped], dropped


def hashed_features(text, buckets=256):
    """Hashed unigram and bigram counts, as a {bucket: count} dict."""
    w = words(text)
    feats = w + [a + " " + b for a, b in zip(w, w[1:])]
    out = {}
    for f in feats:
        b = stable_hash(f) % buckets
        out[b] = out.get(b, 0) + 1
    return out


def bucket_probs(texts, buckets=256, alpha=1.0):
    counts = [0.0] * buckets
    for t in texts:
        for b, c in hashed_features(t, buckets).items():
            counts[b] += c
    total = sum(counts) + alpha * buckets
    return [(c + alpha) / total for c in counts]


def dsir_log_weights(raw, target_texts, buckets=256, alpha=1.0):
    """For each raw document, the sum over its features of log(target prob / raw prob). raw is {id: text}.
    A simplified version of importance resampling with hashed n-gram features."""
    pt = bucket_probs(target_texts, buckets, alpha)
    pr = bucket_probs(list(raw.values()), buckets, alpha)
    out = {}
    for i, t in raw.items():
        out[i] = sum(c * math.log(pt[b] / pr[b]) for b, c in hashed_features(t, buckets).items())
    return out


def gumbel_top_k(log_weights, k, seed=0):
    """Sample k ids without replacement with probability proportional to exp(log weight)."""
    rng = random.Random(seed)
    keys = {}
    for i in sorted(log_weights):
        u = min(max(rng.random(), 1e-12), 1 - 1e-12)
        keys[i] = log_weights[i] - math.log(-math.log(u))
    return rank_ids(keys, k)


TOPICS = {
    "code": ["function", "compile", "variable", "return", "loop", "array", "debug", "parser", "module", "thread"],
    "food": ["simmer", "garlic", "butter", "recipe", "oven", "pepper", "dough", "sauce", "spoon", "broth"],
    "sport": ["match", "goal", "season", "coach", "league", "score", "pitch", "referee", "trophy", "sprint"],
}


def make_topic_docs(n_per_topic=100, doc_len=14, seed=0, topics=None):
    """Made-up documents. Each is random words from one topic's word list plus some common words.
    Returns {id: (topic, text)}."""
    rng = random.Random(seed)
    common = ["the", "a", "of", "and", "to", "in", "is", "with"]
    docs = {}
    for t in sorted(topics or TOPICS):
        for _ in range(n_per_topic):
            ws = [rng.choice(TOPICS[t]) if rng.random() < 0.6 else rng.choice(common) for _ in range(doc_len)]
            docs[len(docs)] = (t, " ".join(ws))
    return docs


def selection_share(docs, selected, topic):
    return mean(1.0 if docs[i][0] == topic else 0.0 for i in selected)


# ---------------------------------------------------------------- 3. gradient similarity (a LESS-like proxy)
def logistic_p(w, x):
    return sigmoid(dot(w, x))


def example_grad(w, x, y):
    """Gradient of the logistic loss for one example: (p - y) * x."""
    p = logistic_p(w, x)
    return [(p - y) * xi for xi in x]


def train_logistic(data, steps=200, lr=0.5, w0=None, dim=None):
    """Full-batch gradient descent on (x, y) pairs. Deterministic."""
    d = dim or len(data[0][0])
    w = list(w0) if w0 is not None else [0.0] * d
    n = len(data)
    for _ in range(steps):
        g = [0.0] * d
        for x, y in data:
            gi = example_grad(w, x, y)
            for j in range(d):
                g[j] += gi[j]
        w = [wj - lr * gj / n for wj, gj in zip(w, g)]
    return w


def accuracy(w, data):
    return mean(1.0 if (logistic_p(w, x) >= 0.5) == (y == 1) else 0.0 for x, y in data)


def _unit(v):
    n = norm(v)
    return [x / n for x in v]


def make_less_world(n_per_task=150, dim=8, seed=0, big_frac=0.2, big_scale=4.0, label_noise=0.05,
                    n_support=8, n_test=300):
    """Three made-up tasks share one linear model. Task A is the target. Task B is unrelated.
    Task C partly contradicts A. A fraction of examples has a large feature norm (a stand-in for long text)."""
    rng = random.Random(seed)
    wA = _unit([rng.gauss(0, 1) for _ in range(dim)])
    wB = _unit([rng.gauss(0, 1) for _ in range(dim)])
    wC = _unit([-0.8 * a + 0.6 * b for a, b in zip(wA, wB)])
    dirs = {"A": wA, "B": wB, "C": wC}

    def draw(task, scale_ok):
        x = [rng.gauss(0, 1) for _ in range(dim)]
        big = scale_ok and rng.random() < big_frac
        if big:
            x = [big_scale * v for v in x]
        y = 1 if dot(dirs[task], x) > 0 else 0
        if scale_ok and rng.random() < label_noise:
            y = 1 - y
        return x, y, big

    pool = []
    for task in "ABC":
        for _ in range(n_per_task):
            x, y, big = draw(task, True)
            pool.append({"id": len(pool), "task": task, "x": x, "y": y, "big": big})
    support = [draw("A", False)[:2] for _ in range(n_support)]
    test = [draw("A", False)[:2] for _ in range(n_test)]
    return {"pool": pool, "support": support, "test": test, "dirs": dirs}


def make_projection(d_in, d_out, seed=0):
    """A Rademacher random projection matrix (entries +1 or -1, scaled). The JL lemma says it roughly keeps dot products."""
    rng = random.Random(seed)
    s = 1.0 / math.sqrt(d_out)
    return [[s if rng.random() < 0.5 else -s for _ in range(d_in)] for _ in range(d_out)]


def project(mat, v):
    return [dot(row, v) for row in mat]


def less_scores(pool, support, w, proj=None, use_cosine=True):
    """Score each pool example by how well its gradient lines up with the mean gradient of the target examples.
    use_cosine=False uses the raw dot product instead, which favours large gradients."""
    gt = [0.0] * len(w)
    for x, y in support:
        for j, v in enumerate(example_grad(w, x, y)):
            gt[j] += v / len(support)
    if proj is not None:
        gt = project(proj, gt)
    out = {}
    for ex in pool:
        g = example_grad(w, ex["x"], ex["y"])
        if proj is not None:
            g = project(proj, g)
        out[ex["id"]] = cosine(g, gt) if use_cosine else dot(g, gt)
    return out


def less_demo(seed=0, frac=0.2, warmup_frac=0.05, warmup_steps=20, proj_dim=None):
    """Train on a random, a gradient-selected (cosine), a dot-product-selected and the full pool.
    Report test accuracy on target task A and how many selected examples came from each task."""
    world = make_less_world(seed=seed)
    pool = world["pool"]
    rng = random.Random(seed + 100)
    k = int(len(pool) * frac)
    warm = rng.sample(pool, max(2, int(len(pool) * warmup_frac)))
    dim = len(pool[0]["x"])
    w0 = train_logistic([(e["x"], e["y"]) for e in warm], steps=warmup_steps, dim=dim)
    proj = make_projection(dim, proj_dim, seed + 1) if proj_dim else None

    def fit(ids):
        by_id = {e["id"]: e for e in pool}
        return train_logistic([(by_id[i]["x"], by_id[i]["y"]) for i in ids], dim=dim)

    picks = {
        "random": sorted(e["id"] for e in rng.sample(pool, k)),
        "less_cosine": rank_ids(less_scores(pool, world["support"], w0, proj, True), k),
        "dot_product": rank_ids(less_scores(pool, world["support"], w0, proj, False), k),
        "full_pool": [e["id"] for e in pool],
    }
    by_id = {e["id"]: e for e in pool}
    rows = {}
    for name, ids in picks.items():
        w = fit(ids)
        rows[name] = {"n": len(ids), "acc_A": round(accuracy(w, world["test"]), 3),
                      "from_A": sum(by_id[i]["task"] == "A" for i in ids),
                      "from_C": sum(by_id[i]["task"] == "C" for i in ids),
                      "big": sum(by_id[i]["big"] for i in ids)}
    return rows


# ---------------------------------------------------------------- 4. label noise and data maps
def make_blobs(n_per_class, k=3, dim=2, sep=3.0, seed=0, spreads=None):
    """k Gaussian clusters on a circle. spreads gives each cluster its own standard deviation. Returns (points, true labels)."""
    rng = random.Random(seed)
    spreads = spreads or [1.0] * k
    xs, ys = [], []
    for c in range(k):
        ang = 2 * math.pi * c / k
        centre = [sep * math.cos(ang), sep * math.sin(ang)] + [0.0] * (dim - 2)
        for _ in range(n_per_class):
            xs.append([m + rng.gauss(0, spreads[c]) for m in centre])
            ys.append(c)
    return xs, ys


def flip_labels(labels, rate, k, seed=0):
    """Make some labels wrong. A flipped label moves to the next class (c + 1) mod k, so the noise is class-conditional.
    Returns (noisy labels, sorted ids that were flipped)."""
    rng = random.Random(seed)
    noisy, flipped = list(labels), []
    for i, y in enumerate(labels):
        if rng.random() < rate:
            noisy[i] = (y + 1) % k
            flipped.append(i)
    return noisy, flipped


def centroid_probs(train_x, train_y, test_x, k, tau=1.5):
    """A simple classifier: softmax of minus squared distance to each class mean, scaled by tau."""
    cent = []
    for c in range(k):
        pts = [x for x, y in zip(train_x, train_y) if y == c]
        d = len(train_x[0])
        cent.append([mean(p[j] for p in pts) if pts else 0.0 for j in range(d)])
    out = []
    for x in test_x:
        logits = [-sum((a - b) ** 2 for a, b in zip(x, c)) / (2 * tau * tau) for c in cent]
        m = max(logits)
        e = [math.exp(v - m) for v in logits]
        s = sum(e)
        out.append([v / s for v in e])
    return out


def cross_val_probs(xs, labels, k, folds=5, seed=0, tau=1.5):
    """Out-of-sample predicted probabilities: every example is scored by a model that did not train on it."""
    n = len(xs)
    order = list(range(n))
    random.Random(seed).shuffle(order)
    fold_of = {i: pos % folds for pos, i in enumerate(order)}
    probs = [None] * n
    for f in range(folds):
        tr = [i for i in range(n) if fold_of[i] != f]
        te = [i for i in range(n) if fold_of[i] == f]
        p = centroid_probs([xs[i] for i in tr], [labels[i] for i in tr], [xs[i] for i in te], k, tau)
        for i, pi in zip(te, p):
            probs[i] = pi
    return probs


def class_thresholds(probs, labels, k):
    """t_j: the mean predicted probability of class j over the examples labelled j."""
    t = []
    for j in range(k):
        ps = [p[j] for p, y in zip(probs, labels) if y == j]
        t.append(mean(ps))
    return t


def confident_joint(probs, labels, k):
    """C[i][j]: examples labelled i that are confidently class j. Confidently means p_j >= t_j, and among
    the classes that pass, take the one with the largest probability. Returns (matrix, {example id: j})."""
    t = class_thresholds(probs, labels, k)
    C = [[0] * k for _ in range(k)]
    assigned = {}
    for idx, (p, y) in enumerate(zip(probs, labels)):
        ok = [j for j in range(k) if p[j] >= t[j]]
        if not ok:
            continue
        j = max(ok, key=lambda c: (p[c], -c))
        C[y][j] += 1
        assigned[idx] = j
    return C, assigned


def find_label_issues(probs, labels, k):
    """Ids whose confident class differs from the given label, worst first (lowest self-confidence)."""
    _C, assigned = confident_joint(probs, labels, k)
    bad = [i for i, j in assigned.items() if j != labels[i]]
    return sorted(bad, key=lambda i: (probs[i][labels[i]], i))


def self_confidence_rank(probs, labels):
    """All ids, least confident in the given label first. The simplest baseline."""
    return sorted(range(len(labels)), key=lambda i: (probs[i][labels[i]], i))


def precision_recall(flagged, truth):
    flagged, truth = set(flagged), set(truth)
    tp = len(flagged & truth)
    return (tp / len(flagged) if flagged else 0.0), (tp / len(truth) if truth else 0.0)


def label_noise_demo(n_per_class=100, rate=0.1, seed=0, spreads=(0.6, 0.9, 2.2)):
    xs, ys = make_blobs(n_per_class, 3, 2, 3.0, seed, list(spreads))
    noisy, flipped = flip_labels(ys, rate, 3, seed + 1)
    probs = cross_val_probs(xs, noisy, 3, seed=seed + 2)
    cl = find_label_issues(probs, noisy, 3)
    base = self_confidence_rank(probs, noisy)[:len(cl)]
    p_cl, r_cl = precision_recall(cl, flipped)
    p_b, r_b = precision_recall(base, flipped)
    return {"n": len(xs), "flipped": len(flipped), "cl_flagged": len(cl), "cl_precision": round(p_cl, 3),
            "cl_recall": round(r_cl, 3), "base_precision": round(p_b, 3), "base_recall": round(r_b, 3),
            "unfixed_error_rate": round(mean(1.0 if probs[i].index(max(probs[i])) != noisy[i] else 0.0
                                             for i in range(len(xs))), 3)}


def label_noise_study(seeds=8, **kw):
    """Average label_noise_demo over several seeds."""
    rows = [label_noise_demo(seed=sd, **kw) for sd in range(seeds)]
    keys = ["flipped", "cl_flagged", "cl_precision", "cl_recall", "base_precision", "base_recall"]
    return {k: round(mean(r[k] for r in rows), 3) for k in keys}


def training_dynamics(p_true_by_epoch, correct_by_epoch):
    """Data-map statistics for one example. confidence is the mean probability of the given label across epochs,
    variability is the standard deviation of that probability, and correctness is the share of epochs predicted right."""
    return {"confidence": mean(p_true_by_epoch), "variability": stdev(p_true_by_epoch),
            "correctness": mean(1.0 if c else 0.0 for c in correct_by_epoch)}


def train_logistic_history(data, epochs=10, lr=0.1, seed=0):
    """One-example-at-a-time SGD for a binary task. After each epoch, record p(given label) and whether it is right
    for every example. Returns (p_hist[i][e], correct_hist[i][e])."""
    rng = random.Random(seed)
    d = len(data[0][0])
    w = [0.0] * d
    order = list(range(len(data)))
    p_hist = [[] for _ in data]
    c_hist = [[] for _ in data]
    for _ in range(epochs):
        rng.shuffle(order)
        for i in order:
            x, y = data[i]
            g = example_grad(w, x, y)
            w = [wj - lr * gj for wj, gj in zip(w, g)]
        for i, (x, y) in enumerate(data):
            p1 = logistic_p(w, x)
            p_hist[i].append(p1 if y == 1 else 1 - p1)
            c_hist[i].append((p1 >= 0.5) == (y == 1))
    return p_hist, c_hist


def data_map_regions(stats, fraction=1 / 3):
    """Split ids into the three regions of a data map: hard (lowest confidence), easy (highest confidence)
    and ambiguous (highest variability). An id can be in more than one set, so ambiguous is taken first."""
    ids = sorted(stats)
    n = max(1, int(len(ids) * fraction))
    amb = rank_ids({i: stats[i]["variability"] for i in ids}, n)
    rest = [i for i in ids if i not in set(amb)]
    hard = rank_ids({i: stats[i]["confidence"] for i in rest}, n, largest=False)
    easy = rank_ids({i: stats[i]["confidence"] for i in rest if i not in set(hard)}, n)
    return {"ambiguous": sorted(amb), "hard": sorted(hard), "easy": sorted(easy)}


def data_map_demo(n=300, noise=0.08, seed=0, fraction=0.15):
    rng = random.Random(seed)
    xs = [[rng.gauss(0, 1), rng.gauss(0, 1), 1.0] for _ in range(n)]
    clean = [1 if x[0] + 0.5 * x[1] > 0 else 0 for x in xs]
    noisy, flipped = [], []
    for i, y in enumerate(clean):
        if rng.random() < noise:
            noisy.append(1 - y)
            flipped.append(i)
        else:
            noisy.append(y)
    ph, ch = train_logistic_history(list(zip(xs, noisy)), epochs=10, lr=0.1, seed=seed)
    stats = {i: training_dynamics(ph[i], ch[i]) for i in range(n)}
    reg = data_map_regions(stats, fraction)
    f = set(flipped)
    return {"n": n, "flipped": len(f),
            "share_flipped_in_hard": round(len(f & set(reg["hard"])) / max(1, len(reg["hard"])), 3),
            "share_flipped_in_easy": round(len(f & set(reg["easy"])) / max(1, len(reg["easy"])), 3),
            "share_flipped_in_ambiguous": round(len(f & set(reg["ambiguous"])) / max(1, len(reg["ambiguous"])), 3),
            "base_rate": round(len(f) / n, 3)}


# ---------------------------------------------------------------- 5. agreement among annotators
def percent_agreement(a, b):
    return mean(1.0 if x == y else 0.0 for x, y in zip(a, b))


def _marginals(vals):
    n = len(vals)
    out = {}
    for v in vals:
        out[v] = out.get(v, 0) + 1
    return {k: c / n for k, c in out.items()}


def cohen_kappa(a, b):
    """Two coders. Chance agreement uses each coder's own label shares."""
    po = percent_agreement(a, b)
    ma, mb = _marginals(a), _marginals(b)
    pe = sum(ma.get(c, 0.0) * mb.get(c, 0.0) for c in set(ma) | set(mb))
    return 1.0 if pe >= 1.0 else (po - pe) / (1 - pe)


def scott_pi(a, b):
    """Two coders. Chance agreement uses the pooled label shares of both coders."""
    po = percent_agreement(a, b)
    pooled = _marginals(list(a) + list(b))
    pe = sum(p * p for p in pooled.values())
    return 1.0 if pe >= 1.0 else (po - pe) / (1 - pe)


def weighted_kappa(a, b, dist):
    """Cohen's kappa with partial credit. dist(x, y) is 0 for equal labels and 1 for maximal disagreement."""
    do = mean(dist(x, y) for x, y in zip(a, b))
    ma, mb = _marginals(a), _marginals(b)
    de = sum(pa * pb * dist(x, y) for x, pa in ma.items() for y, pb in mb.items())
    return 1.0 if de == 0 else 1 - do / de


def fleiss_multi_pi(ratings):
    """Fleiss's coefficient for any number of raters, with the same number r of raters on every item.
    It generalises Scott's pi (pooled shares), not Cohen's kappa, although it is often called Fleiss's kappa.
    ratings is a list with one list of labels per item."""
    r = len(ratings[0])
    if r < 2 or any(len(x) != r for x in ratings):
        raise ValueError("every item needs the same number of raters, at least two")
    ao = 0.0
    for item in ratings:
        counts = {}
        for v in item:
            counts[v] = counts.get(v, 0) + 1
        ao += sum(c * (c - 1) for c in counts.values()) / (r * (r - 1))
    ao /= len(ratings)
    pooled = _marginals([v for item in ratings for v in item])
    pe = sum(p * p for p in pooled.values())
    return 1.0 if pe >= 1.0 else (ao - pe) / (1 - pe)


def nominal_distance(a, b):
    return 0.0 if a == b else 1.0


def squared_distance(a, b):
    return float((a - b) ** 2)


def krippendorff_alpha(units, dist=nominal_distance):
    """Krippendorff's alpha. units is a list with one list of values per item. None means a missing rating,
    and the number of raters may differ per item. Items with fewer than two values are ignored."""
    vals = [[v for v in u if v is not None] for u in units]
    vals = [u for u in vals if len(u) >= 2]
    coinc = {}
    for u in vals:
        m = len(u)
        for i in range(m):
            for j in range(m):
                if i != j:
                    key = (u[i], u[j])
                    coinc[key] = coinc.get(key, 0.0) + 1.0 / (m - 1)
    n_c = {}
    for (c, _k), v in coinc.items():
        n_c[c] = n_c.get(c, 0.0) + v
    n = sum(n_c.values())
    if n <= 1:
        return 1.0
    do = sum(v * dist(c, k) for (c, k), v in coinc.items()) / n
    de = sum(n_c[c] * n_c[k] * dist(c, k) for c in n_c for k in n_c) / (n * (n - 1))
    return 1.0 if de == 0 else 1 - do / de


def ensemble_max(labels):
    """The most severe label wins. This is how the SWE-bench Verified post ensembles three annotators."""
    return max(labels)


def ensemble_median(labels):
    s = sorted(labels)
    return s[len(s) // 2]


def keep_by_severity(annotations, ensemble=ensemble_max, threshold=2):
    """annotations is {id: [[labels of criterion 1 by annotator], [labels of criterion 2 by annotator], ...]}.
    An item is kept only when every criterion's ensemble label is below threshold."""
    return sorted(i for i, crits in annotations.items() if all(ensemble(c) < threshold for c in crits))


def simulate_annotation(n_items=500, n_annotators=3, accuracy=0.7, seed=0, severity_probs=(0.45, 0.25, 0.2, 0.1)):
    """Made-up severity labels 0 to 3 for two criteria. Each annotator reports the true label with probability
    accuracy and otherwise one step off. Returns (annotations, ids whose true severity is below 2 on both)."""
    rng = random.Random(seed)
    ann, ok = {}, []
    for i in range(n_items):
        truth = [rng.choices(range(4), severity_probs)[0] for _ in range(2)]
        crits = []
        for t in truth:
            labs = []
            for _ in range(n_annotators):
                if rng.random() < accuracy:
                    labs.append(t)
                else:
                    step = rng.choice([-1, 1])
                    labs.append(min(3, max(0, t + step)))
            crits.append(labs)
        ann[i] = crits
        if all(t < 2 for t in truth):
            ok.append(i)
    return ann, ok


def selection_quality(kept, truly_ok, n_total):
    """Precision of the kept set (how many are truly fine), recall of the fine items, and the share kept."""
    kept, truly_ok = set(kept), set(truly_ok)
    tp = len(kept & truly_ok)
    return {"kept_share": round(len(kept) / n_total, 3),
            "precision": round(tp / len(kept), 3) if kept else 0.0,
            "recall": round(tp / len(truly_ok), 3) if truly_ok else 0.0}


# ---------------------------------------------------------------- 6. item response theory and tiny benchmarks
def kendall_tau_b(x, y):
    """Kendall's tau-b rank correlation, which handles ties. Returns 0.0 when one side is constant."""
    n = len(x)
    conc = disc = tx = ty = 0
    for i in range(n):
        for j in range(i + 1, n):
            dx, dy = x[i] - x[j], y[i] - y[j]
            if dx == 0 and dy == 0:
                continue
            if dx == 0:
                tx += 1
            elif dy == 0:
                ty += 1
            elif (dx > 0) == (dy > 0):
                conc += 1
            else:
                disc += 1
    den = math.sqrt((conc + disc + tx) * (conc + disc + ty))
    return 0.0 if den == 0 else (conc - disc) / den


def pearson(x, y):
    mx, my = mean(x), mean(y)
    sx, sy = stdev(x), stdev(y)
    if sx == 0 or sy == 0:
        return 0.0
    return mean((a - mx) * (b - my) for a, b in zip(x, y)) / (sx * sy)


def make_irt_world(n_models, n_items, seed=0, dims=1):
    """Made-up response matrix. The chance that model m gets item i right is sigmoid(alpha_i . theta_m - beta_i).
    With dims=2 the abilities are two correlated skills and each item loads on them in its own mix."""
    rng = random.Random(seed)
    theta, alpha, beta = [], [], []
    for _ in range(n_models):
        t0 = rng.gauss(0, 1)
        theta.append([t0] if dims == 1 else [t0, 0.5 * t0 + math.sqrt(0.75) * rng.gauss(0, 1)])
    for _ in range(n_items):
        a = math.exp(rng.gauss(0, 0.35))
        if dims == 1:
            alpha.append([a])
        else:
            phi = rng.random() * math.pi / 2
            alpha.append([a * math.cos(phi), a * math.sin(phi)])
        beta.append(rng.gauss(0, 1.2))
    resp = [[1 if rng.random() < sigmoid(dot(alpha[i], theta[m]) - beta[i]) else 0 for i in range(n_items)]
            for m in range(n_models)]
    return {"resp": resp, "theta": theta, "alpha": alpha, "beta": beta}


def fit_irt(resp, model="2pl", epochs=150, lr=2.0):
    """Fit a one-dimensional IRT model by gradient ascent with weak priors. 1pl fixes every alpha at 1 (the Rasch model).
    Returns (theta list, alpha list, beta list). Priors: theta ~ N(0,1), alpha pulled toward 1, beta lightly shrunk."""
    M, N = len(resp), len(resp[0])
    acc_m = [mean(r) for r in resp]
    acc_i = [mean(resp[m][i] for m in range(M)) for i in range(N)]
    theta = [logit(a * 0.98 + 0.01) / 1.5 for a in acc_m]
    beta = [-logit(a * 0.98 + 0.01) for a in acc_i]
    alpha = [1.0] * N
    free_alpha = model == "2pl"
    for _ in range(epochs):
        gt = [0.0] * M
        ga = [0.0] * N
        gb = [0.0] * N
        for m in range(M):
            tm = theta[m]
            row = resp[m]
            for i in range(N):
                e = row[i] - sigmoid(alpha[i] * tm - beta[i])
                gt[m] += alpha[i] * e
                ga[i] += tm * e
                gb[i] -= e
        for m in range(M):
            theta[m] += lr * (gt[m] - theta[m]) / N
        for i in range(N):
            beta[i] += lr * (gb[i] - 0.1 * beta[i]) / M
            if free_alpha:
                alpha[i] += lr * (ga[i] - (alpha[i] - 1.0)) / M
                alpha[i] = min(max(alpha[i], 0.05), 5.0)
    return theta, alpha, beta


def fit_theta(alpha, beta, items, y):
    """MAP ability of one new model from its answers y[i] on a subset of items, with the item parameters fixed."""
    t = 0.0
    for _ in range(30):
        g = -t
        h = 1.0
        for i in items:
            p = sigmoid(alpha[i] * t - beta[i])
            g += alpha[i] * (y[i] - p)
            h += alpha[i] ** 2 * p * (1 - p)
        t += g / h
    return t


def estimate_pirt(alpha, beta, items, y):
    """Predicted accuracy on all items from a model's answers on a few: fit its ability, then average the predicted chance."""
    t = fit_theta(alpha, beta, items, y)
    return mean(sigmoid(alpha[i] * t - beta[i]) for i in range(len(alpha)))


def fisher_information(alpha_i, beta_i, theta):
    p = sigmoid(alpha_i * theta - beta_i)
    return alpha_i ** 2 * p * (1 - p)


def select_random(n_items, k, seed=0):
    return sorted(random.Random(seed).sample(range(n_items), k))


def select_stratified(item_acc, k):
    """Sort items by how often the training models got them right, and take k evenly spaced ones."""
    order = sorted(range(len(item_acc)), key=lambda i: (item_acc[i], i))
    n = len(order)
    return sorted(order[int((j + 0.5) * n / k)] for j in range(k))


def select_irt_grid(alpha, beta, thetas, k):
    """For ability levels spread over the fitted models, pick the item with the most Fisher information at that level
    (taking levels in turn) until k items are chosen."""
    ts = sorted(thetas)
    n = len(ts)
    grid = [ts[int((j + 0.5) * n / k)] for j in range(k)]
    chosen = []
    for g in grid:
        best = max((i for i in range(len(alpha)) if i not in chosen),
                   key=lambda i: (fisher_information(alpha[i], beta[i], g), -i))
        chosen.append(best)
    return sorted(chosen)


def select_anchors(resp_train, k):
    """Greedy k-center on the columns of the response matrix (an item is a vector of right/wrong over models).
    Returns (anchor ids, weights) where an anchor's weight is the share of items closest to it."""
    M, N = len(resp_train), len(resp_train[0])
    cols = [tuple(resp_train[m][i] for m in range(M)) for i in range(N)]

    def d(a, b):
        return sum(x != y for x, y in zip(cols[a], cols[b])) / M

    var = [pass_variance(mean(c)) for c in cols]
    start = min(range(N), key=lambda i: (-var[i], i))
    anchors = [start]
    dmin = [d(i, start) for i in range(N)]
    while len(anchors) < k:
        nxt = min(range(N), key=lambda i: (-dmin[i], i))
        anchors.append(nxt)
        for i in range(N):
            dmin[i] = min(dmin[i], d(i, nxt))
    owner = [min(range(len(anchors)), key=lambda a: (d(i, anchors[a]), a)) for i in range(N)]
    w = [owner.count(a) / N for a in range(len(anchors))]
    return anchors, w


def weighted_accuracy(items, weights, y):
    return sum(w * y[i] for i, w in zip(items, weights)) / sum(weights)


def tiny_benchmark_study(seed=0, n_train=40, n_test=24, n_items=200, k=24, dims=1, epochs=120):
    """Fit IRT on train models, choose k items, then estimate each held-out model's full-benchmark accuracy from
    those k items. Report Kendall tau (ranking) and mean absolute error (accuracy) for several recipes."""
    world = make_irt_world(n_train + n_test, n_items, seed, dims)
    R = world["resp"]
    train, test = R[:n_train], R[n_train:]
    theta, alpha, beta = fit_irt(train, "2pl", epochs)
    _t1, _a1, b1 = fit_irt(train, "1pl", epochs)
    a1 = [1.0] * n_items
    item_acc = [mean(train[m][i] for m in range(n_train)) for i in range(n_items)]
    full = [mean(r) for r in test]
    rnd = select_random(n_items, k, seed + 7)
    strat = select_stratified(item_acc, k)
    irt_sel = select_irt_grid(alpha, beta, theta, k)
    anc, anc_w = select_anchors(train, k)
    recipes = {
        "random, mean": lambda y: mean(y[i] for i in rnd),
        "stratified by difficulty, mean": lambda y: mean(y[i] for i in strat),
        "anchor points, weighted mean": lambda y: weighted_accuracy(anc, anc_w, y),
        "random, 2PL ability": lambda y: estimate_pirt(alpha, beta, rnd, y),
        "IRT-chosen, 2PL ability": lambda y: estimate_pirt(alpha, beta, irt_sel, y),
        "IRT-chosen, 1PL ability": lambda y: estimate_pirt(a1, b1, select_irt_grid(a1, b1, theta, k), y),
    }
    out = {}
    for name, f in recipes.items():
        est = [f(y) for y in test]
        out[name] = {"tau": round(kendall_tau_b(est, full), 3),
                     "mae": round(mean(abs(e - t) for e, t in zip(est, full)), 4)}
    return out


def tiny_benchmark_average(seeds=20, **kw):
    """Average tiny_benchmark_study over several made-up worlds. Returns {recipe: {tau, tau_sd, mae}}."""
    rows = [tiny_benchmark_study(seed=s, **kw) for s in range(seeds)]
    out = {}
    for name in rows[0]:
        taus = [r[name]["tau"] for r in rows]
        out[name] = {"tau": round(mean(taus), 3), "tau_sd": round(stdev(taus), 3),
                     "mae": round(mean(r[name]["mae"] for r in rows), 4)}
    return out


# ---------------------------------------------------------------- 7. EvalGen-style alignment and criteria drift
@dataclass(frozen=True)
class Assertion:
    """A yes/no check on one output. fn(output) is True when the output passes."""
    name: str
    criterion: str
    fn: object = field(compare=False, hash=False, repr=False)


def coverage(fails, good):
    """Of the outputs the user graded bad, the share that the assertion set fails. fails[i] is True when some assertion fails output i."""
    bad = [i for i in range(len(good)) if not good[i]]
    return mean(1.0 if fails[i] else 0.0 for i in bad) if bad else 0.0


def false_failure_rate(fails, good):
    """Of the outputs the user graded good, the share that the assertion set fails anyway."""
    ok = [i for i in range(len(good)) if good[i]]
    return mean(1.0 if fails[i] else 0.0 for i in ok) if ok else 0.0


def alignment(cov, ffr):
    """Harmonic mean of coverage and (1 - false failure rate)."""
    a, b = cov, 1.0 - ffr
    return 0.0 if a + b == 0 else 2 * a * b / (a + b)


def set_fails(assertions, outputs):
    return [any(not a.fn(o) for a in assertions) for o in outputs]


def selectivity(assertion, outputs):
    """The pass rate of an assertion over a set of outputs."""
    return mean(1.0 if assertion.fn(o) else 0.0 for o in outputs)


def failure_score(output, assertions, sel):
    """Sum of the selectivity of the assertions that fail this output. A high score means a rarely-failing assertion
    failed it, so it is probably bad. The paper's formula and its prose disagree on the indicator. This follows the prose."""
    return sum(sel[a.name] for a in assertions if not a.fn(output))


def grading_order(outputs, assertions, policy, seed=0):
    """The order in which to ask a person to grade outputs. policy is random, highest, lowest or alternating."""
    ids = [o["id"] for o in outputs]
    if policy == "random":
        ids = list(ids)
        random.Random(seed).shuffle(ids)
        return ids
    sel = {a.name: selectivity(a, outputs) for a in assertions}
    score = {o["id"]: failure_score(o, assertions, sel) for o in outputs}
    hi = rank_ids(score)
    if policy == "highest":
        return hi
    lo = rank_ids(score, largest=False)
    if policy == "lowest":
        return lo
    if policy == "alternating":
        order, seen = [], set()
        for a, b in zip(hi, lo):
            for x in (a, b):
                if x not in seen:
                    seen.add(x)
                    order.append(x)
        return order
    raise ValueError("unknown policy " + policy)


def select_per_criterion(assertions, outputs, graded_ids, good_by_id, ffr_max=0.2):
    """For each criterion pick the assertion with the best alignment on the graded outputs, among those whose false
    failure rate is at most ffr_max. Returns {criterion: Assertion or None}."""
    sub = [o for o in outputs if o["id"] in graded_ids]
    good = [good_by_id[o["id"]] for o in sub]
    chosen = {}
    for crit in sorted({a.criterion for a in assertions}):
        best, best_score = None, -1.0
        for a in sorted((x for x in assertions if x.criterion == crit), key=lambda x: x.name):
            f = set_fails([a], sub)
            ffr = false_failure_rate(f, good)
            if ffr > ffr_max:
                continue
            s = alignment(coverage(f, good), ffr)
            if s > best_score:
                best, best_score = a, s
        chosen[crit] = best
    return chosen


def make_outputs(n=100, seed=0):
    """Made-up outputs of an entity-extraction prompt: the share of extracted entities that are proper nouns,
    the word count, and whether a hashtag was copied."""
    rng = random.Random(seed)
    out = []
    for i in range(n):
        out.append({"id": i, "proper_frac": rng.choices([0.0, 1 / 3, 0.5, 2 / 3, 1.0], [10, 15, 15, 20, 40])[0],
                    "words": rng.randint(20, 90), "hashtag": rng.random() < 0.3})
    return out


def user_grade_strict(o):
    """Version 1 of the made-up user's rule: every entity is a proper noun, and at most 60 words."""
    return o["proper_frac"] >= 1.0 and o["words"] <= 60


def user_grade_relaxed(o):
    """Version 2, after the user has seen outputs: most entities are proper nouns is enough."""
    return o["proper_frac"] >= 0.5 and o["words"] <= 60


def candidate_assertions():
    out = []
    for t in (1.0, 0.6, 0.5, 0.3):
        out.append(Assertion(f"proper>={t}", "proper_nouns", lambda o, t=t: o["proper_frac"] >= t - 1e-9))
    for cap in (40, 50, 60, 80):
        out.append(Assertion(f"words<={cap}", "length", lambda o, cap=cap: o["words"] <= cap))
    out.append(Assertion("no_hashtag", "hashtags", lambda o: not o["hashtag"]))
    return out


def set_alignment(assertions, outputs, good):
    f = set_fails(assertions, outputs)
    ffr = false_failure_rate(f, good)
    return alignment(coverage(f, good), ffr)


def criteria_drift_demo(n=200, n_grade=20, seed=0, ffr_max=0.2, policy="random"):
    """A user grades n_grade outputs with the strict rule. The assertions picked from that look good on those grades.
    Then the user's rule relaxes. The old pick scores worse against the new grades. Regrading the same outputs under the
    new rule and picking again recovers."""
    outs = make_outputs(n, seed)
    cands = candidate_assertions()
    graded = set(grading_order(outs, cands, policy, seed)[:n_grade])
    g1 = {o["id"]: user_grade_strict(o) for o in outs}
    g2 = {o["id"]: user_grade_relaxed(o) for o in outs}
    pick1 = select_per_criterion(cands, outs, graded, g1, ffr_max)
    pick2 = select_per_criterion(cands, outs, graded, g2, ffr_max)
    set1 = [a for a in pick1.values() if a]
    set2 = [a for a in pick2.values() if a]
    good1 = [g1[o["id"]] for o in outs]
    good2 = [g2[o["id"]] for o in outs]
    return {"graded": n_grade, "picked_early": sorted(a.name for a in set1), "picked_after_drift": sorted(a.name for a in set2),
            "early_set_vs_strict_grades": round(set_alignment(set1, outs, good1), 3),
            "early_set_vs_relaxed_grades": round(set_alignment(set1, outs, good2), 3),
            "regraded_set_vs_relaxed_grades": round(set_alignment(set2, outs, good2), 3)}


def sampling_study(policies=("random", "highest", "lowest", "alternating"), worlds=10, n=100, n_grade=16):
    """Alignment against the strict grades on all outputs, when the user grades only n_grade outputs chosen by each policy.
    Returns {policy: {mean, sd}} over several made-up worlds."""
    cands = candidate_assertions()
    res = {p: [] for p in policies}
    for w in range(worlds):
        outs = make_outputs(n, w)
        g = {o["id"]: user_grade_strict(o) for o in outs}
        good = [g[o["id"]] for o in outs]
        for p in policies:
            graded = set(grading_order(outs, cands, p, seed=w)[:n_grade])
            picks = [a for a in select_per_criterion(cands, outs, graded, g).values() if a]
            res[p].append(set_alignment(picks, outs, good))
    return {p: {"mean": round(mean(v), 3), "sd": round(stdev(v), 3)} for p, v in res.items()}


def paper_table_alignment():
    """Alignment recomputed from the coverage and false failure rates printed in the EvalGen paper's Table 1."""
    rows = {"medical EvalGen and SPADE": (0.33, 0.10), "product EvalGen": (0.73, 0.39), "product SPADE": (0.49, 0.39)}
    return {k: round(100 * alignment(c, f), 2) for k, (c, f) in rows.items()}


# ---------------------------------------------------------------- 8. trace-to-dataset pipeline with quality gates
@dataclass
class Trace:
    id: int
    task_id: str
    repo: str
    prompt: str
    actions: list
    passed: bool
    consent_train: bool = True
    defect: str = ""        # ground truth for the lab only: what was injected, "" for none


REPOS = ["libalpha", "libbeta", "libgamma", "libdelta", "libomega", "libsigma"]
NOUNS = ["date", "row", "path", "unit", "cfg", "key", "vec", "text", "blob", "id", "grid", "tok"]
ACTS = ["parse", "merge", "split", "norm", "read", "join", "scale", "trim", "load", "sort", "pad", "count"]
FUNCS = [f"{a}_{n}" for a in ACTS for n in NOUNS]
SYMPTOMS = ["returns an empty list for repeated keys.", "raises a type error when the input is a tuple.",
            "drops the last item of an odd-length input.", "ignores the timezone argument.",
            "loops forever on an empty string.", "mixes up rows and columns on wide input.",
            "gives a wrong total after the second call.", "fails on paths that end with a slash.",
            "returns none when the key is missing.", "rounds half values the wrong way.",
            "skips blank lines at the top.", "keeps stale values between calls."]
VERBS = ["view", "grep", "edit", "run"]
EMAIL_TAIL = " Please email jo.lee@example.com when it is done."


def make_prompt(repo, fn, symptom, rng=None):
    """A task prompt. The example numbers make each prompt unique. Without rng they are fixed."""
    r = rng or random.Random(0)
    a, b, c, d = [r.randint(100, 999) for _ in range(4)]
    return (f"In {repo} the function {fn} {symptom} Example: {fn} of {a} and {b} gives {c} but should give {d}. "
            "Fix the bug and make the failing tests pass.")


def _reword(prompt, rng):
    """Change one word in the middle of a prompt: a near copy, not an exact one."""
    w = prompt.split()
    i = rng.randrange(3, len(w) - 6)
    w[i] = w[i] + "x"
    return " ".join(w)


def make_eval_prompts(n=12, seed=0):
    rng = random.Random(seed)
    seen, out = set(), []
    while len(out) < n:
        p = make_prompt(rng.choice(REPOS), rng.choice(FUNCS), rng.choice(SYMPTOMS), rng)
        if p not in seen:
            seen.add(p)
            out.append(p)
    return out


def make_trace_pool(n_tasks=150, seed=0, eval_prompts=None):
    """A made-up pool of agent traces with known injected defects. Defect names: loop, tamper (edits the tests),
    pii, leak (prompt is a near copy of an eval prompt), near_dup (task is a near copy of an earlier task),
    truncated (no submit step), weak_tests (passes, but only because the tests are weak, and nothing can see it)."""
    rng = random.Random(seed)
    eval_prompts = eval_prompts or make_eval_prompts(12, seed + 1)
    tasks, used = [], set(eval_prompts)
    for t in range(n_tasks):
        kind = rng.choices(["plain", "near_dup", "leak"], [86, 8, 6])[0] if t >= 3 else "plain"
        if kind == "near_dup":
            src = rng.choice([x for x in tasks if x["kind"] == "plain"])
            prompt, repo = _reword(src["prompt"], rng), src["repo"]
        elif kind == "leak":
            prompt = _reword(rng.choice(eval_prompts), rng)
            repo = rng.choice(REPOS)
        else:
            while True:
                repo = rng.choice(REPOS)
                prompt = make_prompt(repo, rng.choice(FUNCS), rng.choice(SYMPTOMS), rng)
                if prompt not in used:
                    used.add(prompt)
                    break
        p = rng.choice([0.1, 0.3, 0.5, 0.7, 0.85, 0.97])
        tasks.append({"id": f"t{t:03d}", "repo": repo, "prompt": prompt, "kind": kind, "p": p,
                      "weak": rng.random() < 0.05})
    traces = []
    for t in tasks:
        for _ in range(rng.randint(3, 10)):
            acts, last = [], ""
            for _ in range(rng.randint(8, 20)):
                v = rng.choice([x for x in VERBS if x != last or rng.random() < 0.3])
                acts.append(f"{v} src/{rng.choice(FUNCS)}.py")
                last = v
            acts.append("submit")
            passed = rng.random() < t["p"]
            defect = {"near_dup": "near_dup", "leak": "leak"}.get(t["kind"], "")
            prompt = t["prompt"]
            r = rng.random()
            if r < 0.06:
                k = rng.randrange(1, len(acts))
                acts = acts[:k] + ["view src/core.py"] * 12 + acts[k:]
                defect, passed = "loop", rng.random() < 0.4
            elif r < 0.11:
                acts.insert(len(acts) - 1, "edit tests/test_core.py")
                defect, passed = "tamper", True
            elif r < 0.15:
                prompt += EMAIL_TAIL
                defect = "pii"
            elif r < 0.19:
                acts = acts[:-1]
                defect = "truncated"
            elif t["weak"] and rng.random() < 0.5:
                defect, passed = "weak_tests", True
            traces.append(Trace(len(traces), t["id"], t["repo"], prompt, acts, passed, rng.random() > 0.12, defect))
    return traces


def longest_run(actions):
    """Length of the longest run of the same first word (the base command, without arguments)."""
    best = run = 0
    prev = None
    for a in actions:
        w = a.split()[0] if a.split() else ""
        run = run + 1 if w == prev else 1
        prev = w
        best = max(best, run)
    return best


_EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
_PHONE = re.compile(r"\b\d{3}[-. ]\d{4}\b")


def has_pii(text):
    return bool(_EMAIL.search(text) or _PHONE.search(text))


def gate_consent(traces, **_):
    return [t for t in traces if t.consent_train]


def gate_resolved(traces, **_):
    return [t for t in traces if t.passed]


def gate_complete(traces, **_):
    return [t for t in traces if t.actions and t.actions[-1] == "submit"]


def gate_loops(traces, max_run=10, **_):
    return [t for t in traces if longest_run(t.actions) < max_run]


def gate_tamper(traces, **_):
    return [t for t in traces if not any(a.startswith("edit tests/") or "sys.exit(0)" in a or "conftest" in a
                                         for a in t.actions)]


def gate_pii(traces, **_):
    return [t for t in traces if not has_pii(t.prompt)]


def gate_leak(traces, eval_prompts=(), leak_threshold=0.6, **_):
    texts = {("e", i): p for i, p in enumerate(eval_prompts)}
    texts.update({("t", t.id): t.prompt for t in traces})
    bad = set()
    for a, b, _j in near_duplicates(texts, leak_threshold):
        if a[0] != b[0]:
            bad.add(a[1] if a[0] == "t" else b[1])
    return [t for t in traces if t.id not in bad]


def gate_dedupe_tasks(traces, dup_threshold=0.6, **_):
    """Drop every trace of a task whose prompt is a near copy of an earlier task's prompt."""
    prompts = {}
    for t in sorted(traces, key=lambda x: x.id):
        prompts.setdefault(t.task_id, t.prompt)
    _kept, dropped = dedupe_keep_first(prompts, dup_threshold)
    return [t for t in traces if t.task_id not in dropped]


def gate_not_trivial(traces, rates=None, max_pass_rate=0.9, **_):
    """Drop tasks that the agent almost always solves. rates is {task id: pass rate over all attempts}."""
    rates = rates or {}
    return [t for t in traces if rates.get(t.task_id, 0.0) <= max_pass_rate]


def gate_cap(traces, cap=3, **_):
    return cap_per_group(sorted(traces, key=lambda t: t.id), lambda t: t.task_id, cap)


GATES = [("consent", gate_consent), ("resolved", gate_resolved), ("complete", gate_complete), ("loops", gate_loops),
         ("tamper", gate_tamper), ("pii", gate_pii), ("leak", gate_leak), ("dedupe tasks", gate_dedupe_tasks),
         ("not trivial", gate_not_trivial), ("cap per task", gate_cap)]


def task_pass_rates(traces):
    by = {}
    for t in traces:
        by.setdefault(t.task_id, []).append(1.0 if t.passed else 0.0)
    return {k: mean(v) for k, v in by.items()}


def run_pipeline(traces, eval_prompts=(), skip=(), **kw):
    """Run the gates in order. Returns {kept, funnel: [(gate, n_in, n_out)]}. skip names gates to leave out."""
    rates = task_pass_rates(traces)
    cur = list(traces)
    funnel = [("pool", len(cur), len(cur))]
    for name, gate in GATES:
        if name in skip:
            continue
        nxt = gate(cur, eval_prompts=eval_prompts, rates=rates, **kw)
        funnel.append((name, len(cur), len(nxt)))
        cur = nxt
    return {"kept": cur, "funnel": funnel}


def defect_report(pool, kept):
    """For each injected defect: how many were in the pool and how many are still in the kept set.
    Also how many clean, passing traces the gates removed. Ground truth is available only because the lab made the data."""
    kept_ids = {t.id for t in kept}
    out = {}
    for d in sorted({t.defect for t in pool if t.defect}):
        ids = [t.id for t in pool if t.defect == d]
        out[d] = {"in_pool": len(ids), "kept": sum(i in kept_ids for i in ids)}
    good = [t.id for t in pool if not t.defect and t.passed and t.consent_train]
    out["clean_passing"] = {"in_pool": len(good), "kept": sum(i in kept_ids for i in good)}
    return out


def wilson_interval(k, n, z=1.96):
    """A 95 percent Wilson interval for a share k out of n."""
    if n == 0:
        return (0.0, 1.0)
    p = k / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (max(0.0, c - h), min(1.0, c + h))


def audit_sample(kept, n, seed=0):
    """A sample for human review: round robin over repos so that no repo is missed."""
    rng = random.Random(seed)
    by = {}
    for t in kept:
        by.setdefault(t.repo, []).append(t)
    for v in by.values():
        rng.shuffle(v)
    out = []
    while len(out) < min(n, len(kept)):
        for r in sorted(by):
            if by[r] and len(out) < n:
                out.append(by[r].pop())
    return sorted(out, key=lambda t: t.id)


def dataset_card(kept, funnel, eval_prompts=()):
    return {"traces": len(kept), "tasks": len({t.task_id for t in kept}), "repos": sorted({t.repo for t in kept}),
            "gates": [f[0] for f in funnel[1:]], "eval_prompts_checked": len(eval_prompts),
            "known_gaps": ["weak tests are not detected by any gate", "gates only catch what they were written for",
                           "all traces and defects here are made up"]}


def artstein_example():
    """The two-coder example from Artstein and Poesio (2008), Table 4: 100 utterances, three labels. Returns (coder A, coder B)."""
    pairs = ([("stat", "stat")] * 46 + [("ireq", "stat")] * 6 + [("ireq", "ireq")] * 32 + [("ireq", "chck")] * 6 +
             [("chck", "chck")] * 10)
    return [p[0] for p in pairs], [p[1] for p in pairs]


def artstein_distance(a, b):
    """The weights used in that paper: check is half way between statement and info request."""
    if a == b:
        return 0.0
    return 0.5 if "chck" in (a, b) else 1.0


def prevalence_demo():
    """Ninety percent agreement and a kappa below zero, because almost every item has the same label."""
    a = ["yes"] * 90 + ["yes"] * 5 + ["no"] * 5
    b = ["yes"] * 90 + ["no"] * 5 + ["yes"] * 5
    return {"agreement": round(percent_agreement(a, b), 3), "kappa": round(cohen_kappa(a, b), 3),
            "alpha": round(krippendorff_alpha([[x, y] for x, y in zip(a, b)]), 3)}


def main():
    print("== 1. informativeness: find the tasks with the most signal (made-up pass probabilities)")
    for r in informativeness_demo():
        print("  attempts per task %3d  overlap with true top 50: %.2f  share with 0<passes<k: %.2f"
              % (r["k"], r["overlap_with_truth"], r["share_0_lt_passes_lt_k"]))
    print("== 2. diversity and dedupe")
    rng = random.Random(1)
    centres = [(0, 0), (8, 0), (0, 8), (8, 8), (4, 4)]
    pts = {i: (centres[0][0] + rng.gauss(0, 1), centres[0][1] + rng.gauss(0, 1)) for i in range(150)}
    for c in centres[1:]:
        for _ in range(10):
            pts[len(pts)] = (c[0] + rng.gauss(0, 0.7), c[1] + rng.gauss(0, 0.7))
    kc = k_center_greedy(pts, 10)
    rr = [covering_radius(pts, random.Random(sd).sample(sorted(pts), 10)) for sd in range(20)]
    print("  covering radius with 10 centres: k-center %.2f, random mean %.2f (20 draws)" % (covering_radius(pts, kc), mean(rr)))
    docs = make_topic_docs(100, seed=3)
    target = [t for _tp, t in make_topic_docs(10, seed=4, topics=["code"]).values()]
    lw = dsir_log_weights({i: d[1] for i, d in docs.items()}, target)
    picked = gumbel_top_k(lw, 60, seed=5)
    print("  importance resampling toward code docs: 60 of 300 picked, share code %.2f (random expects 0.33)"
          % selection_share(docs, picked, "code"))
    pool = make_trace_pool(150, 0)
    prompts = {}
    for t in pool:
        prompts.setdefault(t.task_id, t.prompt)
    print("  prompts %d, near-duplicate pairs at 0.6: %d, groups: %d"
          % (len(prompts), len(near_duplicates(prompts, 0.6)), len(duplicate_groups(prompts, 0.6))))
    print("== 3. gradient similarity proxy, 10 seeds, 20 percent of a 450-example pool")
    rows = [less_demo(sd) for sd in range(10)]
    for name in ["random", "less_cosine", "dot_product", "full_pool"]:
        print("  %-12s target accuracy %.3f   from task A %.1f   from task C %.1f   large-norm %.1f"
              % (name, mean(r[name]["acc_A"] for r in rows), mean(r[name]["from_A"] for r in rows),
                 mean(r[name]["from_C"] for r in rows), mean(r[name]["big"] for r in rows)))
    print("== 4. label noise and data maps")
    print("  confident learning vs lowest self-confidence, 8 seeds:", label_noise_study())
    dm = [data_map_demo(seed=sd) for sd in range(8)]
    print("  data map, share of flipped labels: hard %.2f ambiguous %.2f easy %.2f (base rate %.2f)"
          % (mean(r["share_flipped_in_hard"] for r in dm), mean(r["share_flipped_in_ambiguous"] for r in dm),
             mean(r["share_flipped_in_easy"] for r in dm), mean(r["base_rate"] for r in dm)))
    print("== 5. agreement")
    a, b = artstein_example()
    print("  Artstein and Poesio example: agreement %.2f  pi %.4f  kappa %.4f  weighted kappa %.4f"
          % (percent_agreement(a, b), scott_pi(a, b), cohen_kappa(a, b), weighted_kappa(a, b, artstein_distance)))
    print("  alpha nominal %.4f, alpha with their weights %.4f"
          % (krippendorff_alpha([[x, y] for x, y in zip(a, b)]),
             krippendorff_alpha([[x, y] for x, y in zip(a, b)], artstein_distance)))
    print("  prevalence demo:", prevalence_demo())
    ann, ok = simulate_annotation(2000, 3, 0.7, seed=0)
    for name, ens in (("max of 3", ensemble_max), ("median of 3", ensemble_median)):
        print("  keep rule %-12s" % name, selection_quality(keep_by_severity(ann, ens), ok, len(ann)))
    one = {i: [[c[0]] for c in v] for i, v in ann.items()}
    print("  keep rule %-12s" % "one annotator", selection_quality(keep_by_severity(one, ensemble_max), ok, len(ann)))
    print("  true fine items: %.3f of all" % (len(ok) / len(ann)))
    print("== 6. tiny benchmarks from IRT, 20 made-up worlds, 60 fit models, 30 held-out, 200 items, 24 chosen")
    for dims in (1, 2):
        print("  abilities in %d dimension(s)" % dims)
        res = tiny_benchmark_average(20, n_train=60, n_test=30, n_items=200, k=24, dims=dims)
        for name, r in res.items():
            print("    %-32s tau %.3f (sd %.3f)  mean abs error %.4f" % (name, r["tau"], r["tau_sd"], r["mae"]))
    print("== 7. EvalGen-style alignment")
    print("  paper Table 1 recomputed:", paper_table_alignment())
    for sd in range(3):
        print("  drift demo seed %d:" % sd, criteria_drift_demo(seed=sd))
    print("  sampling policies, 20 worlds, 16 grades each:", sampling_study(worlds=20))
    print("== 8. trace-to-dataset pipeline")
    ev = make_eval_prompts(12, 1)
    res = run_pipeline(pool, ev)
    for g, n_in, n_out in res["funnel"][1:]:
        print("  %-13s %4d -> %4d" % (g, n_in, n_out))
    print("  defects in pool and still kept:", {k: (v["in_pool"], v["kept"]) for k, v in defect_report(pool, res["kept"]).items()})
    skip = run_pipeline(pool, ev, skip=("complete", "tamper"))
    print("  without the complete and tamper gates:", {k: (v["in_pool"], v["kept"]) for k, v in
                                                       defect_report(pool, skip["kept"]).items() if k in ("truncated", "tamper")})
    sample = audit_sample(res["kept"], 40, seed=2)
    weak = sum(t.defect == "weak_tests" for t in sample)
    lo, hi = wilson_interval(weak, len(sample))
    print("  audit of %d kept traces: %d with weak tests, 95 percent interval %.3f to %.3f; true rate %.3f"
          % (len(sample), weak, lo, hi, sum(t.defect == "weak_tests" for t in res["kept"]) / len(res["kept"])))
    print("== 9. data card")
    print(json.dumps(dataset_card(res["kept"], res["funnel"], ev), indent=1, sort_keys=True))


if __name__ == "__main__":
    main()

Run the demo

$ python selection_quality_toolkit.py
== 1. informativeness: find the tasks with the most signal (made-up pass probabilities)
  attempts per task   1  overlap with true top 50: 0.16  share with 0<passes<k: 0.00
  attempts per task   4  overlap with true top 50: 0.32  share with 0<passes<k: 0.58
  attempts per task  16  overlap with true top 50: 0.56  share with 0<passes<k: 0.87
  attempts per task  64  overlap with true top 50: 0.80  share with 0<passes<k: 1.00
== 2. diversity and dedupe
  covering radius with 10 centres: k-center 2.07, random mean 7.45 (20 draws)
  importance resampling toward code docs: 60 of 300 picked, share code 1.00 (random expects 0.33)
  prompts 150, near-duplicate pairs at 0.6: 12, groups: 12
== 3. gradient similarity proxy, 10 seeds, 20 percent of a 450-example pool
  random       target accuracy 0.528   from task A 29.1   from task C 29.8   large-norm 17.7
  less_cosine  target accuracy 0.695   from task A 52.4   from task C 14.2   large-norm 18.1
  dot_product  target accuracy 0.645   from task A 51.7   from task C 15.6   large-norm 24.7
  full_pool    target accuracy 0.540   from task A 150.0   from task C 150.0   large-norm 91.4
== 4. label noise and data maps
  confident learning vs lowest self-confidence, 8 seeds: {'flipped': 31.375, 'cl_flagged': 36.25, 'cl_precision': 0.793, 'cl_recall': 0.918, 'base_precision': 0.772, 'base_recall': 0.894}
  data map, share of flipped labels: hard 0.50 ambiguous 0.08 easy 0.00 (base rate 0.09)
== 5. agreement
  Artstein and Poesio example: agreement 0.88  pi 0.7995  kappa 0.8013  weighted kappa 0.8163
  alpha nominal 0.8005, alpha with their weights 0.8156
  prevalence demo: {'agreement': 0.9, 'kappa': -0.053, 'alpha': -0.047}
  keep rule max of 3     {'kept_share': 0.351, 'precision': 0.999, 'recall': 0.723}
  keep rule median of 3  {'kept_share': 0.477, 'precision': 0.97, 'recall': 0.954}
  keep rule one annotator {'kept_share': 0.469, 'precision': 0.925, 'recall': 0.895}
  true fine items: 0.485 of all
== 6. tiny benchmarks from IRT, 20 made-up worlds, 60 fit models, 30 held-out, 200 items, 24 chosen
  abilities in 1 dimension(s)
    random, mean                     tau 0.761 (sd 0.069)  mean abs error 0.0719
    stratified by difficulty, mean   tau 0.776 (sd 0.058)  mean abs error 0.0611
    anchor points, weighted mean     tau 0.626 (sd 0.068)  mean abs error 0.1017
    random, 2PL ability              tau 0.735 (sd 0.065)  mean abs error 0.0618
    IRT-chosen, 2PL ability          tau 0.793 (sd 0.043)  mean abs error 0.0514
    IRT-chosen, 1PL ability          tau 0.778 (sd 0.058)  mean abs error 0.0562
  abilities in 2 dimension(s)
    random, mean                     tau 0.751 (sd 0.075)  mean abs error 0.0740
    stratified by difficulty, mean   tau 0.767 (sd 0.053)  mean abs error 0.0649
    anchor points, weighted mean     tau 0.643 (sd 0.099)  mean abs error 0.0933
    random, 2PL ability              tau 0.731 (sd 0.079)  mean abs error 0.0634
    IRT-chosen, 2PL ability          tau 0.822 (sd 0.040)  mean abs error 0.0507
    IRT-chosen, 1PL ability          tau 0.776 (sd 0.070)  mean abs error 0.0571
== 7. EvalGen-style alignment
  paper Table 1 recomputed: {'medical EvalGen and SPADE': 48.29, 'product EvalGen': 66.46, 'product SPADE': 54.35}
  drift demo seed 0: {'graded': 20, 'picked_early': ['no_hashtag', 'proper>=0.6', 'words<=50'], 'picked_after_drift': ['proper>=0.6', 'words<=50'], 'early_set_vs_strict_grades': 0.777, 'early_set_vs_relaxed_grades': 0.627, 'regraded_set_vs_relaxed_grades': 0.744}
  drift demo seed 1: {'graded': 20, 'picked_early': ['proper>=1.0', 'words<=60'], 'picked_after_drift': ['proper>=0.6', 'words<=60'], 'early_set_vs_strict_grades': 1.0, 'early_set_vs_relaxed_grades': 0.672, 'regraded_set_vs_relaxed_grades': 0.893}
  drift demo seed 2: {'graded': 20, 'picked_early': ['no_hashtag', 'proper>=1.0', 'words<=40'], 'picked_after_drift': ['no_hashtag', 'proper>=0.5', 'words<=60'], 'early_set_vs_strict_grades': 0.656, 'early_set_vs_relaxed_grades': 0.388, 'regraded_set_vs_relaxed_grades': 0.863}
  sampling policies, 20 worlds, 16 grades each: {'random': {'mean': 0.878, 'sd': 0.144}, 'highest': {'mean': 0.551, 'sd': 0.105}, 'lowest': {'mean': 0.712, 'sd': 0.084}, 'alternating': {'mean': 0.574, 'sd': 0.069}}
== 8. trace-to-dataset pipeline
  consent       1008 ->  902
  resolved       902 ->  500
  complete       500 ->  482
  loops          482 ->  461
  tamper         461 ->  417
  pii            417 ->  402
  leak           402 ->  376
  dedupe tasks   376 ->  358
  not trivial    358 ->  264
  cap per task   264 ->  201
  defects in pool and still kept: {'leak': (50, 0), 'loop': (62, 0), 'near_dup': (59, 10), 'pii': (32, 0), 'tamper': (51, 0), 'truncated': (34, 0), 'weak_tests': (13, 3), 'clean_passing': (334, 188)}
  without the complete and tamper gates: {'tamper': (51, 24), 'truncated': (34, 11)}
  audit of 40 kept traces: 0 with weak tests, 95 percent interval 0.000 to 0.088; true rate 0.015
== 9. data card
{
 "eval_prompts_checked": 12,
 "gates": [
  "consent",
  "resolved",
  "complete",
  "loops",
  "tamper",
  "pii",
  "leak",
  "dedupe tasks",
  "not trivial",
  "cap per task"
 ],
 "known_gaps": [
  "weak tests are not detected by any gate",
  "gates only catch what they were written for",
  "all traces and defects here are made up"
 ],
 "repos": [
  "libalpha",
  "libbeta",
  "libdelta",
  "libgamma",
  "libomega",
  "libsigma"
 ],
 "tasks": 89,
 "traces": 201
}

Exercises

1. Attempts and cost. Run informativeness_demo with ks=(2, 3, 5, 8, 12) and plot overlap against attempts. Fit a line through the log of the attempts. How many attempts would you need for 90% overlap? Why is the extrapolation risky [1]?
2. Break the difficulty rule. Add flaky tasks to make_task_pool whose outcome is a fair coin, whatever the ability. How many land in the top 50 by pass_variance? Add a check that separates flaky tasks from tasks that are truly at 50%, using more attempts. What does it cost?
3. Outliers. Add 5 outliers far from the data and run k_center_greedy. How many of the first 10 picks are outliers? Add a rule to ignore points that are far from all others, and say what it hides.
4. The banding curve. Try 8 bands of 8 rows and 32 bands of 2 rows. Predict from the formula how often a pair of Jaccard 0.6 is a candidate, then measure it on 200 made pairs. Report the false candidate rate at Jaccard 0.2 too [18].
5. Order of gates. Run run_pipeline with the dedupe gate first, and then last. Count the kept traces and the kept near-copies. Which order would you choose, and why?
6. Cosine against dot product. In make_less_world, raise big_frac to 0.5 and big_scale to 8. How does the gap between cosine and dot product change? Explain it with the paper's reason for using cosine [6].
7. Warmup. Change warmup_frac to 0.01, 0.05 and 0.5. Does the choice of examples change? Why does LESS need a warmup at all [6]?
8. A projection. Run less_demo with proj_dim=2, 4. How much of the accuracy remains? The paper tried projection sizes from 1024 to 8192 and found that every size beat random selection, with larger sizes doing a little better [6]. Your toy is much smaller, so compare the pattern, not the sizes.
9. Hide the noise. In label_noise_study, flip labels in one direction only, for example always from class 0 to class 1. Does confident learning still find them? Does the self-confidence baseline? What does a systematic error do to both [17]?
10. Agreement tables. Make three tables of two coders, with 80% agreement each: balanced labels, 90% one label, and one coder with a bias. Compute pi, kappa and alpha. Which one moves most, and why [22]?
11. Keep rules. In simulate_annotation, change the accuracy to 0.9 and to 0.5. At what accuracy does the max-of-3 rule stop being worth its recall loss, if a bad item costs five times a lost good item [23]?
12. Tiny benchmark. Run tiny_benchmark_average with k=12 and k=48. Where does IRT-chosen stop beating stratified? Then change the held-out models to be much stronger than the fit models and see what happens [25].
13. Grading policies. In sampling_study, add a policy that grades the outputs at the median score. Compare it with the others. Does it match the paper's remark that the middle is not uncertainty sampling [2]?
14. Add a defect. Add a new defect kind to make_trace_pool (for example, a trace that prints the answer from memory without running anything). Write a gate for it, with three tests. How many does it catch, and what does it wrongly drop?
15. Audit size. For true defect rates 1%, 3% and 10%, compute how often an audit of 40 sees zero cases, using audit_sample over many seeds. Compare with 0.99⁴⁰, 0.97⁴⁰ and 0.9⁴⁰ (my arithmetic in 2.10).

The tests: test_selection_quality_toolkit.py

The 197 tests check the shared helpers (hashing, words, sigmoid, logit, cosine, overlap); the informativeness scores, ranking and caps; the made-up task pool; farthest-first, the covering radius and score-first diversity; shingles, MinHash, banding, near duplicates and groups, and de-duplication; hashed features, bucket probabilities, the importance weights and the Gumbel draw; the logistic gradient, training and the projection; the gradient-similarity world and scores; the blobs, label flips, centroid probabilities, cross-validation, class thresholds, the confident joint and the issue finder; training dynamics and data-map regions; every agreement coefficient, including the published numbers of Artstein and Poesio; the ensemble and keep rules and the annotation simulation; Kendall's tau-b; the IRT world, the fit, the ability estimate, Fisher information and each item selection; the coverage, false failure rate and alignment formulas, including the paper's Table 1; the grading policies, the per-criterion choice and the drift demo; the trace generator, each gate, the pipeline, the defect report, the Wilson interval, the audit and the data card; and the demo itself. Writing the demo found real faults in my first design, which I fixed: an "alternating" grading policy that graded only extreme examples, so every assertion tied and no drift showed; a prompt template that made unrelated tasks near-duplicates, so the pipeline dropped nearly everything; a loop-defect generator that raised an error; and a gradient toy that was too noisy to show any selection effect. I also ran a mutation check: I also ran a mutation check, in which I made small deliberate changes to the code, one at a time, and ran the tests on each. The first run caught 265 of 326 changes. I dropped 7 of the 61 that were missed, because they cannot change what the code returns: a different test inside the sigmoid that returns the same value; the largest hash in place of the smallest in MinHash (the same estimator on a reversed order); the band number in the bucket key (two different hash functions would have to give the same 61-bit value); the choice of root and the path compression in the union-find (the same groups come out); and two guards in Krippendorff's alpha (a unit with one value adds nothing, and the total weight is either 0 or at least 2). For the other 54 I wrote 34 new tests. Some check what the code means, and some pin exact values from the made-up worlds, so they would also flag an intended change. Then I ran all 319 changes again. A change that makes the tests run for more than four minutes counts as caught. In the last run the tests caught every change. That says the tests are sensitive. It does not say the code is correct.

$ python test_selection_quality_toolkit.py -v
test_accuracy_counts_a_tie_as_class_one (__main__.TestAddedAfterMutationRun.test_accuracy_counts_a_tie_as_class_one) ... ok
test_alpha_weights_units_with_three_ratings (__main__.TestAddedAfterMutationRun.test_alpha_weights_units_with_three_ratings) ... ok
test_anchor_points_on_a_small_response_matrix (__main__.TestAddedAfterMutationRun.test_anchor_points_on_a_small_response_matrix) ... ok
test_banding_rate_matches_the_formula (__main__.TestAddedAfterMutationRun.test_banding_rate_matches_the_formula) ... ok
test_baseline_flags_as_many_as_confident_learning (__main__.TestAddedAfterMutationRun.test_baseline_flags_as_many_as_confident_learning) ... ok
test_centroid_probs_gives_a_missing_class_the_origin (__main__.TestAddedAfterMutationRun.test_centroid_probs_gives_a_missing_class_the_origin) ... ok
test_centroid_probs_uses_tau_squared_and_is_stable (__main__.TestAddedAfterMutationRun.test_centroid_probs_uses_tau_squared_and_is_stable) ... ok
test_drift_demo_and_sampling_study_pinned_values (__main__.TestAddedAfterMutationRun.test_drift_demo_and_sampling_study_pinned_values) ... ok
test_drift_demo_uses_exactly_n_grade_graded_outputs (__main__.TestAddedAfterMutationRun.test_drift_demo_uses_exactly_n_grade_graded_outputs) ... ok
test_dsir_weight_counts_repeated_features (__main__.TestAddedAfterMutationRun.test_dsir_weight_counts_repeated_features) ... ok
test_duplicate_prompts_are_redrawn (__main__.TestAddedAfterMutationRun.test_duplicate_prompts_are_redrawn) ... ok
test_easy_region_never_overlaps_the_hard_region (__main__.TestAddedAfterMutationRun.test_easy_region_never_overlaps_the_hard_region) ... ok
test_email_pattern_accepts_a_plus_or_dash_before_the_at_sign (__main__.TestAddedAfterMutationRun.test_email_pattern_accepts_a_plus_or_dash_before_the_at_sign) ... ok
test_fit_ends_where_the_penalised_gradient_is_zero (__main__.TestAddedAfterMutationRun.test_fit_ends_where_the_penalised_gradient_is_zero) ... ok
test_fit_starts_from_accuracy_logits (__main__.TestAddedAfterMutationRun.test_fit_starts_from_accuracy_logits) ... ok
test_fit_theta_is_a_map_estimate (__main__.TestAddedAfterMutationRun.test_fit_theta_is_a_map_estimate) ... ok
test_gumbel_draw_keeps_a_low_weight_item_rare_but_not_never (__main__.TestAddedAfterMutationRun.test_gumbel_draw_keeps_a_low_weight_item_rare_but_not_never) ... ok
test_gumbel_top_one_follows_softmax_of_log_weights (__main__.TestAddedAfterMutationRun.test_gumbel_top_one_follows_softmax_of_log_weights) ... ok
test_history_correctness_agrees_with_the_given_label_probability (__main__.TestAddedAfterMutationRun.test_history_correctness_agrees_with_the_given_label_probability) ... ok
test_irt_grid_breaks_ties_toward_the_lower_id (__main__.TestAddedAfterMutationRun.test_irt_grid_breaks_ties_toward_the_lower_id) ... ok
test_irt_world_in_two_dimensions_loads_items_on_both_skills (__main__.TestAddedAfterMutationRun.test_irt_world_in_two_dimensions_loads_items_on_both_skills) ... ok
test_label_noise_demo_and_study_pinned_values (__main__.TestAddedAfterMutationRun.test_label_noise_demo_and_study_pinned_values) ... ok
test_leak_tasks_are_reworded_eval_prompts (__main__.TestAddedAfterMutationRun.test_leak_tasks_are_reworded_eval_prompts) ... ok
test_less_demo_pinned_values_for_seed_zero (__main__.TestAddedAfterMutationRun.test_less_demo_pinned_values_for_seed_zero) ... ok
test_less_world_noise_and_big_norms_only_in_the_pool (__main__.TestAddedAfterMutationRun.test_less_world_noise_and_big_norms_only_in_the_pool) ... ok
test_near_duplicate_tasks_copy_a_plain_task_with_one_word_changed (__main__.TestAddedAfterMutationRun.test_near_duplicate_tasks_copy_a_plain_task_with_one_word_changed) ... ok
test_pipeline_computes_pass_rates_before_any_gate (__main__.TestAddedAfterMutationRun.test_pipeline_computes_pass_rates_before_any_gate) ... ok
test_pool_pass_rates_and_loop_outcomes (__main__.TestAddedAfterMutationRun.test_pool_pass_rates_and_loop_outcomes) ... ok
test_prompt_has_the_worked_example_and_fixed_numbers_without_rng (__main__.TestAddedAfterMutationRun.test_prompt_has_the_worked_example_and_fixed_numbers_without_rng) ... ok
test_reword_changes_one_middle_word (__main__.TestAddedAfterMutationRun.test_reword_changes_one_middle_word) ... ok
test_stratified_selection_takes_evenly_spaced_ranks (__main__.TestAddedAfterMutationRun.test_stratified_selection_takes_evenly_spaced_ranks) ... ok
test_task_dedupe_uses_the_first_trace_by_id (__main__.TestAddedAfterMutationRun.test_task_dedupe_uses_the_first_trace_by_id) ... ok
test_tau_b_with_ties_on_both_sides (__main__.TestAddedAfterMutationRun.test_tau_b_with_ties_on_both_sides) ... ok
test_tiny_benchmark_study_pinned_values_for_seed_zero (__main__.TestAddedAfterMutationRun.test_tiny_benchmark_study_pinned_values_for_seed_zero) ... ok
test_alpha_by_hand (__main__.TestAgreement.test_alpha_by_hand) ... ok
test_alpha_is_symmetric_in_rater_order (__main__.TestAgreement.test_alpha_is_symmetric_in_rater_order) ... ok
test_alpha_with_unequal_numbers_of_raters (__main__.TestAgreement.test_alpha_with_unequal_numbers_of_raters) ... ok
test_artstein_and_poesio_numbers (__main__.TestAgreement.test_artstein_and_poesio_numbers) ... ok
test_artstein_distance (__main__.TestAgreement.test_artstein_distance) ... ok
test_cohen_kappa_by_hand (__main__.TestAgreement.test_cohen_kappa_by_hand) ... ok
test_distances (__main__.TestAgreement.test_distances) ... ok
test_ensembles (__main__.TestAgreement.test_ensembles) ... ok
test_fleiss_by_hand (__main__.TestAgreement.test_fleiss_by_hand) ... ok
test_fleiss_equals_scott_for_two_raters (__main__.TestAgreement.test_fleiss_equals_scott_for_two_raters) ... ok
test_fleiss_validates_rater_counts (__main__.TestAgreement.test_fleiss_validates_rater_counts) ... ok
test_keep_by_severity (__main__.TestAgreement.test_keep_by_severity) ... ok
test_max_rule_trades_recall_for_precision (__main__.TestAgreement.test_max_rule_trades_recall_for_precision) ... ok
test_percent_agreement (__main__.TestAgreement.test_percent_agreement) ... ok
test_prevalence_demo (__main__.TestAgreement.test_prevalence_demo) ... ok
test_scott_pi_by_hand (__main__.TestAgreement.test_scott_pi_by_hand) ... ok
test_selection_quality (__main__.TestAgreement.test_selection_quality) ... ok
test_simulate_annotation (__main__.TestAgreement.test_simulate_annotation) ... ok
test_weighted_kappa_by_hand (__main__.TestAgreement.test_weighted_kappa_by_hand) ... ok
test_weighted_with_zero_one_distance_is_cohen (__main__.TestAgreement.test_weighted_with_zero_one_distance_is_cohen) ... ok
test_main_prints_nine_sections (__main__.TestDemo.test_main_prints_nine_sections) ... ok
test_bucket_probs_sum_to_one_and_smooth (__main__.TestDiversity.test_bucket_probs_sum_to_one_and_smooth) ... ok
test_covering_radius (__main__.TestDiversity.test_covering_radius) ... ok
test_dsir_prefers_documents_like_the_target (__main__.TestDiversity.test_dsir_prefers_documents_like_the_target) ... ok
test_duplicate_groups_and_dedupe (__main__.TestDiversity.test_duplicate_groups_and_dedupe) ... ok
test_duplicate_groups_chain_through_union (__main__.TestDiversity.test_duplicate_groups_chain_through_union) ... ok
test_gumbel_top_k (__main__.TestDiversity.test_gumbel_top_k) ... ok
test_gumbel_top_k_follows_the_weights (__main__.TestDiversity.test_gumbel_top_k_follows_the_weights) ... ok
test_hashed_features_count_unigrams_and_bigrams (__main__.TestDiversity.test_hashed_features_count_unigrams_and_bigrams) ... ok
test_k_center_beats_random_on_clustered_points (__main__.TestDiversity.test_k_center_beats_random_on_clustered_points) ... ok
test_k_center_start_argument_and_limits (__main__.TestDiversity.test_k_center_start_argument_and_limits) ... ok
test_k_center_starts_at_smallest_id_then_goes_far (__main__.TestDiversity.test_k_center_starts_at_smallest_id_then_goes_far) ... ok
test_lsh_candidates_include_true_pairs_and_are_sorted (__main__.TestDiversity.test_lsh_candidates_include_true_pairs_and_are_sorted) ... ok
test_lsh_requires_bands_that_divide (__main__.TestDiversity.test_lsh_requires_bands_that_divide) ... ok
test_minhash_agreement_estimates_jaccard (__main__.TestDiversity.test_minhash_agreement_estimates_jaccard) ... ok
test_minhash_shape_and_determinism (__main__.TestDiversity.test_minhash_shape_and_determinism) ... ok
test_near_duplicates_threshold (__main__.TestDiversity.test_near_duplicates_threshold) ... ok
test_score_first_diverse (__main__.TestDiversity.test_score_first_diverse) ... ok
test_score_first_diverse_distance_is_inclusive_and_ties_go_to_small_id (__main__.TestDiversity.test_score_first_diverse_distance_is_inclusive_and_ties_go_to_small_id) ... ok
test_shingles_and_jaccard (__main__.TestDiversity.test_shingles_and_jaccard) ... ok
test_threshold_is_inclusive (__main__.TestDiversity.test_threshold_is_inclusive) ... ok
test_topic_docs (__main__.TestDiversity.test_topic_docs) ... ok
test_union_find_keeps_smallest_id_as_root (__main__.TestDiversity.test_union_find_keeps_smallest_id_as_root) ... ok
test_alternating_has_no_repeats (__main__.TestEvalGen.test_alternating_has_no_repeats) ... ok
test_candidates (__main__.TestEvalGen.test_candidates) ... ok
test_coverage_and_ffr_edge_cases (__main__.TestEvalGen.test_coverage_and_ffr_edge_cases) ... ok
test_coverage_ffr_alignment_by_hand (__main__.TestEvalGen.test_coverage_ffr_alignment_by_hand) ... ok
test_drift_demo (__main__.TestEvalGen.test_drift_demo) ... ok
test_drift_demo_uses_the_policy (__main__.TestEvalGen.test_drift_demo_uses_the_policy) ... ok
test_ffr_limit_is_inclusive (__main__.TestEvalGen.test_ffr_limit_is_inclusive) ... ok
test_grading_order (__main__.TestEvalGen.test_grading_order) ... ok
test_outputs_and_rules (__main__.TestEvalGen.test_outputs_and_rules) ... ok
test_paper_table_one (__main__.TestEvalGen.test_paper_table_one) ... ok
test_sampling_study (__main__.TestEvalGen.test_sampling_study) ... ok
test_select_per_criterion (__main__.TestEvalGen.test_select_per_criterion) ... ok
test_select_per_criterion_ties_go_to_the_name_that_sorts_first (__main__.TestEvalGen.test_select_per_criterion_ties_go_to_the_name_that_sorts_first) ... ok
test_selectivity_and_failure_score (__main__.TestEvalGen.test_selectivity_and_failure_score) ... ok
test_set_alignment (__main__.TestEvalGen.test_set_alignment) ... ok
test_set_fails_is_any (__main__.TestEvalGen.test_set_fails_is_any) ... ok
test_accuracy_threshold (__main__.TestGradientSimilarity.test_accuracy_threshold) ... ok
test_dot_product_prefers_large_norm_examples (__main__.TestGradientSimilarity.test_dot_product_prefers_large_norm_examples) ... ok
test_label_noise_flips_some_labels (__main__.TestGradientSimilarity.test_label_noise_flips_some_labels) ... ok
test_less_demo_fraction (__main__.TestGradientSimilarity.test_less_demo_fraction) ... ok
test_less_demo_selects_more_target_examples_than_random (__main__.TestGradientSimilarity.test_less_demo_selects_more_target_examples_than_random) ... ok
test_less_demo_with_projection_runs (__main__.TestGradientSimilarity.test_less_demo_with_projection_runs) ... ok
test_less_scores_prefer_aligned_gradients (__main__.TestGradientSimilarity.test_less_scores_prefer_aligned_gradients) ... ok
test_less_scores_use_mean_target_gradient (__main__.TestGradientSimilarity.test_less_scores_use_mean_target_gradient) ... ok
test_less_scores_with_projection (__main__.TestGradientSimilarity.test_less_scores_with_projection) ... ok
test_logistic_p_and_grad (__main__.TestGradientSimilarity.test_logistic_p_and_grad) ... ok
test_projection_keeps_cosines_roughly (__main__.TestGradientSimilarity.test_projection_keeps_cosines_roughly) ... ok
test_train_logistic_divides_by_n (__main__.TestGradientSimilarity.test_train_logistic_divides_by_n) ... ok
test_train_logistic_learns_a_separable_rule (__main__.TestGradientSimilarity.test_train_logistic_learns_a_separable_rule) ... ok
test_train_logistic_starts_from_w0_and_zero_steps_returns_it (__main__.TestGradientSimilarity.test_train_logistic_starts_from_w0_and_zero_steps_returns_it) ... ok
test_world_directions_and_norms (__main__.TestGradientSimilarity.test_world_directions_and_norms) ... ok
test_world_shape_and_labels (__main__.TestGradientSimilarity.test_world_shape_and_labels) ... ok
test_dot_norm_cosine_euclid (__main__.TestHelpers.test_dot_norm_cosine_euclid) ... ok
test_logit_inverts_sigmoid_and_clips (__main__.TestHelpers.test_logit_inverts_sigmoid_and_clips) ... ok
test_mean_and_stdev (__main__.TestHelpers.test_mean_and_stdev) ... ok
test_overlap_is_a_share_of_the_first (__main__.TestHelpers.test_overlap_is_a_share_of_the_first) ... ok
test_sigmoid_both_signs (__main__.TestHelpers.test_sigmoid_both_signs) ... ok
test_stable_hash_is_stable_and_bits_matter (__main__.TestHelpers.test_stable_hash_is_stable_and_bits_matter) ... ok
test_words_lowercases_and_splits (__main__.TestHelpers.test_words_lowercases_and_splits) ... ok
test_average_matches_the_mean_of_studies (__main__.TestIRT.test_average_matches_the_mean_of_studies) ... ok
test_estimate_pirt (__main__.TestIRT.test_estimate_pirt) ... ok
test_fisher_information (__main__.TestIRT.test_fisher_information) ... ok
test_fit_orders_models_and_items_by_accuracy (__main__.TestIRT.test_fit_orders_models_and_items_by_accuracy) ... ok
test_fit_recovers_abilities_and_difficulties (__main__.TestIRT.test_fit_recovers_abilities_and_difficulties) ... ok
test_fit_theta (__main__.TestIRT.test_fit_theta) ... ok
test_fit_theta_uses_alpha_and_beta (__main__.TestIRT.test_fit_theta_uses_alpha_and_beta) ... ok
test_fit_theta_uses_only_the_given_items (__main__.TestIRT.test_fit_theta_uses_only_the_given_items) ... ok
test_full_benchmark_selection_is_exact (__main__.TestIRT.test_full_benchmark_selection_is_exact) ... ok
test_irt_grid_does_not_repeat_items (__main__.TestIRT.test_irt_grid_does_not_repeat_items) ... ok
test_irt_selection_beats_random_on_error (__main__.TestIRT.test_irt_selection_beats_random_on_error) ... ok
test_kendall_tau_b (__main__.TestIRT.test_kendall_tau_b) ... ok
test_one_parameter_model_fixes_alpha (__main__.TestIRT.test_one_parameter_model_fixes_alpha) ... ok
test_pearson (__main__.TestIRT.test_pearson) ... ok
test_select_anchors_by_hand (__main__.TestIRT.test_select_anchors_by_hand) ... ok
test_select_irt_grid (__main__.TestIRT.test_select_irt_grid) ... ok
test_select_random (__main__.TestIRT.test_select_random) ... ok
test_select_stratified (__main__.TestIRT.test_select_stratified) ... ok
test_stronger_models_get_more_right (__main__.TestIRT.test_stronger_models_get_more_right) ... ok
test_study_structure_and_determinism (__main__.TestIRT.test_study_structure_and_determinism) ... ok
test_weighted_accuracy (__main__.TestIRT.test_weighted_accuracy) ... ok
test_world_shapes (__main__.TestIRT.test_world_shapes) ... ok
test_world_two_dimensions (__main__.TestIRT.test_world_two_dimensions) ... ok
test_cap_per_group_keeps_first_in_order (__main__.TestInformativeness.test_cap_per_group_keeps_first_in_order) ... ok
test_demo_more_attempts_find_more_of_the_true_top (__main__.TestInformativeness.test_demo_more_attempts_find_more_of_the_true_top) ... ok
test_difficulty_and_sometimes_passes (__main__.TestInformativeness.test_difficulty_and_sometimes_passes) ... ok
test_entropy (__main__.TestInformativeness.test_entropy) ... ok
test_least_confidence_and_margin (__main__.TestInformativeness.test_least_confidence_and_margin) ... ok
test_pass_probability_uses_ability_minus_difficulty (__main__.TestInformativeness.test_pass_probability_uses_ability_minus_difficulty) ... ok
test_pass_rate_and_variance (__main__.TestInformativeness.test_pass_rate_and_variance) ... ok
test_rank_ids_orders_and_breaks_ties_by_id (__main__.TestInformativeness.test_rank_ids_orders_and_breaks_ties_by_id) ... ok
test_sample_passes_range_seed_and_extremes (__main__.TestInformativeness.test_sample_passes_range_seed_and_extremes) ... ok
test_sample_passes_rate_matches_probability (__main__.TestInformativeness.test_sample_passes_rate_matches_probability) ... ok
test_task_pool_is_seeded_and_has_the_right_shape (__main__.TestInformativeness.test_task_pool_is_seeded_and_has_the_right_shape) ... ok
test_true_signal_peaks_where_p_is_half (__main__.TestInformativeness.test_true_signal_peaks_where_p_is_half) ... ok
test_vote_entropy_and_disagreement (__main__.TestInformativeness.test_vote_entropy_and_disagreement) ... ok
test_blob_spreads (__main__.TestLabelNoise.test_blob_spreads) ... ok
test_blobs (__main__.TestLabelNoise.test_blobs) ... ok
test_centroid_probs (__main__.TestLabelNoise.test_centroid_probs) ... ok
test_centroid_probs_missing_class (__main__.TestLabelNoise.test_centroid_probs_missing_class) ... ok
test_class_thresholds (__main__.TestLabelNoise.test_class_thresholds) ... ok
test_confident_joint_by_hand (__main__.TestLabelNoise.test_confident_joint_by_hand) ... ok
test_confident_joint_prefers_the_larger_probability_and_lower_class_on_ties (__main__.TestLabelNoise.test_confident_joint_prefers_the_larger_probability_and_lower_class_on_ties) ... ok
test_confident_joint_skips_unconfident_examples (__main__.TestLabelNoise.test_confident_joint_skips_unconfident_examples) ... ok
test_cross_val_probs_are_out_of_sample (__main__.TestLabelNoise.test_cross_val_probs_are_out_of_sample) ... ok
test_data_map_hard_region_is_full_of_flipped_labels (__main__.TestLabelNoise.test_data_map_hard_region_is_full_of_flipped_labels) ... ok
test_data_map_regions (__main__.TestLabelNoise.test_data_map_regions) ... ok
test_find_label_issues_orders_by_self_confidence (__main__.TestLabelNoise.test_find_label_issues_orders_by_self_confidence) ... ok
test_flip_labels (__main__.TestLabelNoise.test_flip_labels) ... ok
test_history_probability_is_of_the_given_label (__main__.TestLabelNoise.test_history_probability_is_of_the_given_label) ... ok
test_history_shapes_and_learning (__main__.TestLabelNoise.test_history_shapes_and_learning) ... ok
test_noise_demo_finds_most_flips (__main__.TestLabelNoise.test_noise_demo_finds_most_flips) ... ok
test_noise_study_averages (__main__.TestLabelNoise.test_noise_study_averages) ... ok
test_precision_recall (__main__.TestLabelNoise.test_precision_recall) ... ok
test_self_confidence_rank (__main__.TestLabelNoise.test_self_confidence_rank) ... ok
test_training_dynamics (__main__.TestLabelNoise.test_training_dynamics) ... ok
test_audit_sample_covers_repos (__main__.TestPipeline.test_audit_sample_covers_repos) ... ok
test_consent_rate (__main__.TestPipeline.test_consent_rate) ... ok
test_dataset_card (__main__.TestPipeline.test_dataset_card) ... ok
test_defect_report_counts (__main__.TestPipeline.test_defect_report_counts) ... ok
test_defect_traces_look_like_their_defect (__main__.TestPipeline.test_defect_traces_look_like_their_defect) ... ok
test_eval_prompts (__main__.TestPipeline.test_eval_prompts) ... ok
test_gate_cap (__main__.TestPipeline.test_gate_cap) ... ok
test_gate_dedupe_tasks (__main__.TestPipeline.test_gate_dedupe_tasks) ... ok
test_gate_dedupe_uses_the_first_trace_by_id (__main__.TestPipeline.test_gate_dedupe_uses_the_first_trace_by_id) ... ok
test_gate_leak (__main__.TestPipeline.test_gate_leak) ... ok
test_gate_leak_ignores_copies_among_traces (__main__.TestPipeline.test_gate_leak_ignores_copies_among_traces) ... ok
test_gate_not_trivial (__main__.TestPipeline.test_gate_not_trivial) ... ok
test_gates_catch_what_they_target (__main__.TestPipeline.test_gates_catch_what_they_target) ... ok
test_gates_one_by_one (__main__.TestPipeline.test_gates_one_by_one) ... ok
test_has_pii (__main__.TestPipeline.test_has_pii) ... ok
test_longest_run (__main__.TestPipeline.test_longest_run) ... ok
test_make_prompt (__main__.TestPipeline.test_make_prompt) ... ok
test_pool_eval_prompts_and_leaks (__main__.TestPipeline.test_pool_eval_prompts_and_leaks) ... ok
test_pool_has_every_defect_and_is_seeded (__main__.TestPipeline.test_pool_has_every_defect_and_is_seeded) ... ok
test_reword_changes_one_word (__main__.TestPipeline.test_reword_changes_one_word) ... ok
test_run_pipeline_funnel (__main__.TestPipeline.test_run_pipeline_funnel) ... ok
test_run_pipeline_skip_and_cap_option (__main__.TestPipeline.test_run_pipeline_skip_and_cap_option) ... ok
test_task_pass_rates (__main__.TestPipeline.test_task_pass_rates) ... ok
test_trace_defaults (__main__.TestPipeline.test_trace_defaults) ... ok
test_wilson_interval (__main__.TestPipeline.test_wilson_interval) ... ok

----------------------------------------------------------------------
Ran 197 tests in 8.490s

OK

5. Discussion questions

Use these in a study group or a class. Try to answer with an example and a source. Give each about 5 to 10 minutes.

1. LIMA used 1,000 examples, and SWE-smith used 5,016 trajectories from expert runs on 8,686 of its 50 thousand tasks. For a project you care about, would you expect the LIMA pattern or the SWE-smith pattern? What would you test first [1] [3]?
2. Take the five questions in 2.1 (correct, informative, different, on target, allowed). Pick a dataset you know. Which question is the weakest part of it today, and what is the cheapest check you could run?
3. The "0 < passes < k" filter keeps tasks the agent sometimes solves. SWE-smith found that difficulty did not predict training value. Give two reasons the filter could still be useful, and two reasons it could mislead [1].
4. Selection by "most surprising" can pick up wrong labels. How would you keep the useful hard examples and drop the wrong ones? What would you read by hand first [15] [17]?
5. LESS needs a few examples of the target skill. Where would you get them for an agent that books travel? What goes wrong if the examples are unusual [6]?
6. De-duplicate before the other gates, or after? Describe a dataset where each order is right [18].
7. The SWE-bench Verified team took the most severe of three labels and said it is probably overzealous. Describe a case where you would take the median instead, and what you would tell users of the dataset [23].
8. Two coders agree on 90% of items and kappa is below zero. What do you do next? What would you report in a dataset card [22]?
9. A tiny benchmark of 24 items gives a tau of about 0.8 in the lab. Is that good enough to pick between two models? What decisions would you leave to the full benchmark [25] [27]?
10. The trace gates in 2.8 catch only what they were written for. Name a defect in agent traces that none of them catches. How would you find it, and how many traces would you have to read [1]?
11. Criteria drift says you learn what you mean while you grade. Design a grading process for a team of three that handles it: rounds, dates, re-grading, and who decides when the criteria change [2].
12. SWE-smith's validity check shows that a task breaks tests. The SWE-bench Verified annotators asked if the issue text was enough and if the tests were fair. Which of these would you add to a synthetic pipeline, and how would you do it cheaply [1] [23]?

6. Quiz (10 questions)

Pick one answer for each question, then press Check answers. Check answers 0 of 10 answered.

1. What did the LIMA authors find in their ablations on a 7B model [3]?
   • Doubling the number of training examples gave steady gains in answer quality, while more varied prompts and cleaner answers made little difference at all
   • Every change they tried made no difference, because the 1,000 examples had already taught the model all of the knowledge that it needed to give good answers
   • Adding human preference training on top of the examples was the only change that helped, since the plain examples alone gave no measurable gain
   • More varied prompts and filtered, higher-quality answers helped, while doubling the amount of data from one source did not improve the response quality
2. In SWE-smith, how did task difficulty relate to training value [1]?
   • Harder tasks gave much better students, so the authors kept only hard tasks, and they report a strong rise in student score with each step in the difficulty
   • Difficulty tracked whether the expert solved the task, but the authors saw no strong link between difficulty and how the student scored after training
   • Easy tasks gave the best students, since they have the most resolved runs, and the authors dropped every task that the expert model failed to solve
   • The authors never rated difficulty, because their execution check already keeps only the tasks that are right for training and they had no use for a rating
3. Why does LESS score examples by cosine similarity of gradients, and not by the dot product [6]?
   • The gradient norm shrinks as the completion gets longer, so a dot product upweights short sequences, and the authors report much worse results with it
   • Cosine is cheaper to compute than a dot product, and the authors say that the saving is what lets the method run on a large pool of instruction data
   • A dot product cannot be used with the Adam optimizer at all, since the optimizer rescales every gradient and so the sign of the score is not defined
   • Cosine makes all examples score the same, which removes the need for a warmup run and gives a fair draw from the pool for every one of the target tasks
4. The lab uses 16 bands of 4 rows for MinHash. For a pair of texts with Jaccard similarity 0.5, what is the chance that they become a candidate pair (my arithmetic)?
   • About 0.50, since the chance that one signature value matches equals the Jaccard similarity and banding leaves the chance of a candidate unchanged
   • About 0.06, because that is 0.5 to the power 4, and a pair becomes a candidate only if one particular band matches in all four of its rows
   • About 0.64, from 1 minus (1 minus 0.5 to the power 4) to the power 16, because a band needs all four rows to match and there are sixteen bands
   • About 0.99, because any pair with a similarity of one half or more is nearly always found when a signature is cut into sixteen bands of four rows
5. Two coders label 100 items. Both say "yes" on 90 items. On 5 items the first says "yes" and the second "no", and on 5 the reverse. What is Cohen's kappa (my arithmetic) [22]?
   • About 0.90, because kappa equals the observed agreement whenever the two coders have the same label shares, and here both give "yes" for 95 items
   • About −0.05, because the chance agreement is 0.95 squared plus 0.05 squared, which is 0.905, a little above the observed agreement of 0.90
   • Exactly 0, because the coders never agree on the rare label, and so kappa cannot be anything but zero when the rare label is never shared
   • About 0.45, because kappa is half of the observed agreement when one label is far more common than the other, and the chance term is then one half
6. The SWE-bench Verified team labeled each sample three times and "took the highest severity". What does that rule do [23]?
   • It averages the three severity labels, so that one extreme label cannot remove a sample, and it keeps most of the original samples in the final set of 500
   • It lets the most experienced annotator decide each sample alone, since the other two labels are used only to train new annotators during onboarding
   • It keeps a sample only if all three annotators gave exactly the same label, so it measures agreement and not severity, and it needs no cut-off
   • It drops a sample if any one of the three annotators gave a severe label (2 or 3) on a criterion, which means more false positives but a cleaner kept set
7. In confident learning, what is the threshold t_j for class j [17]?
   • The average predicted probability of class j among the examples whose given label is class j, which the paper calls the expected self-confidence
   • The share of all examples whose given label is class j, so that a class with many examples gets a high bar before any label error is counted
   • A fixed value of one half for every class, so that an example is flagged whenever its predicted probability for another class is more than 0.5
   • The largest predicted probability that any example in the dataset has for class j, so that only the single most confident example is ever counted
8. An item has discrimination α and difficulty β in the two-parameter model. For which models does it carry the most Fisher information?
   • Models of every ability equally, because the information of an item does not depend on the ability of the model that answers it in this model
   • Models with the lowest ability, because they fail the item most often and every failure adds the same amount of information about the item
   • Models with ability near β / α, where the chance to pass is one half, and more so when α is larger, so it splits models
   • Models with the highest ability, since they pass the item, and a pass is always worth more than a fail when the item is hard to solve
9. What is "criteria drift" in the EvalGen paper [2]?
   • The slow loss of accuracy of an LLM grader as the model behind it is updated, which makes grades given last month differ from the grades given today
   • People need criteria to grade outputs, but grading helps them define the criteria, so they add new criteria and reinterpret old ones as they go on grading
   • A tool that changes its own assertions at random between runs, so that two users of the same tool see different report cards for the same set of outputs
   • The difference between the grades of two annotators who were given the same written criteria, which is measured with an agreement coefficient such as kappa
10. In the lab's trace pipeline, 1.5% of the 201 kept traces have weak tests. If you audit 40 kept traces chosen at random, how likely is it that you see none (my arithmetic)?
    • About 55%, because each trace is clean with probability 0.985, and 0.985 to the power 40 is about 0.55, so the audit more often than not sees nothing
    • About 1.5%, because the audit shows a defect with the same probability as the true rate of the defect in the kept set, and so a miss is as rare as the defect
    • About 5%, because audits are designed to give a 95% chance of finding any defect that is present, whatever its rate in the set that was audited
    • About 0%, because forty traces is enough to cover the whole kept set of 201 once a stratified sample over the repositories is used for the audit

Answer key

7. Graded assignment and rubric

Title: Select, clean and audit a dataset on a mock.
Time: 5 to 6 hours. Total: 100 points. Work: solo.

The task

Extend the lab code and write a short memo (three pages at most). Mock data is fine. You may use real data if you have it, but say which parts used it, get consent for any real person's data, and scrub it before you store it. Build your own domain. Do not reuse the lab's task pool, its three linear tasks, its Gaussian classes, its response matrix, its entity-extraction outputs or its repository names, and do not reuse its numbers. Label every number that comes from a mock or from made-up data. Keep the code, the data and the memo consistent with each other [1] [2].

Part A. Informativeness and diversity (25 points)

• Make a mock pool of at least 300 tasks in a domain that is not the lab's (for example, spreadsheet formulas, regular expressions or unit conversions). Give each task a hidden pass probability. Score every task from k = 1, 4, 16 and 64 simulated attempts with the pass-rate variance, and report the overlap with the true top 50 for each k. Say what random picking would give (my arithmetic in the lab notes) [1] [16].
• Add flaky tasks with a fair-coin outcome. Show how many reach the top 50, and one way to tell them from tasks that are truly at 50%. Say what the way costs.
• Make 200 mock prompts with 10 planted near-duplicate pairs. Find them with MinHash and banding. Predict, from the formula in 2.10, how often a pair of Jaccard 0.5 is a candidate, then measure it on at least 200 made pairs. Report both numbers [18].
• Choose 30 points from 300 with farthest-first and with random draws. Report the covering radius of each (the random one as a mean and a range over at least 20 draws). Add 5 outliers and report what happens to the first 10 picks [7].
• Name three limits of your mock pool.

Part B. Targeted selection and label noise (25 points)

• Build your own toy world with a target task, an unrelated task and a conflicting task, in a model that is not the lab's logistic model with three linear tasks (for example, a small decision rule or a nearest-centroid model with its own gradient-like score). Select 20% of the pool by a similarity of example-level scores to a few target examples, with a cosine and with a raw dot product. Report accuracy on the target for random, cosine, dot product and the full pool, as a mean and a range over at least 5 seeds [6].
• Show what happens when 20% of the pool has a large norm. Say which of the two scores is hurt and why, in your own words [6].
• Make a three-class mock data set with class-conditional label noise at 10% and 30%. Compare confident learning with the self-confidence baseline by precision and recall, over at least 5 seeds. Then flip labels in one direction only, and report both methods again [17].
• Compute a data map (confidence and variability over at least 10 epochs) on a binary task. Report the share of flipped labels in the hard, ambiguous and easy regions, and the base rate [15].
• Name three limits of your toy world.

Part C. Agreement and a tiny benchmark (25 points)

• Simulate three annotators who give severity labels 0 to 3 on two criteria for at least 400 items, with accuracy of your choice. Compute percent agreement, pi, kappa and alpha for two label prevalences. Show a case with high agreement and a poor coefficient, and explain it [22].
• Compare three keep rules: the maximum of three labels, the median, and one annotator. Give a table of the kept share, the precision and the recall against the true labels. State which you would use, for which purpose, and what you would tell the users of the data [23].
• Make a mock response matrix with at least 60 models and 300 items. Fit a 1PL and a 2PL model on 40 models. Choose 24 and 48 items by random draw, by stratified difficulty, and by Fisher information. Report Kendall tau-b and the mean absolute error on the 20 held-out models, as a mean and a range over at least 10 seeds [25] [26].
• Write what you would and would not claim from the tiny benchmark. Remember that your response matrix is made up.
• Name three limits of your simulation.

Part D. A trace pipeline, an audit and the memo (25 points)

• Make at least 500 mock agent traces for your own domain, with at least three defects that you invent and plant (for example, a trace that prints the answer without running anything). Build at least eight gates, including three for your new defects. Give a funnel table, and a table of how many planted defects of each kind are in the pool and in the kept set [1].
• Take an audit sample of 40 kept traces. Report the Wilson interval for the defect rate that the audit finds, and compute how many traces an audit would need to have a 95% chance of seeing at least one defect at a true rate of 1.5% (my arithmetic is 198 in the lab notes; show your own working) [1].
• Write a data card for your kept set in your own words. It must list the gates, the known gaps and the date [21].
• Write a memo of two pages at most. Your team has 40,000 agent traces from the last quarter and wants to fine-tune a model. Decide whether to use them, how to select them, what the quality gates are, who audits and how many traces, which labels need more than one annotator, and the stop criteria. Give a clear yes or no and three limits of your evidence [1] [2] [3].

What to hand in

• Your selection_quality_toolkit.py (or your own file names), your mock data, and the tests. All tests must pass.
• A memo in PDF or Markdown with the tables from Parts A to D, the data card and the memo from Part D.
• A sources list for any reading you cite.

Rubric

Criterion	Full marks	Partial	Points
A. Informativeness and diversity	A new pool of 300 or more tasks, the overlap table for four values of k with a random baseline, a flaky-task test with a cost, 10 planted near-duplicate pairs with a predicted and a measured candidate rate, covering radii with a range and an outlier test, three limits.	Fewer tasks, or no random baseline, or no flaky tasks, or no prediction, or a single random draw for the radius.	25
B. Targeted selection and label noise	An own toy world, four arms with a mean and a range over five seeds, a large-norm test with a reason, confident learning against a baseline at two noise rates and for one-way noise, a data map with region shares, three limits.	One seed, or no large-norm test, or only one noise rate, or no one-way test, or no data map.	25
C. Agreement and a tiny benchmark	Four coefficients at two prevalences with an explained case, a keep-rule table with a recommendation, a response matrix with 1PL and 2PL fits, three item-selection recipes at two sizes with ranges over ten seeds, an honest claim, three limits.	One prevalence, or no keep-rule table, or no ranges, or claims that go beyond a made-up matrix.	25
D. Pipeline, audit and memo	500 or more traces, three new defects with gates, a funnel and a defect table, an audit with a Wilson interval and your own sample-size working, a data card with gaps, a two-page memo with a clear decision, audit plan, annotation rule and stop criteria, three limits.	Fewer gates, or no defect table, or no sample-size working, or a memo without a decision or without stop criteria.	25
Total			100

Grade bands: 90 to 100 excellent. 75 to 89 good. 60 to 74 needs work. Under 60: redo and resubmit. Plain writing is rewarded: short sentences, no filler. Label your own arithmetic as your own.

8. Glossary

Term	Meaning
Active learning	Choosing which examples to label so that a model improves the most from the fewest labels [16].
Alignment (EvalGen)	The harmonic mean of coverage and one minus the false failure rate [2].
Anchor points	A small set of items that stand for the rest of a benchmark, chosen by how model confidence moves together [26].
Audit	A human check of a random sample of a finished dataset. It can show a defect that is common. It cannot show one that is rare (2.10).
Banding (LSH)	Cutting a MinHash signature into bands. Two texts are candidates if any band matches in all rows. The chance is 1 − (1 − sr)b for Jaccard s, r rows and b bands (my arithmetic, 2.10) [18].
Cohen's kappa	Chance-corrected agreement of two coders, where chance comes from each coder's own label shares [22].
Confident joint	A count matrix of examples by given label and by the label that the model is confident about. Confident learning builds it with per-class thresholds [17].
Confident learning	A way to find label errors from out-of-sample predicted probabilities and class thresholds [17].
Coreset	A small weighted subset that stands in for a large set. CRAIG picks one by gradients [11].
Coverage (EvalGen)	Of the outputs the user graded bad, the share that the assertion set fails [2].
Covering radius	The largest distance from any point to its nearest chosen point. Farthest-first keeps it small (2.4).
Criteria drift	The way people's criteria change while they grade outputs, because grading helps them define the criteria [2].
Data card	A short document that says what a dataset is, where it came from, what was done to it and what it must not be used for. Related to datasheets for datasets [21].
Data map	A plot of each example's confidence and variability over training. It has hard, ambiguous and easy regions [15].
Data programming	Writing labeling functions whose noisy votes are combined by a model. Snorkel is a system for it [19] [20].
De-duplication	Removing exact and near copies from a corpus [18].
Difficulty filter	A filter that keeps tasks that the agent sometimes, and not always or never, solves, such as 0 < passes < k. It is noisy (2.2) [1].
Discrimination	The parameter α of an item in a two-parameter IRT model. It sets how fast the chance of a correct answer grows with ability [25].
DSIR	Data Selection with Importance Resampling. It picks raw text whose hashed n-gram features look like a target set [7].
EL2N	The norm of the error vector early in training. A high value marks a hard example [12].
Ensemble (max severity)	Combining several labels by taking the most severe. The SWE-bench Verified team used it [23].
Fair-coin task	A mock task whose outcome does not depend on the agent. Its pass rate is near 50%, so a difficulty score ranks it as informative (2.2).
False failure rate	Of the outputs the user graded good, the share that the assertion set fails anyway [2].
Farthest-first	A greedy way to pick points. Each new pick is the point farthest from all picks so far. It is also called k-center greedy (2.4).
Fisher information	How much an item tells you about a model's ability. In the lab's 2PL model it is α2 p(1 − p) (2.7).
Fleiss's multi-pi	A coefficient for many coders. It generalizes Scott's pi, and is often called Fleiss's kappa [22].
Forgetting event	An example that was classified right at one epoch and wrong at the next. Some examples are forgotten often and some never [13].
Gate	A rule that a trace must pass to enter the dataset. It catches only what it was written for (2.8).
Gumbel top-k	A way to draw k items without replacement with probability proportional to a weight, by adding Gumbel noise to the log weights (2.3).
IRT	Item response theory. The chance of a correct answer is a function of the model's ability and the item's difficulty, and in the 2PL form its discrimination [25].
Jaccard similarity	The size of the overlap of two sets divided by the size of their union. The lab applies it to sets of word shingles [18].
Kendall's tau-b	A rank correlation that handles ties. The lab uses it to compare a tiny benchmark's ranking with the full one (2.7).
Krippendorff's alpha	An agreement coefficient that handles missing ratings, any number of coders and weighted distances [22].
LESS	Low-rank gradiEnt Similarity Search. It picks the training examples whose gradients are most similar to those of a few target examples [6].
MinHash	A short signature of a set such that the chance two signature numbers match equals the Jaccard similarity of the sets [18].
Prevalence problem	When one label is very common, chance-corrected coefficients can be low or negative although percent agreement is high [22].
Rater filter	Using a model to score examples and keeping the ones above a cut. AlpaGasus, Deita and QuRating are examples [8] [9] [10].
Scott's pi	Chance-corrected agreement of two coders, where chance comes from the pooled label shares [22].
Selectivity	The pass rate of an assertion over a set of outputs. EvalGen uses it to choose which outputs to ask a person to grade [2].
Superficial Alignment Hypothesis	The claim of the LIMA authors that a model learns its knowledge in pretraining and alignment mostly teaches a format [3].
Weak supervision	Training from labels that come from rules, heuristics or models, and not from careful human labeling [19] [20].
Wilson interval	A confidence interval for a share. The lab uses it for the defect rate that an audit finds (2.10).

9. References

Entries marked (required) are the course's required reading, and (course, additional) is the course's additional reading. Entries marked (added) are my own additions. The course site and the slide test, [4] and [5], are the course's own pages. "Abstract level" means that I read the abstract and not the whole paper. Where an entry names more, that is what I read. Web pages were read in October 2026.

• [1] Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., and 6 others. "SWE-smith: Scaling Data for Software Engineering Agents." NeurIPS Datasets and Benchmarks 2025. arXiv:2504.21798 (2025). The first author, John Yang, and the last author, Diyi Yang, teach the course. https://arxiv.org/abs/2504.21798 (I read the abstract, the introduction, sections 2 to 4 and the appendix on difficulty rating, and skimmed the rest; I read the arXiv text, and the NeurIPS version may differ; the results are the authors' own claims; I did not run their code) (required)
• [2] Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., and Arawjo, I. "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences." UIST 2024. arXiv:2404.12272 (2024). https://arxiv.org/abs/2404.12272 (I read the abstract, the offline evaluation, the description of the user study, the findings on criteria drift and the appendix on sampling and alignment, and skimmed the rest; the study has nine participants; I read arXiv v1) (required)
• [3] Zhou, C., Liu, P., Xu, P., Iyer, S., and 11 others. "LIMA: Less Is More for Alignment." NeurIPS 2023. arXiv:2305.11206 (2023). https://arxiv.org/abs/2305.11206 (I read the abstract, the introduction, the data section, the agreement paragraph of the human evaluation, the ablations, the multi-turn section and the discussion; I read arXiv v1; the authors' own claims; the ablation compares different sources, and the authors say other factors may differ) (course, additional)
• [4] Stanford CS329Z: Engineering AI Agents, course site and schedule (Diyi Yang, Michael Ryan, John Yang). https://cs329z.stanford.edu/ (the course's own page; it lists Week 7, "Data Selection and Quality" (Monday, 2 November 2026), with the topics and the readings, and says the schedule is tentative)
• [5] CS329Z slide addresses lecture12.pdf and lecture13.pdf, tried once on 29 October 2026. https://web.stanford.edu/class/cs329z/slides/lecture12.pdf (both addresses returned "not found" (HTTP 404), so I could not use slides for this week; the second address is lecture13.pdf in the same folder)
• [6] Xia, M., Malladi, S., Gururangan, S., Arora, S., and 1 other. "LESS: Selecting Influential Data for Targeted Instruction Tuning." arXiv:2402.04333 (2024). https://arxiv.org/abs/2402.04333 (abstract level, and I also read the passages on the default setting, the cosine normalization and the projection dimension; the authors' own claims) (added)
• [7] Xie, S. M., Santurkar, S., Ma, T., and Liang, P. "Data Selection for Language Models via Importance Resampling." arXiv:2302.03169 (2023). https://arxiv.org/abs/2302.03169 (abstract level; the authors' own claims) (added)
• [8] Chen, L., Li, S., Yan, J., Wang, H., and 7 others. "AlpaGasus: Training A Better Alpaca with Fewer Data." arXiv:2307.08701 (2023). https://arxiv.org/abs/2307.08701 (abstract level, and the paper text for the 9,229 count; the authors' own claims; the rater is ChatGPT) (added)
• [9] Liu, W., Zeng, W., He, K., Jiang, Y., and 1 other. "What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning." arXiv:2312.15685 (2023). The method is called Deita. https://arxiv.org/abs/2312.15685 (abstract level; the authors' own claims) (added)
• [10] Wettig, A., Gupta, A., Malik, S., and Chen, D. "QuRating: Selecting High-Quality Data for Training Language Models." arXiv:2402.09739 (2024). Alexander Wettig is also an author of [1]. https://arxiv.org/abs/2402.09739 (abstract level; the authors' own claims) (added)
• [11] Mirzasoleiman, B., Bilmes, J., and Leskovec, J. "Coresets for Data-efficient Training of Machine Learning Models." arXiv:1906.01827 (2019). The method is called CRAIG. https://arxiv.org/abs/1906.01827 (abstract level; the authors' own claims) (added)
• [12] Paul, M., Ganguli, S., and Dziugaite, G. K. "Deep Learning on a Data Diet: Finding Important Examples Early in Training." arXiv:2107.07075 (2021). https://arxiv.org/abs/2107.07075 (abstract level; the authors' own claims) (added)
• [13] Toneva, M., Sordoni, A., Tachet des Combes, R., Trischler, A., and 2 others. "An Empirical Study of Example Forgetting during Deep Neural Network Learning." arXiv:1812.05159 (2018). https://arxiv.org/abs/1812.05159 (abstract level; the authors' own claims) (added)
• [14] Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and 1 other. "Beyond neural scaling laws: beating power law scaling via data pruning." arXiv:2206.14486 (2022). https://arxiv.org/abs/2206.14486 (abstract level; the authors' own claims) (added)
• [15] Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., and 3 others. "Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics." arXiv:2009.10795 (2020). The paper calls the plots data maps. https://arxiv.org/abs/2009.10795 (abstract level; the authors' own claims) (added)
• [16] Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., and 4 others. "A Survey of Deep Active Learning." arXiv:2009.00236 (2020). https://arxiv.org/abs/2009.00236 (abstract level; a survey, so I did not check its coverage) (added)
• [17] Northcutt, C. G., Jiang, L., and Chuang, I. L. "Confident Learning: Estimating Uncertainty in Dataset Labels." arXiv:1911.00068 (2019). https://arxiv.org/abs/1911.00068 (I read the abstract, the introduction and the definition of the confident joint; I did not check the theory or the experiments; the authors' own claims) (added)
• [18] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., and 3 others. "Deduplicating Training Data Makes Language Models Better." arXiv:2107.06499 (2021). https://arxiv.org/abs/2107.06499 (abstract level; I did not check which methods the two tools use) (added)
• [19] Ratner, A., De Sa, C., Wu, S., Selsam, D., and 1 other. "Data Programming: Creating Large Training Sets, Quickly." arXiv:1605.07723 (2016). https://arxiv.org/abs/1605.07723 (abstract level; the authors' own claims) (added)
• [20] Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., and 2 others. "Snorkel: Rapid Training Data Creation with Weak Supervision." arXiv:1711.10160 (2017). https://arxiv.org/abs/1711.10160 (abstract level; the authors' own user-study claims) (added)
• [21] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., and 3 others. "Datasheets for Datasets." arXiv:1803.09010 (2018). https://arxiv.org/abs/1803.09010 (abstract level; the lab's data card is my own simplification, not the datasheet questionnaire) (added)
• [22] Artstein, R., and Poesio, M. "Inter-Coder Agreement for Computational Linguistics." Computational Linguistics 34(4), 555 to 596 (2008). https://aclanthology.org/J08-4004/ (I read the whole survey; I used its two-coder example and its definitions of pi, kappa, multi-pi and alpha, and I did not check every formula) (added)
• [23] OpenAI. "Introducing SWE-bench Verified." Blog post, 13 August 2024, updated 24 February 2025. Vendor-authored; the author list includes John Yang and Carlos E. Jimenez, who are also authors of [1] and of SWE-bench. https://openai.com/index/introducing-swe-bench-verified/ (I read the whole page, including the annotation rubric text on it; I did not open the released annotations; the numbers are OpenAI's own claims) (added)
• [24] OpenAI. "Why SWE-bench Verified no longer measures frontier coding capabilities" (page title "Why we no longer evaluate SWE-bench Verified"). Vendor-authored. Reused from Week 6b. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ (I read the opening sections as a search-result excerpt; my fetch of the page was refused, so I did not read the rest, and I did not read its date) (added)
• [25] Polo, F. M., Weber, L., Choshen, L., Sun, Y., and 2 others. "tinyBenchmarks: evaluating LLMs with fewer examples." arXiv:2402.14992 (2024). Also the course's additional reading for next lecture. https://arxiv.org/abs/2402.14992 (I read the abstract and the sections on key findings and limitations; the authors' own claims; next lecture goes deeper) (added)
• [26] Vivek, R., Ethayarajh, K., Yang, D., and Kiela, D. "Anchor Points: Benchmarking Models with Much Fewer Examples." arXiv:2309.08638 (2023). One of its authors, Diyi Yang, teaches the course. https://arxiv.org/abs/2309.08638 (abstract level; the authors' own claims) (added)
• [27] Perlitz, Y., Bandel, E., Gera, A., Arviv, O., and 5 others. "Efficient Benchmarking of Language Models." arXiv:2308.11696 (2023). https://arxiv.org/abs/2308.11696 (abstract level; the authors' own claims) (added)
• [28] Pan, J., Wang, X., Neubig, G., Jaitly, N., and 3 others. "Training Software Engineering Agents and Verifiers with SWE-Gym." arXiv:2412.21139 (2024). https://arxiv.org/abs/2412.21139 (abstract level; the claim about repeatedly solved tasks is quoted from [1], and I did not find it in this abstract) (added)
• [29] Shankar, S. "Data Flywheels for LLM Applications." Blog post, 1 July 2024. An essay with no experiments. Reused from Week 6b. https://www.sh-reya.com/blog/ai-engineering-flywheel/ (I read the whole post for Week 6b and did not reread it for this week) (added)