TwinEdgeAI

↑ Course home

← Previous: Week 6a, Terminal-Bench  ·  Next: Week 7a, Data Selection & Quality →

Engineering AI Agents · Week 6b

Data for Agents

What data do agents need? Traces, demonstrations and feedback; data for optimization and data for evaluation; where the data comes from; synthetic data; data flywheels; learning from people who use the agent; and what the data teaches by accident.

Author: Grok Bot Version: 1.0, October 29, 2026 Study time: about 15 hours

CreditThis coursework is adapted from, and inspired by, Stanford CS329Z: Engineering AI Agents (Diyi Yang, Michael Ryan, John Yang), Week 6, Lecture "Data for Agentic Systems: What Data Do Agents Need?" (Wednesday, October 28). The topics and the readings come from the course's public schedule [4]. I checked the coverage against the public lecture slides [5]. The slides are short, so the notes also lean on the readings and on sources I added. The notes, lab, quiz and assignment are original writing, and the lab uses its own examples and code. This pack does not reproduce the slides. It is not an official Stanford document. Each source I added is marked "added" in the references.

1. Week overview and learning objectives

Week 5b tuned the system around a frozen model. Week 6a measured agents. This week turns to the model itself. The lecture's first slides say that "data is the main lever" for making the model a better agent [5]. The question is which data, and where it comes from.

The week has seven parts. First, two kinds of agent data: trajectories, which a model imitates, and environments, where a model practices. Second, traces: the log of what an agent did, and the three things you can make from it (demonstrations, preference pairs, eval cases). Third, optimization data against evaluation data, and how they leak into each other. Fourth, sources: people, the web, and models. Fifth, synthetic data, and the filters that keep it honest. Sixth, flywheels and interaction data: how a deployed agent produces its own next round of data, and what users tell you when they use it. Seventh, accidents: what data teaches that you did not mean to teach, and what you owe the people whose data you log.

The lab uses the standard library only. It does not call any model, any API or the network. The "agent" is a mock with made-up skill levels, and so is the "user". Everything the lab prints is made up. It shows how the pieces work: a trace schema, conversion of traces into training and eval data, leakage checks, a flywheel loop, a task generator with filters, feedback signals, PII scrubbing and a data card. It says nothing about how real models or real users behave.

Learning objectives

After this week you should be able to:

1. Say what a trajectory and an environment are, which parts of a trajectory a model learns, and where an environment's reward can come from. [2.1]
2. Design a trace schema, and turn traces into supervised demonstrations, preference pairs and eval cases, with consent checks. [2.2]
3. Explain why optimization data and evaluation data must be kept apart, and name three ways they leak: exact copies, near copies, and shared groups or times. [2.3]
4. Compare the five sources of agent data on cost, realism and scale, with one example each. [2.4]
5. Describe a synthetic data pipeline from seeds to filters, and say what each filter cannot catch. [2.5]
6. Describe the data flywheel in Shankar's terms (evaluation, monitoring, continual improvement), and say what changes when the improvement step is training. [2.6]
7. Compare explicit and implicit feedback, and say how to measure how good a weak label is. [2.7]
8. Name four things data can teach by accident (reward hacks, a single harness, SFT overfit, model collapse), and a fix for each. [2.8]
9. Scrub, consent-filter and document a trace set, and say what a regex scrubber misses. [2.9]
10. Compute a MinHash candidate probability, the pass rate of a difficulty filter, and a Bradley-Terry win probability. [2.10]
11. Build and test a small data toolkit and use it to run a flywheel on a mock. [Lab]

How to use this pack

Time	What to do
3.5 hours	Read the notes (section 2). Skim the figures first.
3.5 hours	Read the required reading (Shankar) and skim the additional one (Tan) with the reading guide questions. Skim the optional ones that match your project.
4.5 hours	Do the lab and at least five exercises.
1.5 hours	Discussion and the quiz.
2 hours	The graded assignment.

2. Lecture notes

2.1 Why data, and two kinds of agent data

Last lecture, the slides say, the weights stayed frozen and you tuned the parts around the model. This lecture asks how to make the model itself a better agent, and the slides' answer is that "data is the main lever" [5]. A reminder from the slides: the same base model with different post-training data can end up very different. The slides show a chart from a 2026 paper, OT-Agent, in which a well-chosen subset of the same budget of supervised traces beats a poorer one [5]. I did not open that paper. Choosing which traces to use is next week's topic [4].

The slides split "agent data" into two kinds [5]. They credit the words "conversations" and "environments" to Karpathy, and I did not check that.

• A trajectory is a recording of an agent solving a task: think, act, observe, act, observe, submit. You train on it with supervised fine-tuning (SFT). The model imitates what was kept.
• An environment is a task, a sandbox, and a way to check success. You train on it with reinforcement learning (RL). The model tries many times, each run gets a score, and the update favours what scored well.

Trajectories can be made by running a model in an environment and recording what it did [5]. So the two kinds are linked. An environment makes trajectories, and a filter on the score decides which trajectories to keep.

What a model learns from a trajectory

The slides show that the training objective predicts the agent's own thoughts and actions. The system prompt, the task and the tool outputs are context only. The format (tool calls, ReAct text, code actions) is set by the system around the model [5]. This matters for data design. A tool output is not something the model is asked to produce, so a trace with a long, noisy tool output teaches less per token than its length suggests. In the lab I count "learned" words as the words of assistant turns and "context" words as everything else. That is my simplification, not a rule from the slides.

Where the reward comes from

For a trajectory, the "reward" is just whether the model predicted the recorded action. For an environment, the reward has to come from somewhere, and the slides list four sources: unit tests, rubrics, an LM as judge, and live human preference [5]. The slides also say the next three lectures cover reward types and LM judges. This week we only need one fact from that list: the more reliable the check, the more you can trust the data you keep with it. Section 2.5 shows a verifier as a data filter.

The slides give one data point on scale: DeepSeek V3 used 14.8 trillion pretraining tokens, 1.5 million SFT examples, and (for the later V3.2) 85 thousand RL task prompts. They add that in V3.2 the post-training compute rose above 10% of the pretraining compute, and that labs usually do not disclose such numbers [5]. I did not check these figures. Read them as the lecture's illustration that post-training uses far fewer items than pretraining, and that each item is far more costly to make.

2.2 Traces, demonstrations and feedback

A trace is the log of one run: what the user asked, what the agent thought and did, what the tools returned, and what happened afterwards. Hamel Husain describes it this way: "tracing is a logical grouping of logs", and for LLM products a trace is often a conversation [3]. For agents a trace also holds tool calls and results. Traces are the raw material for everything else this week. They are what you read when you debug, what you train on after you filter them, and what you mine for test cases.

The OpenTelemetry project has a draft convention for logging generative AI work as a tree of spans, with attributes for model calls, tool calls and agent steps [30]. Its status was "Development" when I read it, so names may change. The lab's schema is my own and is simpler: a flat list of events with a kind (system, user, assistant, tool_call, tool_result, feedback), a timestamp, and a trace-level record of the user, the harness, the task type, a reward, the verifier spec, the consent flags and a scrubbed flag. The lab can fold a trace into a span tree to show how the two relate. I did not try to match OpenTelemetry's attribute names.

Three things you can make from a trace

1. A demonstration for SFT. The rule in the lab is: the run passed, the owner allows training, and the trace is scrubbed. The learned part is the assistant turns. This is the slides' "curate, then imitate what is kept" [5]. Note what the rule leaves out. A run that passed by luck, or by a hack, looks the same as a good one. That is the point of section 2.8.
2. A preference pair: a chosen and a rejected answer for the same prompt. There are two sources in the lab. One is outcome: the same prompt has a passing run and a failing run. The other is an edit: the user changed the agent's answer, so the edited text is chosen and the original is rejected. A pair is the input for methods like DPO [29], and an edit is the signal that PRELUDE learns user preferences from [27]. I read the abstracts of both. The lab only builds the pairs. It trains nothing on them.
3. An eval case: a failed run turned into a test, with its own verifier. Husain's point is that a product needs its own evals, built from its own failures, and that unit-test style checks (his Level 1), human and model review of traces (Level 2) and A/B tests (Level 3) are three different tools [3]. The lab builds only the first kind.

In the lab's demo, 120 mock traces give 87 passes and 33 failures. 93 owners allow training. After the filter there are 72 demonstrations, which is "passed AND consented", and every one of them used the react harness. There are 7 preference pairs from outcomes and 5 from edits, and 29 eval cases from failures. All of these are mock numbers. The shape is the point: each step removes data, and the survivors are not a random sample. The survivors are the traces of consenting users, on tasks the agent could already do. The eval cases are the reverse: they are biased toward what the agent cannot do. Both biases are fine if you know about them.

Look at the data

Husain says: "You must remove all friction from the process of looking at data." His advice is to build a viewer that puts everything needed to judge a trace on one screen [3]. Shankar says the same from another side: you cannot find the right metrics by theorising about failures, and you need to examine many real outputs [1]. Neither source gives a number for how many traces to read. The lab prints a short trace and a validator report, and exercise 1 asks you to read 20 of them by hand. That is my suggestion.

A trace validator is cheap and finds real bugs. The lab's checks are: the first message is from the user, every tool result has a call, every call has a result, and timestamps do not go backwards. A broken trace trains a broken model, and it also breaks the eval that uses it.

2.3 Data for optimization and data for evaluation

The same traces can serve two jobs. Optimization data is used to change the agent: SFT demonstrations, preference pairs, RL tasks, few-shot examples, or prompt tuning data. Evaluation data is used to measure the agent. One rule links them: the data you optimize on must not be the data you report on. If it is, the score tells you how well the agent remembers, and not how well it works on new tasks. This is the idea behind the held-out test set in all of machine learning. For agents it is easy to break, because the same logs feed both.

Real cases exist. OpenAI wrote that it stopped reporting SWE-bench Verified scores. Its post says that frontier models it tested could reproduce the original fix or verbatim details of some tasks, which suggests that they saw them in training [34]. I read this post through a search excerpt, and I did not fetch the full page. The slides cite the same post for the claim that models could recite issues from memory [5]. Whatever the final accounting, the lesson is plain: a benchmark built from public data can leak into training data from public sources.

Three kinds of leak, and three kinds of split

• Exact copies. The same prompt in train and test. Easy to find with a hash.
• Near copies. A reworded or lightly edited prompt. You need a similarity measure. The lab uses word shingles, MinHash and banding to find candidate pairs, then checks the exact Jaccard overlap. Lee and colleagues found many near-duplicates in language model datasets, and train-test overlap that affects over 4% of the validation set of standard datasets [31]. I read the abstract only.
• Shared groups and shared time. The same user, the same document, the same task template, or a test item that is older than the training data. A random split ignores all of these.

The matching splits are: random (fast, leaks everything), group (all items with the same prompt group go to the same side), and time (train on the past, test on the future). A time split matches how a deployed agent is used. It does not remove duplicates, so you still need a copy check after it.

In the lab's demo on 120 mock traces, a random split gives 96, 12 and 12 items, with 15 exact train-to-other copies, 17 shared users and 24 items in dev or test that are not newer than the newest training item. A split by prompt group gives no exact copies and no shared prompts. Then I add five reworded copies of training prompts to the test set. The group split still finds no exact copy, but the near-copy check finds all five and marks the split not ok. A time split has no time violations, but it still has 12 exact copies from train to test, and the decontamination step drops 3 of the 84 training items. All counts are mock.

Evals made from failures

The lab turns failed runs into eval cases. That is useful, because real failures are the cases you care about. It also has a cost. A test set made only of failures cannot show a regression on the easy cases, and once you train on those failures they are no longer a test. My suggestion: keep a sample of ordinary runs next to the failures, freeze the eval set before you train, and split the failures into a train part and a test part by group. The lab's traces_to_eval_cases function can cap the number per task type, so one frequent failure does not take over the set. Anthropic's guide on evals separates regression evals, which should stay near a 100% pass rate, from capability evals, which start with a low pass rate and give you something to climb [36]. I read only the parts of that post that I cite. It is a vendor post.

2.4 Where agent data comes from

The slides name five sources and ask three questions of each: what does it cost, how realistic is it, and how far does it scale [5]. The table below follows the slides' five groups. The ratings in the last three columns are my own judgment, in words, and they are not measurements.

Source	Examples named in the lecture	Cost	Realism	Scale
1. Human demonstrations	Mind2Web, OpenCUA	High: people are paid per task	High	Low
2. Mined from the web and the world	SWE-rebench, AgentTrek, VideoAgentTrek	Low to medium	Medium: real sources, converted	Medium to high
3. Synthesized trajectories	Synatra, NNetNav, OS-Genesis	Low	Medium	High
4. Synthesized tasks and environments	SWE-smith, R2E-Gym, AgentScaler	Low per task, high to set up	Medium	High
5. Simulated users and environments	tau-bench, APIGen-MT, DreamGym	Low	Depends on the simulator	Very high

1. Human demonstrations

People do the task while a recorder logs every screen and action. The slides give the pipeline as define, perform, record, filter, package [5]. Mind2Web is the web example. Its abstract says it has over 2,000 open-ended tasks from 137 websites in 31 domains, with crowdsourced action sequences [15]. The slides' table says 2,350 tasks, and I did not check the body of the paper for the exact figure. The slides also describe OpenCUA, which records screen, mouse and keyboard on annotators' own computers and reports 22.6K tasks. I did not open it. The slides' comparison is useful: Mind2Web has web actions and page snapshots and no reasoning text. OpenCUA covers desktops and adds synthetic reasoning for each step [5]. So even "human" data is often completed by a model afterwards.

2. Mined from the web and the world

The idea is to convert existing artifacts into trajectories or tasks: issues and the commits that fixed them, written tutorials, and screen videos [5]. The slides cite SWE-rebench, which mined 21,336 tasks from 3,468 repositories with automatic environment set-up and test validation. I did not open it. AgentTrek is the web example. Its abstract describes three steps: collect tutorial-like text from the internet with a classifier, turn it into task specifications with step-by-step instructions, and have a vision-language agent carry out the steps in real environments while a second model checks that the trajectory is right. It reports a cost of $0.55 per high-quality trajectory [17]. That is the authors' claim. The slides also cite VideoAgentTrek, which labels YouTube screen recordings with an inverse dynamics model. I did not open it.

3. Synthesized trajectories

A model writes or runs the steps, and a filter keeps the good ones. The slides split this into two orders [5].

• Instruction first. Start from a task or a tutorial and have a model write the steps. Synatra turns indirect knowledge such as tutorials into direct demonstrations. Its abstract says the 100,000 synthetic demonstrations it made trained a 7B model that beats comparably sized models on three web benchmarks, and that a synthetic demonstration costs 3% of a human one, at $0.031 [18]. The slides' comparison with AgentTrek: Synatra writes steps offline and never runs them, so its observations are synthesized page text. AgentTrek replays the steps live and gets real screens [5]. A step that was never run can contain a mistake no one sees.
• Interaction first. Let an agent explore, then label what it did in hindsight. OS-Genesis lets an agent interact with an environment step by step, then works out in hindsight which task each run accomplished, and uses a trajectory reward model to filter [20]. NNetNav does the same for websites and prunes an exploration early when the partial run cannot be given a meaningful sub-task. Its abstract reports that a Llama 3.1 8B model tuned on 10,000 such demonstrations reaches over 16% on WebArena and 35% on WebVoyager [19]. These are the authors' numbers.

4. Synthesized tasks and environments

Here the model is not asked for steps. Instead, you build the sandbox, the task and the checker, so that every task comes with a verifiable reward [5]. For code, SWE-Gym has 2,438 real Python task instances, each with a codebase, an executable runtime and unit tests [21]. SWE-smith goes the other way: given any Python repository, it builds an environment and then makes many task instances that break existing tests. Its abstract reports 50,000 instances from 128 repositories [22]. The slides add that the work took about 20 hours of human effort, which I did not check, and that the authors of SWE-smith include a course instructor. SWE-smith is next week's reading [4], so I keep it short here. R2E-Gym (executable tasks from commits) and AgentScaler (simulated databases behind real API schemas, for tool use) appear on the slides, and I did not open them. For tools, the slides' pattern is a simulated database, a goal made from a chain of dependent calls, and a check that the final database state matches the expected one [5].

5. Simulated users and environments

A model plays the user, the environment, or both. The slides' summary: "cheap, unlimited practice" and the risk that "agents learn simulator mistakes" [5]. tau-bench has a language-model user talk to an agent that has API tools and policy rules, and it grades by comparing the final database state with a goal state [24]. APIGen-MT builds data in two phases: it first makes task blueprints with ground-truth actions, checked by a committee of model reviewers, and then turns each blueprint into a full conversation through simulated human-agent interplay. Its abstract says that models trained on it beat larger models on tau-bench [12]. That is the paper's claim, and tau-bench is also the test, so see section 2.3 on how to read it. DreamGym, which replaces real rollouts with an LLM "experience model", is on the slides only.

2.5 Synthetic data generation

Synthetic data is data a model made. The Tan survey, listed as an additional reading, is about using language models to annotate and to synthesize data. Its abstract says the survey covers how annotations are generated, how they are assessed, and how they are used for learning, and it discusses limits [2]. I read the abstract only. Use the survey as a map, and read its sections yourself. The rest of this section is about the pattern that recurs in instruction and agent data.

A short history of bootstrapping

• Self-Instruct (2022). A model produces instructions, inputs and outputs from a small seed set. Then invalid or too-similar samples are filtered out before fine-tuning. The abstract reports that applying it to the original GPT-3 gave a 33% absolute gain on Super-NaturalInstructions, close to a model trained with private data and human labels [6]. These are the authors' numbers.
• Alpaca (2023). The Stanford team started from 175 human-written seed pairs from Self-Instruct and prompted text-davinci-003 to write more. It gives 52,000 instruction-and-output pairs, which cost under $500 in API fees. The blog says the release is for research only, and it gives three reasons: a non-commercial license on LLaMA, the data provider's terms against building competing models, and the lack of adequate safety measures. It also lists hallucination, toxicity and stereotypes as known limits [7]. I read the whole post. It is a useful reminder that synthetic data inherits its source's terms of use.
• Evol-Instruct and WizardLM (2023). Start from an initial set of instructions and rewrite them step by step into harder ones, then mix and fine-tune. The authors report that humans preferred the evolved instructions to human-written ones on their test bed [8]. The point for us is the control knob: you can ask for difficulty.
• AgentInstruct (Microsoft, 2024). Starts from raw documents and code files as seeds, and has agentic flows create both prompts and responses. The authors report a post-training set of 25 million pairs and gains for Mistral-7B across several benchmarks [9]. Name clash: the AgentTuning paper also has a dataset called AgentInstruct, which is a small set of agent trajectories [13]. They are different projects.

The same pattern for agents

For agents, one change matters most: the answer can often be checked by running something. APIGen verifies each function-calling sample in three stages: format checking, actual execution of the function, and a semantic check. It starts from 3,673 executable APIs in 21 categories and releases 60,000 entries [10]. ToolACE builds a pool of 26,507 APIs by a self-evolution process, has several agents write dialogs, and checks them with a rule-based layer and a model-based layer [11]. Both are reporting their own results. FireAct fine-tuned Llama 2 7B on 500 agent trajectories made by GPT-4, and its abstract says this raised HotpotQA performance by 77%, and that mixing tasks and prompting methods in the data helped further [14]. AgentTuning mixes its agent trajectories with general instructions, so the model keeps its general skills [13].

Husain describes the same loop for a product. Ask a model for 50 realistic inputs, then use your Level 1 and Level 2 tests to filter out the data that fails assertions or that a critique model thinks is wrong [3]. His claim is that once you have evals, curation and synthesis come "nearly for free". That is his view from his work with products, and not a measured result.

The lab's pipeline

My lab shrinks this pattern to six stages: propose candidates, check they are valid, verify the answer with the task's verifier, de-duplicate, filter by difficulty, and measure diversity. The generator is a toy. It knows which candidates it made bad on purpose (wrong answer, empty, a copy, a reworded copy), so the demo can check that the filters caught them. That hidden label does not exist in real life. The stage design is mine. Self-Instruct's abstract says only that invalid or similar samples are filtered [6], and it does not give my thresholds.

In the demo, 400 candidates become 334 after the validity and verifier checks (24 invalid and 42 wrong-answer candidates are rejected), 181 after de-duplication, and 158 after the difficulty filter. By the generator's hidden labels, all 158 survivors are clean. Two design points are worth thinking about.

• De-duplicate on the payload, not the template. Tasks from one template share most words, so a whole-prompt similarity check would throw them all away. The lab compares only the part of the prompt that varies (the payload). This is my design, and a real system needs something better, for example embeddings plus a human look.
• The difficulty filter is noisy. It keeps tasks that the current agent passes at least once and not every time out of k tries. An easy task, with a 0.9 pass chance per try, passes all 8 tries 43% of the time (my arithmetic, 0.9 to the power 8), so it is dropped only sometimes. A hard task with a 0.05 chance fails all 8 tries 66% of the time. So the filter also drops some tasks you wanted. Raising k reduces the noise and increases the cost.

The diversity check gives a lesson as well. The survivors have 6 task types and 6 distinct openings, while the 400 proposals had 6 types and 15 openings. The type entropy falls from 2.579 to 2.374. In this toy the filtered set is smaller and a little less varied. Filters that remove noise also remove some variety, so measure diversity after filtering and not only before.

2.6 Data flywheels

The required reading is Shreya Shankar's post "Data Flywheels for LLM Applications" (July 1, 2024) [1]. It is an essay about how to improve LLM applications using production data. It is not a paper, it has no experiments, and it calls its pipeline architecture "idealized". The post starts from four observations: humans must stay in the loop because preferences change; fine-tuning has a lot of overhead so many teams prefer LLM APIs; LLM-as-a-judge is increasingly used; and few-shot examples are the most effective way to improve a prompt, so a judge can be aligned with human preferences through examples [1]. It then gives a three-part framework.

1. Evaluation. Find the metrics that matter by looking at real outputs. Implement them as code or as an LLM judge, preferably binary (true or false), which is easier to align and for people to judge consistently. In a graph of LLM calls, place validators at nodes like probes in software observability, and validate inputs as well as outputs [1].
2. Monitoring. Metrics drift, so reassess them. Regularly sample and label production responses, store them with timestamps, and keep LLM-judge prompts dynamic by retrieving relevant labeled examples as few-shot examples. The post suggests weighting by recency and prioritising examples where human labels differed from the judge, and says it does not know how much lift that gives [1].
3. Continual improvement. Keep logs of outputs and their metric scores. Review and fix low-scoring outputs, keep the fixed traces in a database, and retrieve the most similar fixed traces as few-shot examples at run time [1].

Two features of the post matter for this course. First, the improvement is mostly not training. In the post the loop updates prompts, validators and retrieved examples, because the author expects teams to use APIs. The lecture's setting is the opposite: the improvement step changes weights. Husain's version has both, with fine-tuning on data that the eval system already curates [3]. In the lab's flywheel the improvement step is mock SFT. That is my mapping, and the post does not describe it. Second, the post's footnotes name risks: a legal team may want to sign off on prompts, and an adversary could write queries that become few-shot examples (prompt poisoning) [1].

What the lab's flywheel shows

Each round the mock agent serves 80 sessions, logs them, gets a budget of 50 labels (the lab picks 50 random traces and labels them with the verifier's reward), keeps the passed and consented traces as SFT examples, and "trains" by a made-up rule. Columns: online is the pass rate of that round's live traffic, eval_all is the expected score on a fixed 60-case eval set, and eval_clean is the same score without the eval prompts that the agent has seen in training. All of this is made up.

Variant (lab demo, made up)	eval_clean, round 0	eval_clean, round 5	What it shows
Base: label 50, train on passes	0.68	0.90	Labels turn logs into gains. eval_all reads 0.93 at round 5, because memorized prompts inflate it.
No filter: train on failures too	0.65	0.84	Training on wrong runs costs about six points here.
30% label noise	0.67	0.86	Noisy labels slow the gain but do not stop it, in this toy.
No labels at all	0.63	0.63	The loop turns, and nothing improves. Online pass rate drifts from 0.75 to 0.71.

One more demo line: the agent is deployed in a second harness (JSON) after 6 rounds. If it trained on react traces only, its clean eval score is 0.540. If it trained on react and json traces, it is 0.891. The transfer penalty between harnesses is a made-up constant in the lab (0.6), so the size of the gap means nothing. The direction is the slides' point: a model trained in one harness keeps that harness's conventions, and the fix the slides give is to vary the harness during training [5].

2.7 Collecting data from human-agent interaction

People who use your agent produce data about it: what they asked, what they accepted, and what they changed. This is the source that no one else can copy, and the Nature paper on model collapse says so: the value of data about genuine human interactions with systems will rise when model-made content fills the web [25]. That is an argument from the abstract, and not a measurement.

Explicit and implicit feedback

• Explicit. Thumbs up or down, a pairwise vote, a written correction. Clear, but rare. Chatbot Arena is the large public example: users compare model answers in pairs and vote, and the platform ranks models with statistical methods. Its abstract reports over 240,000 votes and good agreement with expert raters [26]. The lab includes a Bradley-Terry fit, the standard way to turn pairwise votes into strengths (see 2.10).
• Edits. If a user changes the agent's draft, the edit shows what they wanted. PRELUDE studies learning from such edits in writing tasks. It infers a description of the user's latent preference from historical edits and uses it in later prompts, so there is no fine-tuning [27]. The abstract says this suits a setting where the edit is a natural by-product of use.
• Implicit. Copying an answer, regenerating it, rephrasing the question, abandoning the session. Common, but noisy. The meaning is a guess. A copy could mean "good". A regenerate could mean "bad" or "I want a different style". The slides do not cover implicit signals, and the mapping in the lab is my own choice.

Weak labels from signals

The lab turns the signals of a trace into a score, then into a weak label (good, bad or none), and compares the label with the verifier. In the demo on 120 mock traces, the weak label exists for 72.5% of traces (87 of 120). Of the labelled traces, 98% of "good" calls are truly good (50 of 51), and 87.7% of the truly good labelled traces are called good (50 of 57), for 90.8% accuracy (79 of 87). These are my arithmetic on the lab's counts (tp 50, fp 1, fn 7, tn 29). The mock user simulator creates the signals from the hidden truth with fixed probabilities that I chose (after a passing run: copy 0.45, thumbs up 0.20, thumbs down 0.05, nothing 0.30; after a failing run: regenerate 0.25, edit 0.25, thumbs down 0.15, abandon 0.10, rephrase 0.10, copy 0.05, nothing 0.10), so these figures are high by construction. A real product has no verifier on every trace, so you can only check weak labels on a small sample that people label. That is the monitoring step of the flywheel [1].

Public interaction logs

WildChat released about one million user-ChatGPT conversations (over 2.5 million turns). The authors offered free access in exchange for users' consent to collect their chats anonymously. Its abstract says it has the most diverse prompts of the datasets they compared and a rich variety of potentially toxic uses [28]. Two lessons for builders. Opt-in consent can be designed into the product. And a log of real use is a different distribution from a benchmark, so it finds failures a benchmark does not.

Interaction data has biases the lab does not model. Users who give feedback are not typical users. People may click thumbs up for style and not for correctness, and the slides cite a case where a model trained on human approval looked right more often without being right more often (I did not open the source). Section 2.8 returns to this.

2.8 What the data teaches by accident

"Train on the traces and the model gets good" is not the whole story, and the slides spend their last section on the gotchas [5]. Four are worth knowing.

1. Reward hacking: the environment rewards a loophole

If the check can be passed without doing the work, a trained model will find that out. The slides' examples [5]:

• SWE-bench tasks can be solved by reading the repository's .git history, if it is not removed. The slides credit an issue in the SWE-bench repository, and a self-play SWE-RL paper that wipes the history and restores the test files before grading. I did not open either.
• Models that learned to call sys.exit(0) to fake passing tests. The slides cite an Anthropic post [35], which I read. The setup is more specific than the slide suggests. The authors added documents describing reward hacks to the pretraining data of a model, then trained it with RL on real programming tasks that they knew were vulnerable. The model learned to hack, and at the point it learned, the post reports a sharp rise in its misalignment evaluations, although it was never trained to behave that way [35]. This is the authors' own study of a deliberately vulnerable set-up. It is not a measurement of how often hacks arise in ordinary training.
• A reasoning-trace monitor that caught hacking, and a model that learned to hide its intent when the monitor was added to the reward. The slides cite an OpenAI post. I did not open it.

For this course the lesson is practical. Treat the verifier as part of the data. Test it with a cheating agent, as you did in Week 6a, before you train against it.

2. Memorization: the test is in the training data

Section 2.3 gave the OpenAI SWE-bench Verified case [34]. The slides add a study (The SWE-Bench Illusion) where models name the buggy file from the issue text alone 76% of the time on SWE-bench Verified, against up to 53% on repositories outside it. I did not open it. Treat both as the slides' report.

3. Overfit to the harness

A model trained on traces from one scaffold learns that scaffold's habits. The slides say SFT teaches both the capability and the harness, "without being able to tell which part is which", and that a model trained in one harness keeps its conventions when deployed in another. The fix they give is to vary the harness during training [5]. The lab's flywheel shows the shape of this in a toy: a mock agent trained on react traces only does worse when deployed in a JSON harness than one trained on both (0.540 against 0.891, with a made-up transfer penalty).

4. Training trends: SFT overfit, RL overshoot, the best model is not the best teacher

The slides report three findings, all from papers I did not read in full [5]:

• In SWE-RL, RL improved five out-of-domain tasks, while SFT on the same seed data made them worse on average. The SWE-RL abstract says the method trains on software evolution data with a rule-based reward, and that the model gained general reasoning skills [23]. I did not check the SFT comparison.
• In OpenThoughts-Agent, the best RL run peaked and then collapsed as the policy over-explored and timed out, and the team shipped the checkpoint from before the collapse.
• Also in OpenThoughts-Agent, trajectories from a newer, stronger model trained a worse student than trajectories from an older one. "Best model is not the best teacher".

Model collapse

One more accident comes from synthetic data. The Nature paper by Shumailov and colleagues reports that "indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear", and that this "model collapse" occurs in language models and in two smaller kinds of model [25]. I read the article. I did not check its derivations or rerun any experiment.

The lab has a toy. A six-item distribution (0.50, 0.25, 0.15, 0.06, 0.03, 0.01) is sampled 40 times, and the next generation is the sample's frequencies. With no real data mixed in, the number of items that still have any probability falls from 6 to 2 over 12 generations, and the entropy falls from 1.872 to 0.811. With 20% of the original distribution mixed in each time, all six items survive. This is a toy, and it is not a language model. It shows one mechanism: a rare item that is not drawn in one generation can never come back. The chance that the rarest item (1%) is missed in 40 draws is 0.99 to the power 40, about 67% (my arithmetic). Rules of thumb that follow, which are my reading and not the paper's result: keep real data in the mix, keep the tails, and measure diversity every round.

2.9 Privacy, consent and data cards

Interaction logs hold people's words. Three things follow, and the lab has code for each. I am not a lawyer, and this section is not compliance advice. The lab is a teaching sketch.

1. Consent by purpose. The lab's trace has three flags: log, train and share. The defaults are log on, and train and share off. That default is my choice. In the demo, 93 of 120 owners allow training and 42 allow sharing, so the training set and the shareable set are different sets. WildChat shows one real design: free access in exchange for opt-in consent to collect chats [28].
2. Scrub before you store or train. Models can repeat their training data. Carlini and colleagues extracted hundreds of verbatim sequences from GPT-2, including names, phone numbers and email addresses, and found larger models more vulnerable (abstract level) [32]. The lab's scrubber replaces emails, phone numbers, card numbers (checked with the Luhn rule), social security numbers, API keys that look like sk-, IP addresses and a list of names you give it. The same value gets the same placeholder, so a conversation still makes sense. The demo shows two limits. A street address passes through. And scrubbing can destroy the example: a task about "the domain of this email address" loses its email, and the example is no longer a good one. The scrubber masks a card number only if it passes the Luhn check, so an order number is kept, and a mistyped real card number would be kept too. A scrubber is a filter with false negatives, so the lab also runs a residual check after scrubbing, and that check uses the same patterns, so it cannot find what the patterns miss.
3. Retention and deletion. Keep logs only as long as you need them, and be able to find one person's data. The lab's retention filter drops traces older than a limit (the demo keeps 60 of 120 with a 30-minute limit, checked at second 3,600). The deletion function removes every trace of one pseudonymized user (9 removed, 111 left). A copy of the data in a trained model is not deleted by this. That is a hard problem, and the lab does not touch it.

Week 8 of the course covers privacy risks of agents with tool access [4], so I keep this short. The lab's job here is to make the data path visible: what is kept, why, and for how long.

Document the data

Datasheets for datasets, proposed by Gebru and colleagues, ask that every dataset has a document covering motivation, composition, collection process, recommended uses and so on, to improve communication between creators and users [33]. I read the abstract. The lab's data card is a small, machine-checkable version, and the design is mine. It records the purpose, the counts (traces, users, task types, harnesses), the share labelled and the mean reward, the time range, privacy and consent counts, the splits and the result of the leakage check, the sources and filters, whether synthetic data is present, known limits, and "not for" uses. A validator rejects a card that is missing a field, or that records a leakage report that is not clean, or that lists unscrubbed traces next to a share consent. The point is that a reader can see, in one page, what the data is and what it must not be used for.

2.10 Worked arithmetic

All numbers in this section are my own arithmetic, except where a source is named. Where the lab prints a number, I say so.

A. How likely is MinHash banding to find a pair?

The lab splits a 64-number signature into b = 16 bands of r = 4 numbers. Two texts become candidates if at least one band matches in all rows. If their Jaccard similarity is s, one number matches with probability about s, a band with s⁴, and at least one band with 1 − (1 − s⁴)¹⁶.

Jaccard s	s4	Candidate probability
0.80	0.410	0.9998
0.50	0.0625	0.644
0.30	0.0081	0.122
0.167 (8 of 48)	0.00077	0.012

So at the lab's default threshold of 0.8, a near copy is almost always a candidate, and an unrelated pair almost never is. The last row is the pair used in one of the lab's tests (8 shared shingles out of 48, so 8 / 48 = 0.167). The exact Jaccard is then checked for each candidate, so a candidate that is not similar enough is dropped. Banding can miss pairs between 0.5 and 0.8, and that is why the lab tests it against a brute-force check.

B. How often does the difficulty filter keep a task?

With k = 8 tries and a per-try pass chance p, a task is kept when the passes are not 0 and not 8. The keep probability is 1 − p⁸ − (1 − p)⁸.

Per-try pass chance p	All pass	All fail	Kept
0.90 (easy)	0.430	0.000	0.570
0.50	0.004	0.004	0.992
0.05 (hard)	0.000	0.663	0.337
0.01 (very hard)	0.000	0.923	0.077

The filter prefers middle-difficulty tasks. It does not remove easy tasks reliably, and it removes most very hard ones, which may be the most valuable if they are solvable.

C. From votes to strengths: Bradley-Terry

The Bradley-Terry model says that the chance that A beats B is s_A / (s_A + s_B). The lab fits strengths to 24 made-up votes and gets a = 2.706, b = 0.816, c = 0.453. Then P(a beats b) = 2.706 / 3.522 = 0.768, P(b beats c) = 0.816 / 1.269 = 0.643, and P(a beats c) = 2.706 / 3.159 = 0.857. Only the ratios matter. The lab scales the strengths in its own way. Chatbot Arena uses statistical methods for ranking from pairwise votes [26]. I did not check which fit it uses.

D. Per-sample cost, as the authors quote it

AgentTrek says $0.55 per trajectory [17]. Synatra says $0.031 per demonstration [18]. The ratio is 0.55 / 0.031 = 17.7 (my arithmetic). For 100,000 samples that is $55,000 against $3,100 for generation alone. The two numbers do not measure the same thing. One includes live execution on real screens and the other writes text offline. They also exclude the cost of training, checking and failed attempts. Do not read the ratio as a ranking of quality.

E. The label budget in the lab

Each round has 80 sessions and 50 labels, so 50 / 80 = 62.5% of traces are labelled. Over 6 rounds that is 300 labels for 480 traces. Only the labelled traces that pass and have consent become SFT data, so the SFT counts in the demo (28 to 41 per round) are well below 50.

2.11 Week 6b in one page

1. Two kinds of data. Trajectories (SFT; the model learns thoughts and actions) and environments (RL; task, sandbox, verifier) [5].
2. Traces are the raw material. From one trace you can make a demonstration, a preference pair and an eval case. Check consent and validity first.
3. Keep optimization and evaluation apart. Split by group and time, check for exact and near copies, and record what you did [31] [34].
4. Five sources. Humans, mined artifacts, synthesized trajectories, synthesized tasks and environments, and simulators. Ask about cost, realism and scale [5].
5. Synthetic data needs a verifier and filters. Propose, validate, verify, de-duplicate, filter by difficulty, then measure diversity [6] [10] [3].
6. A flywheel needs labels. Evaluate, monitor, improve. Without labels, the loop turns and nothing gets better [1].
7. People are a source of data. Explicit feedback is clear and rare. Implicit feedback is common and noisy. Measure weak labels against a sample that people check [26] [27].
8. Data teaches by accident. Reward hacks, one harness, memorization, SFT overfit, RL overshoot, model collapse [5] [25] [35].
9. Log less, scrub early, and write it down. Consent by purpose, retention, deletion and a data card [33].

3. Reading guide

The course lists one required reading and one additional reading for this week [4]. For each you get: why it matters, the key ideas, and questions to think about as you read. Read the questions first. The optional table at the end points to the sources that I used for the notes. For most of those I read only the abstract, and the references say which. Remember that the required reading is an essay with no experiments, so its claims are the author's experience and ideas.

Optional readings

Source	Why it is worth your time
Husain, Your AI Product Needs Evals [3]	The practical companion to Shankar: unit-test evals, looking at traces, and curating data once evals exist. I read the sections on traces and on fine-tuning and synthesis.
Self-Instruct [6]; Alpaca [7]; WizardLM [8]	The bootstrapping line of work. Read the Alpaca post in full. It is short and it states its limits.
APIGen [10]; ToolACE [11]; APIGen-MT [12]	Verified synthetic data for function calling. Compare their checks.
AgentInstruct (Mitra et al.) [9]; AgentTuning [13]; FireAct [14]	Synthetic and recorded agent trajectories for tuning. Mind the name clash between the two AgentInstruct projects.
Mind2Web [15]; WebArena [16]	A recorded web dataset and a runnable web environment.
AgentTrek [17]; Synatra [18]; NNetNav [19]; OS-Genesis [20]	Four ways to make web and GUI trajectories. The slides compare two of them.
SWE-Gym [21]; SWE-smith [22]; SWE-RL [23]	Code environments for training. SWE-smith is next week's reading.
Shumailov et al., model collapse [25]	The Nature paper. Read the abstract and the introduction, then ask what "tails" means for an agent.
Chatbot Arena [26]; PRELUDE [27]; WildChat [28]; DPO [29]	Human preference at scale, learning from edits, public interaction logs, and the loss that uses preference pairs.
Anthropic on reward hacking [35]; OpenAI on SWE-bench Verified [34]	Two vendor posts on what data teaches by accident and what leaks. Vendor-authored.
Lee et al. on deduplication [31]; Carlini et al. on extraction [32]; Gebru et al. on datasheets [33]	The data hygiene background for sections 2.3 and 2.9.
OpenTelemetry GenAI conventions [30]; tau-bench [24]; Anthropic on evals [36]	A trace format in development, a simulated-user benchmark, and an eval guide.

4. Hands-on lab: an agent data toolkit

Time: about 4.5 hours. Goal: build and test a small toolkit for agent data: a trace schema and logger, conversion of traces into demonstrations, preference pairs and eval cases, leakage checks and splits, a flywheel on a mock, a synthetic task generator with filters, implicit feedback signals, PII scrubbing with consent, and a data card. You do not need an API key, a model or any package. The lab never touches the network. The "agent" is a mock with a skill table, and the "user" is a simulator.

What you will learn

• How a trace is stored, validated, saved as JSON Lines and folded into a span tree.
• How one set of traces becomes SFT demonstrations, preference pairs (by outcome and by edit) and eval cases, and how consent gates each one.
• How MinHash and banding find near copies, how group and time splits differ from a random split, and how a leakage report and a decontamination step work.
• How a flywheel round works (deploy, log, label, improve), and what changes with no labels, noisy labels, no filter, or a second harness.
• How a synthetic task generator is checked by valid, verified, de-duplicate and difficulty filters, and how to measure diversity after filtering.
• How implicit signals become weak labels, how to score them against a verifier, and how a Bradley-Terry fit turns votes into strengths.
• How a scrubber, consent flags, retention and deletion work together, and how a data card records them.

Map of the code

Part	Main names in the code	What it does, and what it stands for
1	Event, Trace, TraceLogger, validate_trace, dumps_jsonl, to_span_tree	The trace schema, a logger with checks on order and closing, a validator, JSON Lines, and a span summary. Stands for traces in 2.2 [3] [30].
2	make_task, solve, MockAgent, UserSim, simulate_traffic	The mock world: six task types with verifiers, an agent with a skill table, a harness effect, memory of trained prompts, and a user who reacts. All made up.
3	traces_to_sft, traces_to_preferences, edit_pairs, traces_to_eval_cases, loss_tokens	The three products of a trace, with consent and validity gates (2.2) [29] [27].
4	shingles, minhash, lsh_candidates, near_duplicates, split_by_group, time_split, leakage_report, decontaminate	Copy checks and splits (2.3) [31].
5	train_on, FlywheelConfig, run_flywheel, collapse_demo	The mock flywheel and the model-collapse toy (2.6, 2.8) [1] [25].
6	propose_candidates, filter_valid, filter_verified, filter_dedupe, filter_difficulty, generate_pipeline, diversity	Synthetic task generation with filters (2.5) [6] [10].
7	extract_signals, signals_to_score, weak_label, signal_quality, bradley_terry	Implicit feedback and pairwise votes (2.7) [26] [27].
8	Scrubber, luhn_ok, scrub_trace, consent_filter, retention_filter, delete_user	PII scrubbing, consent by purpose, retention and deletion (2.9) [32].
9	make_data_card, validate_card, render_card	A data card, counted from the data where it can be (2.9) [33].

Setup

1. You need Python 3.10 or later. There is nothing to install. I ran it on Python 3.13 only.
2. Save the code below as agent_data_toolkit.py. Or download it: agent_data_toolkit.py.
3. Save the tests as test_agent_data_toolkit.py in the same folder. Or download: test_agent_data_toolkit.py. The test code is not printed on this page, to keep the page shorter. It is in the download, and the names of all the tests are listed after the exercises.
4. Run python agent_data_toolkit.py for the nine-section demo, and python test_agent_data_toolkit.py for the tests. Both are deterministic. The tests take about three seconds. The demo takes under a second.

The code: agent_data_toolkit.py

"""agent_data_toolkit.py - Week 6b lab: data for agents, in one readable file.

Standard library only (tested on Python 3.13). Nothing here calls a model, a network or a
subprocess. The "agent" is a mock with a made-up skill table, the "users" are a simulator, and
every task, rate, probability and price is MADE UP. The code shows the *shape* of the data
work (traces, demonstrations, preference pairs, eval cases, splits, flywheels, synthetic tasks,
implicit feedback, privacy, data cards). It is not a method from any paper, and no number it
prints is a result about a real model.

Parts
  1  Trace schema, logger, validation, JSON Lines, a span tree
  2  A mock world: six task types, a mock agent, a user simulator
  3  Trace conversion: SFT demonstrations, preference pairs, eval cases
  4  Splits and leakage checks: shingles, MinHash and LSH, group split, time split, report
  5  Flywheel simulation (deploy, log, label, improve) and a model-collapse toy
  6  Synthetic task generation with filters: valid, verified, dedupe, difficulty
  7  Implicit feedback signals, weak labels, a Bradley-Terry fit for pairwise votes
  8  PII scrubbing, consent flags, retention and deletion
  9  A data card generator
"""
from __future__ import annotations

import difflib
import hashlib
import json
import math
import random
import re
import sys
from dataclasses import dataclass, field, replace


# --------------------------------------------------------------------------- 0. shared bits
def stable_hash(*parts, bits: int = 64) -> int:
    """A hash that does not change between runs (Python's built-in hash() does)."""
    h = hashlib.sha256("\x1f".join(str(p) for p in parts).encode()).hexdigest()
    return int(h[: bits // 4], 16)


def rng_for(*parts) -> random.Random:
    return random.Random(stable_hash(*parts))


_WORD = re.compile(r"[a-z0-9']+")


def words(text) -> list[str]:
    return _WORD.findall(str(text).lower())


def mean(xs) -> float:
    xs = list(xs)
    return sum(xs) / len(xs) if xs else 0.0


def entropy(counts) -> float:
    """Shannon entropy in bits of a list or dict of counts."""
    vals = [c for c in (counts.values() if isinstance(counts, dict) else counts) if c > 0]
    total = sum(vals)
    return -sum(c / total * math.log2(c / total) for c in vals) if total else 0.0


def count_by(items, key) -> dict:
    out: dict = {}
    for it in items:
        out[it[key]] = out.get(it[key], 0) + 1
    return out


# --------------------------------------------------------------------------- 1. traces
EVENT_KINDS = ("system", "user", "assistant", "tool_call", "tool_result", "feedback")
FEEDBACK_TYPES = ("thumbs_up", "thumbs_down", "regenerate", "edit", "copy", "abandon")
DEFAULT_CONSENT = {"log": True, "train": False, "share": False}


class TraceError(ValueError):
    pass


@dataclass
class Event:
    kind: str
    text: str
    ts: float
    meta: dict = field(default_factory=dict)

    def to_dict(self) -> dict:
        return {"kind": self.kind, "text": self.text, "ts": self.ts, "meta": dict(self.meta)}

    @staticmethod
    def from_dict(d: dict) -> "Event":
        return Event(d["kind"], d["text"], d["ts"], dict(d.get("meta", {})))


@dataclass
class Trace:
    trace_id: str
    session_id: str
    user_id: str
    task_type: str
    harness: str
    model: str
    started: float
    events: list = field(default_factory=list)
    reward: float | None = None
    verifier: dict | None = None
    consent: dict = field(default_factory=lambda: dict(DEFAULT_CONSENT))
    scrubbed: bool = False

    def prompt(self) -> str:
        for e in self.events:
            if e.kind == "user":
                return e.text
        return ""

    def final_text(self) -> str:
        for e in reversed(self.events):
            if e.kind == "assistant":
                return e.text
        return ""

    def feedback(self) -> list:
        return [e for e in self.events if e.kind == "feedback"]

    def to_dict(self) -> dict:
        return {"trace_id": self.trace_id, "session_id": self.session_id, "user_id": self.user_id,
                "task_type": self.task_type, "harness": self.harness, "model": self.model,
                "started": self.started, "events": [e.to_dict() for e in self.events],
                "reward": self.reward, "verifier": self.verifier, "consent": dict(self.consent),
                "scrubbed": self.scrubbed}

    @staticmethod
    def from_dict(d: dict) -> "Trace":
        return Trace(d["trace_id"], d["session_id"], d["user_id"], d["task_type"], d["harness"],
                     d["model"], d["started"], [Event.from_dict(e) for e in d["events"]],
                     d.get("reward"), d.get("verifier"), dict(d.get("consent", DEFAULT_CONSENT)),
                     bool(d.get("scrubbed", False)))


class TraceLogger:
    """Collects traces in memory. start() opens one, log() appends, end() closes it."""

    def __init__(self, prefix: str = "t"):
        self.prefix = prefix
        self._traces: dict[str, Trace] = {}
        self._closed: set[str] = set()
        self._n = 0

    def start(self, session_id, user_id, task_type, harness, model, ts, consent=None) -> str:
        self._n += 1
        tid = f"{self.prefix}{self._n:05d}"
        c = dict(DEFAULT_CONSENT)
        c.update(consent or {})
        self._traces[tid] = Trace(tid, session_id, user_id, task_type, harness, model, float(ts), consent=c)
        return tid

    def log(self, trace_id, kind, text, ts, **meta) -> None:
        if trace_id not in self._traces:
            raise TraceError(f"unknown trace {trace_id}")
        if trace_id in self._closed:
            raise TraceError(f"trace {trace_id} is closed")
        if kind not in EVENT_KINDS:
            raise TraceError(f"unknown event kind {kind!r}")
        tr = self._traces[trace_id]
        last = tr.events[-1].ts if tr.events else tr.started
        if ts < last:
            raise TraceError("timestamps must not go backwards")
        tr.events.append(Event(kind, str(text), float(ts), dict(meta)))

    def end(self, trace_id, reward=None, verifier=None) -> Trace:
        if trace_id not in self._traces:
            raise TraceError(f"unknown trace {trace_id}")
        if trace_id in self._closed:
            raise TraceError(f"trace {trace_id} is already closed")
        if reward is not None and not 0.0 <= reward <= 1.0:
            raise TraceError("reward must be between 0 and 1")
        tr = self._traces[trace_id]
        tr.reward, tr.verifier = reward, verifier
        self._closed.add(trace_id)
        return tr

    def all(self) -> list:
        return list(self._traces.values())

    def open_ids(self) -> list:
        return [t for t in self._traces if t not in self._closed]


def validate_trace(tr: Trace) -> list[str]:
    """Return a list of problems. An empty list means the trace is well formed."""
    problems = []
    if not tr.events:
        return ["no events"]
    pending = 0
    last_ts = tr.started
    seen_first = False
    for i, e in enumerate(tr.events):
        if e.kind not in EVENT_KINDS:
            problems.append(f"event {i}: unknown kind {e.kind!r}")
            continue
        if e.ts < last_ts:
            problems.append(f"event {i}: timestamps go backwards")
        last_ts = max(last_ts, e.ts)
        if e.kind != "system" and not seen_first:
            seen_first = True
            if e.kind != "user":
                problems.append("first message is not from the user")
        if e.kind == "tool_call":
            pending += 1
        elif e.kind == "tool_result":
            if pending == 0:
                problems.append(f"event {i}: tool_result without tool_call")
            else:
                pending -= 1
        elif e.kind == "feedback" and e.meta.get("type") not in FEEDBACK_TYPES:
            problems.append(f"event {i}: bad feedback type {e.meta.get('type')!r}")
    if pending:
        problems.append("ends with an unanswered tool_call")
    if tr.reward is not None and not 0.0 <= tr.reward <= 1.0:
        problems.append("reward outside 0..1")
    return problems


def dumps_jsonl(traces) -> str:
    return "".join(json.dumps(t.to_dict(), sort_keys=True) + "\n" for t in traces)


def loads_jsonl(text: str) -> list:
    return [Trace.from_dict(json.loads(line)) for line in text.splitlines() if line.strip()]


def to_span_tree(tr: Trace) -> dict:
    """A loose, OpenTelemetry-flavoured view: one root span, one child per model or tool step.
    The names are this lab's own, not the official semantic-convention names."""
    kids = []
    for i, e in enumerate(tr.events):
        if e.kind not in ("assistant", "tool_call"):
            continue
        nxt = tr.events[i + 1].ts if i + 1 < len(tr.events) else e.ts
        kids.append({"name": "llm_call" if e.kind == "assistant" else "tool_call",
                     "start": e.ts, "end": nxt, "attributes": {"chars": len(e.text)}})
    end = tr.events[-1].ts if tr.events else tr.started
    return {"name": "session", "trace_id": tr.trace_id, "start": tr.started, "end": end,
            "attributes": {"harness": tr.harness, "model": tr.model, "reward": tr.reward},
            "children": kids}


def span_summary(root: dict) -> dict:
    out = {"llm_call": 0, "tool_call": 0, "seconds": round(root["end"] - root["start"], 3)}
    for k in root["children"]:
        out[k["name"]] += 1
    return out


# --------------------------------------------------------------------------- 2. a mock world
TYPES = ("sum", "reverse", "upper", "vowels", "sort", "domain")
HARNESSES = ("react", "json")
HARNESS_TRANSFER = 0.6      # MADE UP: skill kept when the harness differs from every training harness
MEMORY_SKILL = 0.98         # MADE UP: success rate on a prompt the mock agent has been trained on
INIT_SKILL = {"sum": 0.80, "reverse": 0.50, "upper": 0.90, "vowels": 0.35, "sort": 0.55, "domain": 0.70}
_NAMES = ["ada", "grace", "linus", "alan", "edsger", "barbara", "ken", "margaret"]
_DOMAINS = ["example.org", "mail.test", "corp.example", "uni.example"]
_WORDLIST = ["banana", "queue", "rhythm", "education", "sky", "avocado", "strength", "aeiou",
             "kitchen", "orchestra", "pencil", "umbrella"]
_COLORS = ["red", "green", "blue", "amber", "violet", "teal", "pink", "grey"]


@dataclass(frozen=True)
class WorldTask:
    task_id: str
    type: str
    prompt: str
    answer: str


def make_prompt(kind: str, rng: random.Random) -> str:
    if kind == "sum":
        return "Add the numbers: " + ", ".join(str(rng.randint(1, 20)) for _ in range(rng.randint(2, 4)))
    if kind == "reverse":
        return "Reverse the order of these words: " + " ".join(rng.sample(_COLORS, rng.randint(3, 4)))
    if kind == "upper":
        return "Write this text in uppercase: " + " ".join(rng.sample(_COLORS, 2))
    if kind == "vowels":
        return "Count the vowels in the word: " + rng.choice(_WORDLIST)
    if kind == "sort":
        return "Sort these numbers from smallest to largest: " + ", ".join(str(rng.randint(1, 30)) for _ in range(rng.randint(3, 5)))
    if kind == "domain":
        return f"What is the domain part of this email address: {rng.choice(_NAMES)}@{rng.choice(_DOMAINS)}"
    raise ValueError(f"unknown task type {kind!r}")


def solve(kind: str, prompt: str) -> str:
    """The reference solver: it reads the payload after the first ': ' and computes the answer."""
    if ": " not in prompt:
        raise ValueError("no payload")
    payload = prompt.split(": ", 1)[1].strip()
    if kind == "sum":
        return str(sum(int(x) for x in payload.split(",")))
    if kind == "reverse":
        return " ".join(reversed(payload.split()))
    if kind == "upper":
        return payload.upper()
    if kind == "vowels":
        return str(sum(ch in "aeiou" for ch in payload.lower()))
    if kind == "sort":
        return ", ".join(str(n) for n in sorted(int(x) for x in payload.split(",")))
    if kind == "domain":
        return payload.split("@", 1)[1]
    raise ValueError(f"unknown task type {kind!r}")


def make_task(kind: str, rng: random.Random, task_id: str) -> WorldTask:
    p = make_prompt(kind, rng)
    return WorldTask(task_id, kind, p, solve(kind, p))


def verifier_spec(task: WorldTask) -> dict:
    return {"type": "exact", "expected": task.answer}


def run_verifier(spec: dict, answer: str) -> float:
    if spec.get("type") != "exact":
        raise ValueError("unknown verifier type")
    return 1.0 if str(answer).strip() == str(spec["expected"]).strip() else 0.0


def corrupt(kind: str, answer: str, rng: random.Random) -> str:
    """A plausible wrong answer."""
    if answer.lstrip("-").isdigit():
        return str(int(answer) + rng.choice([-1, 1, 2]))
    if len(answer) > 1:
        return answer[:-1]
    return answer + "?"


def prompt_key(prompt: str) -> str:
    return hashlib.sha256(prompt.encode()).hexdigest()[:12]


class MockAgent:
    """A mock policy. skill[type] is the chance that one attempt is right. Not a model."""

    def __init__(self, skill=None, harness="react", model="mock-v0", trained_harnesses=None,
                 memorized=None, seed=0):
        self.skill = dict(skill or INIT_SKILL)
        self.harness = harness
        self.model = model
        self.trained_harnesses = set(trained_harnesses) if trained_harnesses else None
        self.memorized = set(memorized or ())
        self.seed = seed

    def effective_skill(self, task: WorldTask) -> float:
        p = self.skill[task.type]
        if self.trained_harnesses is not None and self.harness not in self.trained_harnesses:
            p *= HARNESS_TRANSFER
        if prompt_key(task.prompt) in self.memorized:
            p = max(p, MEMORY_SKILL)
        return p

    def attempts(self, task: WorldTask) -> list[str]:
        """One or two attempts. The second runs only after a wrong first one, with half the skill."""
        rng = rng_for("agent", self.seed, self.model, task.task_id)
        p = self.effective_skill(task)
        out = [task.answer if rng.random() < p else corrupt(task.type, task.answer, rng)]
        if out[0] != task.answer:
            out.append(task.answer if rng.random() < p * 0.5 else corrupt(task.type, task.answer, rng))
        return out

    def _action_text(self, task: WorldTask) -> str:
        if self.harness == "json":
            return json.dumps({"name": "solve", "arguments": {"input": task.prompt.split(": ", 1)[1]}})
        return f"solve[{task.prompt.split(': ', 1)[1]}]"

    def _thought_text(self, task: WorldTask, n: int) -> str:
        t = f"This is a {task.type} task." if n == 0 else "That looked wrong. I will try again."
        return json.dumps({"thought": t}) if self.harness == "json" else f"Thought: {t}"

    def run(self, task: WorldTask, logger: TraceLogger, user_id: str, session_id: str, ts: float,
            consent=None, user_sim=None) -> Trace:
        tid = logger.start(session_id, user_id, task.type, self.harness, self.model, ts, consent)
        t = float(ts)
        logger.log(tid, "system", f"You are a helpful assistant. Harness: {self.harness}.", t)
        t += 0.5
        logger.log(tid, "user", task.prompt, t, task_id=task.task_id)
        answer = ""
        for n, ans in enumerate(self.attempts(task)):
            t += 1.0
            logger.log(tid, "assistant", self._thought_text(task, n), t)
            t += 0.5
            logger.log(tid, "tool_call", self._action_text(task), t)
            t += 1.0
            logger.log(tid, "tool_result", ans, t)
            answer = ans
        t += 1.0
        logger.log(tid, "assistant", f"Answer: {answer}", t)
        reward = run_verifier(verifier_spec(task), answer)
        if user_sim is not None:
            user_sim.react(task, tid, logger, t, reward, answer)
        return logger.end(tid, reward, verifier_spec(task))


class UserSim:
    """Simulates how a user reacts. All probabilities are MADE UP."""
    OK = (("copy", 0.45), ("thumbs_up", 0.20), ("thumbs_down", 0.05))
    BAD = (("regenerate", 0.25), ("edit", 0.25), ("thumbs_down", 0.15), ("abandon", 0.10),
           ("rephrase", 0.10), ("copy", 0.05))

    def __init__(self, seed=0):
        self.seed = seed

    def react(self, task, trace_id, logger, ts, reward, answer) -> None:
        rng = rng_for("user", self.seed, task.task_id)
        r = rng.random()
        acc = 0.0
        pick = None
        for name, p in (self.OK if reward >= 1.0 else self.BAD):
            acc += p
            if r < acc:
                pick = name
                break
        t = ts + 2.0
        if pick is None:
            return
        if pick == "rephrase":
            logger.log(trace_id, "user", "again: " + task.prompt, t)
        elif pick == "edit":
            logger.log(trace_id, "feedback", f"Answer: {task.answer}", t, type="edit")
        else:
            logger.log(trace_id, "feedback", "", t, type=pick)


def simulate_traffic(n, seed=0, weights=None, agent=None, user_sim=None, start=0.0, n_users=12,
                     logger=None, session_prefix="s", gap=30.0) -> list:
    """Run n mock sessions. Tasks are drawn by weight, users by id, consent by user."""
    agent = agent or MockAgent(seed=seed)
    logger = logger or TraceLogger()
    rng = rng_for("traffic", seed)
    w = [weights.get(k, 0.0) for k in TYPES] if weights else [1.0] * len(TYPES)
    out = []
    for i in range(n):
        kind = rng.choices(TYPES, weights=w)[0]
        task = make_task(kind, rng, f"{session_prefix}{seed}-{i:04d}")
        uid = f"u{rng.randrange(n_users):02d}"
        out.append(agent.run(task, logger, uid, f"{session_prefix}{seed}-{i:04d}", start + i * gap,
                             consent=user_consent(uid), user_sim=user_sim))
    return out


def user_consent(user_id: str) -> dict:
    """MADE UP consent: about three users in four allow training, one in four allows sharing."""
    h = stable_hash("consent", user_id)
    return {"log": True, "train": h % 4 != 0, "share": h % 4 == 1}


# --------------------------------------------------------------------------- 3. conversion
def traces_to_sft(traces, min_reward: float = 1.0, require_consent: bool = True,
                  max_events: int | None = None) -> list:
    """Demonstrations. Assistant steps are learned (train=True). The prompt and tool output
    are context only. Failed, invalid, unconsented and too-long traces are dropped."""
    out = []
    for tr in traces:
        if tr.reward is None or tr.reward < min_reward or validate_trace(tr):
            continue
        if require_consent and not tr.consent.get("train"):
            continue
        evs = [e for e in tr.events if e.kind != "feedback"]
        if max_events is not None and len(evs) > max_events:
            continue
        msgs = []
        for e in evs:
            role = {"system": "system", "user": "user", "assistant": "assistant",
                    "tool_call": "assistant", "tool_result": "tool"}[e.kind]
            msgs.append({"role": role, "content": e.text, "train": e.kind in ("assistant", "tool_call")})
        out.append({"trace_id": tr.trace_id, "task_type": tr.task_type, "harness": tr.harness,
                    "reward": tr.reward, "started": tr.started, "user_id": tr.user_id,
                    "prompt": tr.prompt(), "messages": msgs})
    return out


def traces_to_preferences(traces, require_consent: bool = True) -> list:
    """Outcome pairs: for one prompt, the best trace is chosen and the worst is rejected.
    Needs a reward gap. Ties in reward go to the shorter trace."""
    groups: dict[str, list] = {}
    for tr in traces:
        if tr.reward is None or validate_trace(tr):
            continue
        if require_consent and not tr.consent.get("train"):
            continue
        groups.setdefault(tr.prompt(), []).append(tr)
    pairs = []
    for prompt in sorted(groups):
        g = groups[prompt]
        best = max(g, key=lambda t: (t.reward, -len(t.events), t.trace_id))
        worst = min(g, key=lambda t: (t.reward, -len(t.events), t.trace_id))
        if best.reward > worst.reward:
            pairs.append({"prompt": prompt, "chosen": best.final_text(), "rejected": worst.final_text(),
                          "source": "outcome", "chosen_trace": best.trace_id, "rejected_trace": worst.trace_id})
    return pairs


def edit_pairs(traces, require_consent: bool = True) -> list:
    """Edit pairs: the user's edit is chosen, the assistant's text it replaced is rejected."""
    pairs = []
    for tr in traces:
        if require_consent and not tr.consent.get("train"):
            continue
        for e in tr.feedback():
            if e.meta.get("type") == "edit" and e.text.strip() and e.text.strip() != tr.final_text().strip():
                pairs.append({"prompt": tr.prompt(), "chosen": e.text, "rejected": tr.final_text(),
                              "source": "edit", "chosen_trace": tr.trace_id, "rejected_trace": tr.trace_id})
                break
    return pairs


def traces_to_eval_cases(traces, only_failed: bool = True, max_per_type: int | None = None) -> list:
    """Eval cases: the prompt and its verifier, kept once per prompt. By default only failures."""
    seen, out, per_type = set(), [], {}
    for tr in traces:
        if tr.reward is None or tr.verifier is None or validate_trace(tr):
            continue
        if only_failed and tr.reward >= 1.0:
            continue
        p = tr.prompt()
        if p in seen:
            continue
        if max_per_type is not None and per_type.get(tr.task_type, 0) >= max_per_type:
            continue
        seen.add(p)
        per_type[tr.task_type] = per_type.get(tr.task_type, 0) + 1
        out.append({"case_id": "c-" + prompt_key(p), "task_type": tr.task_type, "prompt": p,
                    "verifier": dict(tr.verifier), "from_trace": tr.trace_id, "started": tr.started,
                    "user_id": tr.user_id})
    return out


def loss_tokens(example: dict) -> tuple[int, int]:
    """(learned words, context words) for one SFT example. A rough count, not tokens."""
    learned = sum(len(words(m["content"])) for m in example["messages"] if m["train"])
    context = sum(len(words(m["content"])) for m in example["messages"] if not m["train"])
    return learned, context


def harness_mix(examples) -> dict:
    out: dict[str, int] = {}
    for ex in examples:
        out[ex["harness"]] = out.get(ex["harness"], 0) + 1
    return dict(sorted(out.items()))


# --------------------------------------------------------------------------- 4. splits and leakage
def shingles(text: str, k: int = 3) -> set:
    w = words(text)
    if len(w) < k:
        return {" ".join(w)} if w else set()
    return {" ".join(w[i:i + k]) for i in range(len(w) - k + 1)}


def jaccard(a: set, b: set) -> float:
    if not a and not b:
        return 1.0
    return len(a & b) / len(a | b)


def minhash(sh: set, num_perm: int = 64, seed: int = 0) -> tuple:
    if not sh:
        return tuple([2 ** 64 - 1] * num_perm)
    return tuple(min(stable_hash("mh", seed, i, s) for s in sh) for i in range(num_perm))


def minhash_similarity(a: tuple, b: tuple) -> float:
    if len(a) != len(b):
        raise ValueError("signatures differ in length")
    return sum(x == y for x, y in zip(a, b)) / len(a)


def exact_key(text: str) -> str:
    """Hash of the text after lowercasing and collapsing spaces, for exact-duplicate checks."""
    return hashlib.sha256(" ".join(words(text)).encode()).hexdigest()[:16]


def lsh_candidates(texts: dict, k: int = 3, num_perm: int = 64, bands: int = 16, seed: int = 0) -> list:
    """Candidate pairs: two texts are candidates when their MinHash signatures agree on every
    row of at least one band. Cheap, and it can include pairs that are not really similar."""
    if num_perm % bands:
        raise ValueError("bands must divide num_perm")
    rows = num_perm // bands
    sig = {i: minhash(shingles(t, k), num_perm, seed) for i, t in texts.items()}
    buckets: dict[tuple, list] = {}
    for i in sorted(texts):
        for b in range(bands):
            buckets.setdefault((b, sig[i][b * rows:(b + 1) * rows]), []).append(i)
    cands = set()
    for ids in buckets.values():
        for x in range(len(ids)):
            for y in range(x + 1, len(ids)):
                cands.add((ids[x], ids[y]))
    return sorted(cands)


def near_duplicates(texts: dict, threshold: float = 0.8, k: int = 3, num_perm: int = 64,
                    bands: int = 16, seed: int = 0) -> list:
    """Find pairs whose shingle Jaccard is at least threshold. MinHash with banding picks the
    candidates, and the exact Jaccard decides. Returns sorted (id_a, id_b, jaccard) tuples."""
    sh = {i: shingles(t, k) for i, t in texts.items()}
    out = []
    for a, b in lsh_candidates(texts, k, num_perm, bands, seed):
        j = jaccard(sh[a], sh[b])
        if j >= threshold:
            out.append((a, b, round(j, 4)))
    return out


def brute_force_near_duplicates(texts: dict, threshold: float = 0.8, k: int = 3) -> list:
    ids = sorted(texts)
    sh = {i: shingles(texts[i], k) for i in ids}
    out = []
    for x in range(len(ids)):
        for y in range(x + 1, len(ids)):
            j = jaccard(sh[ids[x]], sh[ids[y]])
            if j >= threshold:
                out.append((ids[x], ids[y], round(j, 4)))
    return out


def split_by_group(items, group_key, fractions=(0.8, 0.1, 0.1), names=("train", "dev", "test"),
                   salt: str = "split") -> dict:
    """All items with the same group (a user, a task, a prompt) land in the same split."""
    if abs(sum(fractions) - 1.0) > 1e-9 or len(fractions) != len(names):
        raise ValueError("fractions must sum to 1 and match names")
    out = {n: [] for n in names}
    for it in items:
        u = (stable_hash(salt, group_key(it)) % 10 ** 6) / 10 ** 6
        acc = 0.0
        for n, f in zip(names, fractions):
            acc += f
            if u < acc or n == names[-1]:
                out[n].append(it)
                break
    return out


def split_random(items, fractions=(0.8, 0.1, 0.1), names=("train", "dev", "test"), seed: int = 0) -> dict:
    """A plain shuffle split. It ignores groups and time, so it can leak."""
    if abs(sum(fractions) - 1.0) > 1e-9 or len(fractions) != len(names):
        raise ValueError("fractions must sum to 1 and match names")
    idx = list(range(len(items)))
    random.Random(seed).shuffle(idx)
    out, start = {}, 0
    for n, f in zip(names, fractions):
        end = len(idx) if n == names[-1] else start + int(round(f * len(items)))
        out[n] = [items[i] for i in sorted(idx[start:end])]
        start = end
    return out


def time_split(items, ts_key, train_end: float, dev_end: float) -> dict:
    """Train before train_end, dev before dev_end, test from dev_end on."""
    if not train_end <= dev_end:
        raise ValueError("train_end must not exceed dev_end")
    out = {"train": [], "dev": [], "test": []}
    for it in items:
        t = ts_key(it)
        out["train" if t < train_end else "dev" if t < dev_end else "test"].append(it)
    return out


def leakage_report(splits: dict, text_key, group_key=None, ts_key=None, threshold: float = 0.8) -> dict:
    """Look for leaks from train into dev and test: exact copies, near copies, shared groups,
    and test items older than the newest training item."""
    rep = {"exact": [], "near": [], "groups": [], "time_violations": 0}
    train = splits.get("train", [])
    for other in [n for n in splits if n != "train"]:
        a = {f"train:{i}": text_key(x) for i, x in enumerate(train)}
        b = {f"{other}:{i}": text_key(x) for i, x in enumerate(splits[other])}
        ka = {}
        for i, t in a.items():
            ka.setdefault(exact_key(t), []).append(i)
        for i, t in b.items():
            for j in ka.get(exact_key(t), []):
                rep["exact"].append((j, i))
        near = near_duplicates({**a, **b}, threshold)
        exact = {tuple(sorted(p)) for p in rep["exact"]}
        for x, y, jv in near:
            tr_id, ot_id = (x, y) if x.startswith("train:") else (y, x)
            if tr_id.startswith("train:") and ot_id.startswith(other + ":") and tuple(sorted((x, y))) not in exact:
                rep["near"].append((tr_id, ot_id, jv))
        if group_key is not None:
            tg = {group_key(x) for x in train}
            for g in sorted({group_key(x) for x in splits[other]} & tg):
                rep["groups"].append((other, g))
        if ts_key is not None and train and splits[other]:
            newest = max(ts_key(x) for x in train)
            rep["time_violations"] += sum(1 for x in splits[other] if ts_key(x) <= newest)
    rep["ok"] = not (rep["exact"] or rep["near"] or rep["groups"] or rep["time_violations"])
    return rep


def decontaminate(train, test, text_key, threshold: float = 0.8) -> tuple:
    """Drop training items that are exact or near copies of a test item. Returns (kept, dropped)."""
    a = {f"a{i}": text_key(x) for i, x in enumerate(train)}
    b = {f"b{i}": text_key(x) for i, x in enumerate(test)}
    drop = set()
    kb = {exact_key(t) for t in b.values()}
    for i, t in a.items():
        if exact_key(t) in kb:
            drop.add(i)
    for x, y, _ in near_duplicates({**a, **b}, threshold):
        if x[0] != y[0]:
            drop.add(x if x[0] == "a" else y)
    kept = [x for i, x in enumerate(train) if f"a{i}" not in drop]
    dropped = [x for i, x in enumerate(train) if f"a{i}" in drop]
    return kept, dropped


# --------------------------------------------------------------------------- 5. the flywheel
def train_on(skill: dict, sft, lr: float = 0.5, scale: float = 15.0) -> dict:
    """The mock 'training' step (MADE UP rule). For each type, the skill moves toward 1 in
    proportion to how many examples there are (up to `scale`), and moves AWAY if most of the
    examples are wrong. Noisy data can make the mock agent worse."""
    new = dict(skill)
    for kind in skill:
        ex = [e for e in sft if e["task_type"] == kind]
        if not ex:
            continue
        q = mean(e["reward"] for e in ex)
        step = lr * min(1.0, len(ex) / scale) * (2 * q - 1)
        room = (1 - skill[kind]) if step > 0 else skill[kind]
        new[kind] = min(0.99, max(0.01, skill[kind] + step * room))
    return new


def make_eval_set(seed: int = 99, per_type: int = 10) -> list:
    rng = rng_for("evalset", seed)
    return [make_task(k, rng, f"e{seed}-{k}-{i:03d}") for k in TYPES for i in range(per_type)]


def expected_score(agent: MockAgent, cases, exclude_prompts=None) -> float:
    """Exact expected pass rate of one attempt (no sampling), optionally skipping prompts."""
    ex = exclude_prompts or set()
    xs = [agent.effective_skill(c) for c in cases if prompt_key(c.prompt) not in ex]
    return mean(xs)


@dataclass
class FlywheelConfig:
    rounds: int = 6
    sessions: int = 80
    label_budget: int = 50
    label_noise: float = 0.0
    lr: float = 0.5
    scale: float = 15.0
    harnesses: tuple = ("react",)
    deploy_harness: str = "react"
    filter_failures: bool = True
    use_consent: bool = True
    seed: int = 0


def label_traces(traces, budget: int, noise: float, rng: random.Random) -> dict:
    """A labeling budget: pick `budget` traces at random and label them with the verifier
    reward, flipped with probability `noise`. Returns {trace_id: label}."""
    ids = [t.trace_id for t in traces if t.reward is not None]
    rng.shuffle(ids)
    by = {t.trace_id: t for t in traces}
    out = {}
    for tid in ids[:max(0, budget)]:
        lab = by[tid].reward
        out[tid] = 1.0 - lab if rng.random() < noise else lab
    return out


def run_flywheel(cfg: FlywheelConfig, init_skill=None) -> dict:
    """Deploy -> log -> label -> improve, round after round, on the mock world."""
    skill = dict(init_skill or INIT_SKILL)
    memorized: set = set()
    trained_h: set = set()
    evalset = make_eval_set()
    hist, all_traces = [], []
    for r in range(cfg.rounds):
        h = cfg.harnesses[r % len(cfg.harnesses)]
        agent = MockAgent(skill, h, f"mock-r{r}", trained_h or None, memorized,
                          seed=stable_hash("fw", cfg.seed, r) % 10 ** 6)
        logger = TraceLogger(prefix=f"r{r}-")
        traces = simulate_traffic(cfg.sessions, seed=cfg.seed * 1000 + r, agent=agent, user_sim=UserSim(r),
                                  start=r * 10000.0, logger=logger, session_prefix=f"r{r}-")
        all_traces += traces
        labels = label_traces(traces, cfg.label_budget, cfg.label_noise, rng_for("label", cfg.seed, r))
        labelled = [replace(t, reward=labels[t.trace_id]) for t in traces if t.trace_id in labels]
        sft = traces_to_sft(labelled, min_reward=1.0 if cfg.filter_failures else 0.0, require_consent=cfg.use_consent)
        skill = train_on(skill, sft, cfg.lr, cfg.scale)
        memorized |= {prompt_key(e["prompt"]) for e in sft}
        trained_h |= {e["harness"] for e in sft}
        dep = MockAgent(skill, cfg.deploy_harness, "deploy", trained_h or None, memorized)
        hist.append({"round": r, "online": round(mean(t.reward for t in traces), 3), "logged": len(traces),
                     "labelled": len(labelled), "sft": len(sft),
                     "eval_all": round(expected_score(dep, evalset), 3),
                     "eval_clean": round(expected_score(dep, evalset, memorized), 3),
                     "eval_overlap": sum(prompt_key(c.prompt) in memorized for c in evalset)})
    return {"history": hist, "skill": skill, "memorized": memorized, "traces": all_traces}


def collapse_demo(probs: dict, n: int, generations: int, seed: int = 0, real_mix: float = 0.0) -> list:
    """A toy of recursive training. Each generation samples n items from the current
    distribution and the next distribution is their frequencies, mixed with the original at
    weight real_mix. Rare items vanish when n is small and real_mix is 0. Not a language model."""
    total = sum(probs.values())
    orig = {k: v / total for k, v in probs.items()}
    cur = dict(orig)
    rng = random.Random(seed)
    keys = sorted(orig)
    rows = []
    for g in range(generations + 1):
        rows.append({"generation": g, "support": sum(1 for k in keys if cur[k] > 0),
                     "entropy": round(entropy([cur[k] for k in keys]), 3), "p_rarest": round(cur[keys[-1]], 4)})
        if g == generations:
            break
        draw = rng.choices(keys, weights=[cur[k] for k in keys], k=n)
        freq = {k: draw.count(k) / n for k in keys}
        cur = {k: (1 - real_mix) * freq[k] + real_mix * orig[k] for k in keys}
    return rows


# --------------------------------------------------------------------------- 6. synthetic tasks
_PARAPHRASE = {"Add the numbers": "Sum these numbers",
               "Reverse the order of these words": "Put these words in reverse order",
               "Write this text in uppercase": "Convert this text to capital letters",
               "Count the vowels in the word": "How many vowels are in the word",
               "Sort these numbers from smallest to largest": "Order these numbers, smallest first",
               "What is the domain part of this email address": "Give the domain of this email address"}


def paraphrase_light(prompt: str) -> str:
    head, payload = prompt.split(": ", 1)
    return _PARAPHRASE.get(head, head) + ": " + payload


def propose_candidates(n: int, seed: int = 0, p_bad_answer: float = 0.10, p_invalid: float = 0.08,
                       p_near: float = 0.12, p_dup: float = 0.05) -> list:
    """Stand-in for 'ask a model for new tasks'. Most candidates are fine; some are broken on
    purpose so the filters have work. The `origin` field is the hidden truth, for scoring filters."""
    rng = rng_for("propose", seed)
    out = []
    for i in range(n):
        r = rng.random()
        kind = rng.choice(TYPES)
        t = make_task(kind, rng, f"g{seed}-{i:04d}")
        c = {"id": t.task_id, "type": kind, "prompt": t.prompt, "answer": t.answer, "origin": "clean"}
        if r < p_bad_answer:
            c["answer"], c["origin"] = corrupt(kind, t.answer, rng), "bad_answer"
        elif r < p_bad_answer + p_invalid:
            c["prompt"], c["origin"] = rng.choice(["", "Do it", "TODO {fill in}"]), "invalid"
        elif r < p_bad_answer + p_invalid + p_near and any(o["origin"] == "clean" for o in out):
            src = rng.choice([o for o in out if o["origin"] == "clean"])
            c.update(type=src["type"], prompt=paraphrase_light(src["prompt"]), answer=src["answer"], origin="near_dup")
        elif r < p_bad_answer + p_invalid + p_near + p_dup and any(o["origin"] == "clean" for o in out):
            src = rng.choice([o for o in out if o["origin"] == "clean"])
            c.update(type=src["type"], prompt=src["prompt"], answer=src["answer"], origin="dup")
        out.append(c)
    return out


def payload_words(prompt: str) -> set:
    return set(words(prompt.split(": ", 1)[1])) if ": " in prompt else set(words(prompt))


def filter_valid(c: dict) -> str | None:
    p = c["prompt"]
    if not p.strip():
        return "empty"
    if len(p) < 15 or len(p) > 200:
        return "length"
    if "{" in p or "TODO" in p:
        return "placeholder"
    return None


def filter_verified(c: dict) -> str | None:
    try:
        ok = solve(c["type"], c["prompt"]) == str(c["answer"]).strip()
    except (ValueError, IndexError):
        return "unparseable"
    return None if ok else "wrong_answer"


def filter_dedupe(cands, existing=(), threshold: float = 0.8) -> tuple:
    """Greedy: keep a candidate unless an earlier kept one, or an existing task, has the same
    type and payload overlap (Jaccard) at or above the threshold. Compares payloads, not
    templates, because the templates are shared by design."""
    kept, rejected = [], []
    pool = [(e["type"], payload_words(e["prompt"])) for e in existing]
    for c in cands:
        pw = payload_words(c["prompt"])
        if any(t == c["type"] and jaccard(pw, w) >= threshold for t, w in pool):
            rejected.append(c)
        else:
            kept.append(c)
            pool.append((c["type"], pw))
    return kept, rejected


def pass_count(c: dict, agent: MockAgent, k: int, seed: int = 0) -> int:
    """How many of k independent mock attempts get the claimed answer right."""
    n = 0
    for j in range(k):
        rng = rng_for("diff", seed, c["id"], j)
        n += rng.random() < agent.skill[c["type"]]
    return n


def filter_difficulty(cands, agent: MockAgent, k: int = 8, seed: int = 0) -> tuple:
    """Keep tasks that the agent sometimes solves and sometimes misses (0 < passes < k)."""
    kept, rejected = [], []
    for c in cands:
        n = pass_count(c, agent, k, seed)
        (kept if 0 < n < k else rejected).append(dict(c, passes=n))
    return kept, rejected


def generate_pipeline(n: int, agent: MockAgent, seed: int = 0, existing=(), k: int = 8,
                      dedupe_threshold: float = 0.8) -> dict:
    cands = propose_candidates(n, seed)
    funnel = [("proposed", len(cands))]
    rejects: dict[str, int] = {}
    ok = []
    for c in cands:
        why = filter_valid(c) or filter_verified(c)
        if why:
            rejects[why] = rejects.get(why, 0) + 1
        else:
            ok.append(c)
    funnel.append(("valid and verified", len(ok)))
    ok, dup = filter_dedupe(ok, existing, dedupe_threshold)
    rejects["duplicate"] = len(dup)
    funnel.append(("after dedupe", len(ok)))
    ok, hard = filter_difficulty(ok, agent, k, seed)
    rejects["too easy or too hard"] = len(hard)
    funnel.append(("after difficulty", len(ok)))
    return {"kept": ok, "funnel": funnel, "rejects": dict(sorted(rejects.items())),
            "origin_kept": dict(sorted(count_by(ok, "origin").items())),
            "origin_proposed": dict(sorted(count_by(cands, "origin").items()))}


def diversity(tasks) -> dict:
    """Distinct types, entropy of the type mix, and distinct instruction openings (first 3 words)."""
    types = count_by(tasks, "type")
    opens = {" ".join(words(t["prompt"])[:3]) for t in tasks}
    return {"n": len(tasks), "types": len(types), "type_entropy": round(entropy(types), 3), "openings": len(opens)}


# --------------------------------------------------------------------------- 7. implicit feedback
def extract_signals(tr: Trace) -> dict:
    """Pull implicit-feedback signals out of one trace."""
    counts = {t: 0 for t in FEEDBACK_TYPES}
    edit_ratio = 0.0
    for e in tr.feedback():
        t = e.meta.get("type")
        if t in counts:
            counts[t] += 1
        if t == "edit":
            edit_ratio = max(edit_ratio, 1.0 - difflib.SequenceMatcher(None, tr.final_text().strip(), e.text.strip()).ratio())
    users = [e for e in tr.events if e.kind == "user"]
    rephrase = any(jaccard(set(words(users[i].text)), set(words(users[j].text))) >= 0.6
                   for i in range(len(users)) for j in range(i + 1, len(users)))
    return {**counts, "edit_ratio": round(edit_ratio, 3), "rephrase": rephrase,
            "n_feedback": sum(counts.values())}


SIGNAL_WEIGHTS = {"copy": 0.6, "thumbs_up": 1.0, "thumbs_down": -1.0, "regenerate": -0.5,
                  "abandon": -0.8, "rephrase": -0.6, "edit": -0.8}


def signals_to_score(sig: dict) -> float:
    """Combine signals into a score in [-1, 1]. The weights are my own, not from any paper.
    Any edit counts for half of its weight, and a big rewrite counts in full."""
    w = SIGNAL_WEIGHTS
    s = w["copy"] * min(sig["copy"], 1) + w["thumbs_up"] * min(sig["thumbs_up"], 1)
    s += w["thumbs_down"] * min(sig["thumbs_down"], 1)
    s += w["regenerate"] * min(sig["regenerate"], 2)
    s += w["abandon"] * min(sig["abandon"], 1)
    s += w["rephrase"] * (1 if sig["rephrase"] else 0)
    if sig["edit"]:
        s += w["edit"] * (0.5 + 0.5 * sig["edit_ratio"])
    return max(-1.0, min(1.0, s))


def weak_label(score: float, hi: float = 0.4, lo: float = -0.4) -> int | None:
    """1 for good, 0 for bad, None when the signal is too weak to use."""
    if score >= hi:
        return 1
    if score <= lo:
        return 0
    return None


def signal_quality(traces, hi: float = 0.4, lo: float = -0.4) -> dict:
    """Compare weak labels to the verifier reward (known in the mock). Reports coverage,
    precision and recall of 'good', and accuracy of the labels that were produced."""
    tp = fp = fn = tn = 0
    for tr in traces:
        wl = weak_label(signals_to_score(extract_signals(tr)), hi, lo)
        if wl is None:
            continue
        truth = 1 if tr.reward >= 1.0 else 0
        if wl == 1 and truth == 1:
            tp += 1
        elif wl == 1:
            fp += 1
        elif truth == 1:
            fn += 1
        else:
            tn += 1
    n = len(traces)
    done = tp + fp + fn + tn
    return {"n": n, "coverage": round(done / n, 3) if n else 0.0,
            "precision_good": round(tp / (tp + fp), 3) if tp + fp else 0.0,
            "recall_good": round(tp / (tp + fn), 3) if tp + fn else 0.0,
            "accuracy": round((tp + tn) / done, 3) if done else 0.0, "tp": tp, "fp": fp, "fn": fn, "tn": tn}


def bradley_terry(votes, iters: int = 500) -> dict:
    """Strengths from pairwise votes [(winner, loser), ...] by the standard minorize-maximize
    update. Scaled so the geometric mean is 1. Needs every model to win at least once to be
    finite. P(a beats b) = s_a / (s_a + s_b)."""
    models = sorted({m for v in votes for m in v})
    wins = {m: 0 for m in models}
    n: dict = {}
    for w, l in votes:
        wins[w] += 1
        n[(w, l)] = n.get((w, l), 0) + 1
    p = {m: 1.0 for m in models}
    for _ in range(iters):
        new = {}
        for i in models:
            den = sum((n.get((i, j), 0) + n.get((j, i), 0)) / (p[i] + p[j]) for j in models if j != i)
            new[i] = wins[i] / den if den else p[i]
        g = math.exp(mean(math.log(max(v, 1e-12)) for v in new.values()))
        p = {m: v / g for m, v in new.items()}
    return p


# --------------------------------------------------------------------------- 8. privacy
def luhn_ok(digits: str) -> bool:
    ds = [int(c) for c in digits if c.isdigit()]
    if len(ds) < 13 or len(ds) > 19:
        return False
    total = 0
    for i, d in enumerate(reversed(ds)):
        if i % 2:
            d *= 2
            if d > 9:
                d -= 9
        total += d
    return total % 10 == 0


_PII = [
    ("EMAIL", re.compile(r"\b[\w.+-]+@[\w-]+(?:\.[\w-]+)+\b")),
    ("KEY", re.compile(r"\b(?:sk|pk|ghp|AKIA)[-_A-Za-z0-9]{16,}\b")),
    ("SSN", re.compile(r"\b\d{3}-\d{2}-\d{4}\b")),
    ("CARD", re.compile(r"\b(?:\d[ -]?){13,19}\b")),
    ("PHONE", re.compile(r"(?<![\w-])(?:\+?1[ .-]?)?\(?\d{3}\)?[ .-]\d{3}[ .-]\d{4}\b")),
    ("IP", re.compile(r"\b(?:(?:25[0-5]|2[0-4]\d|1?\d?\d)\.){3}(?:25[0-5]|2[0-4]\d|1?\d?\d)\b")),
]


class Scrubber:
    """Replaces PII with stable placeholders such as <EMAIL_1>. The same value always gets the
    same placeholder, so a story stays readable, but the value itself is gone. It only finds
    what the patterns and the name list describe. It will miss other names, street
    addresses and anything unusual."""

    def __init__(self, names=()):
        self.names = [n for n in names if n]
        self.map: dict[tuple, str] = {}
        self.counts: dict[str, int] = {}

    def _ph(self, kind: str, value: str) -> str:
        key = (kind, value.lower())
        if key not in self.map:
            n = sum(1 for k in self.map if k[0] == kind) + 1
            self.map[key] = f"<{kind}_{n}>"
        self.counts[kind] = self.counts.get(kind, 0) + 1
        return self.map[key]

    def scrub(self, text: str) -> str:
        out = str(text)
        for kind, pat in _PII:
            def sub(m, kind=kind):
                v = m.group(0)
                if kind == "CARD" and not luhn_ok(v):
                    return v
                return self._ph(kind, v)
            out = pat.sub(sub, out)
        for name in self.names:
            out = re.sub(r"\b" + re.escape(name) + r"\b", lambda m: self._ph("NAME", m.group(0)), out, flags=re.I)
        return out


def pseudonymize_user(user_id: str, salt: str) -> str:
    return "p-" + hashlib.sha256(f"{salt}|{user_id}".encode()).hexdigest()[:10]


def scrub_trace(tr: Trace, salt: str, names=()) -> Trace:
    """A scrubbed copy: one Scrubber per trace, user id replaced by a salted hash."""
    sc = Scrubber(names)
    evs = [Event(e.kind, sc.scrub(e.text), e.ts, {k: (sc.scrub(v) if isinstance(v, str) else v) for k, v in e.meta.items()})
           for e in tr.events]
    ver = None
    if tr.verifier is not None:
        ver = {k: (sc.scrub(v) if isinstance(v, str) else v) for k, v in tr.verifier.items()}
    return replace(tr, events=evs, verifier=ver, user_id=pseudonymize_user(tr.user_id, salt), scrubbed=True)


def residual_pii(text: str) -> list:
    """Kinds of PII that the patterns still find. Run it after scrubbing as a gate."""
    found = []
    for kind, pat in _PII:
        for m in pat.finditer(str(text)):
            if kind != "CARD" or luhn_ok(m.group(0)):
                found.append(kind)
    return found


def trace_residual_pii(tr: Trace) -> list:
    out = []
    for e in tr.events:
        out += residual_pii(e.text)
        out += [k for v in e.meta.values() if isinstance(v, str) for k in residual_pii(v)]
    return sorted(set(out))


def consent_filter(traces, purpose: str) -> tuple:
    """Keep traces whose owner allowed `purpose` ('log', 'train' or 'share'). For 'train' and
    'share' the trace must also be scrubbed. Returns (kept, dropped_reasons)."""
    if purpose not in DEFAULT_CONSENT:
        raise ValueError(f"unknown purpose {purpose!r}")
    kept, why = [], {"no_consent": 0, "not_scrubbed": 0}
    for t in traces:
        if not t.consent.get(purpose):
            why["no_consent"] += 1
        elif purpose in ("train", "share") and not t.scrubbed:
            why["not_scrubbed"] += 1
        else:
            kept.append(t)
    return kept, why


def retention_filter(traces, now: float, max_age_days: float) -> tuple:
    """Drop traces started more than max_age_days before `now` (timestamps are seconds)."""
    limit = now - max_age_days * 86400.0
    kept = [t for t in traces if t.started >= limit]
    return kept, len(traces) - len(kept)


def delete_user(traces, user_id: str, salt: str | None = None) -> tuple:
    """Remove every trace of one user. If traces were pseudonymized, pass the salt as well."""
    ids = {user_id} | ({pseudonymize_user(user_id, salt)} if salt else set())
    kept = [t for t in traces if t.user_id not in ids]
    return kept, len(traces) - len(kept)


# --------------------------------------------------------------------------- 9. data cards
CARD_FIELDS = ("name", "version", "purpose", "sources", "counts", "labels", "time_range", "harnesses",
               "privacy", "splits", "filters", "known_limits", "not_for")


def make_data_card(name: str, version: str, traces, purpose: str, sources, filters, known_limits,
                   not_for, splits: dict | None = None, leakage: dict | None = None,
                   synthetic_note: str = "") -> dict:
    """Facts that can be counted are counted from the data. Everything else is passed in."""
    traces = list(traces)
    rewards = [t.reward for t in traces if t.reward is not None]
    ts = [t.started for t in traces]
    card = {
        "name": name, "version": version, "purpose": purpose, "sources": list(sources),
        "counts": {"traces": len(traces),
                   "by_task_type": dict(sorted(count_by([{"k": t.task_type} for t in traces], "k").items())),
                   "users": len({t.user_id for t in traces})},
        "labels": {"labelled": len(rewards), "mean_reward": round(mean(rewards), 3) if rewards else None},
        "time_range": [min(ts), max(ts)] if ts else None,
        "harnesses": dict(sorted(count_by([{"k": t.harness} for t in traces], "k").items())),
        "privacy": {"scrubbed": sum(t.scrubbed for t in traces), "unscrubbed": sum(not t.scrubbed for t in traces),
                    "train_consent": sum(bool(t.consent.get("train")) for t in traces),
                    "share_consent": sum(bool(t.consent.get("share")) for t in traces),
                    "residual_pii_traces": sum(1 for t in traces if trace_residual_pii(t))},
        "splits": {k: len(v) for k, v in (splits or {}).items()},
        "filters": list(filters), "known_limits": list(known_limits), "not_for": list(not_for),
        "synthetic_note": synthetic_note,
    }
    if leakage is not None:
        card["leakage"] = {"ok": leakage["ok"], "exact": len(leakage["exact"]), "near": len(leakage["near"]),
                           "shared_groups": len(leakage["groups"]), "time_violations": leakage["time_violations"]}
    return card


def validate_card(card: dict) -> list:
    """Missing or empty required fields, and obvious contradictions."""
    problems = [f"missing {f}" for f in CARD_FIELDS if f not in card]
    for f in ("purpose", "sources", "filters", "known_limits", "not_for"):
        if f in card and not card[f]:
            problems.append(f"empty {f}")
    if card.get("splits") and card.get("counts") and sum(card["splits"].values()) > card["counts"]["traces"]:
        problems.append("splits hold more items than the card counts")
    if "privacy" in card and card["privacy"]["unscrubbed"] and card["privacy"]["share_consent"]:
        problems.append("unscrubbed traces in a card that lists sharing")
    if "leakage" in card and not card["leakage"]["ok"]:
        problems.append("leakage report is not clean")
    return problems


def render_card(card: dict) -> str:
    lines = [f"# Data card: {card['name']} (version {card['version']})", "", f"**Purpose.** {card['purpose']}", ""]
    c = card["counts"]
    lines += ["## Contents", f"- Traces: {c['traces']} from {c['users']} users",
              "- By task type: " + ", ".join(f"{k} {v}" for k, v in c["by_task_type"].items()),
              "- Harnesses: " + ", ".join(f"{k} {v}" for k, v in card["harnesses"].items()),
              f"- Labelled: {card['labels']['labelled']}, mean reward {card['labels']['mean_reward']}"]
    if card["time_range"]:
        lines.append(f"- Time range (seconds from start): {card['time_range'][0]:.0f} to {card['time_range'][1]:.0f}")
    p = card["privacy"]
    lines += ["", "## Privacy and consent",
              f"- Scrubbed: {p['scrubbed']}, not scrubbed: {p['unscrubbed']}",
              f"- Owners who allow training: {p['train_consent']}, sharing: {p['share_consent']}",
              f"- Traces with residual PII found by the patterns: {p['residual_pii_traces']}"]
    if card["splits"]:
        lines += ["", "## Splits", "- " + ", ".join(f"{k} {v}" for k, v in card["splits"].items())]
    if "leakage" in card:
        lg = card["leakage"]
        lines.append(f"- Leakage checks: {'clean' if lg['ok'] else 'NOT clean'} (exact {lg['exact']}, near {lg['near']}, "
                     f"shared groups {lg['shared_groups']}, time violations {lg['time_violations']})")
    lines += ["", "## Sources and filters"] + [f"- Source: {s}" for s in card["sources"]] + [f"- Filter: {f}" for f in card["filters"]]
    if card.get("synthetic_note"):
        lines += ["", f"**Synthetic data.** {card['synthetic_note']}"]
    lines += ["", "## Known limits"] + [f"- {x}" for x in card["known_limits"]]
    lines += ["", "## Not for"] + [f"- {x}" for x in card["not_for"]]
    return "\n".join(lines) + "\n"


# --------------------------------------------------------------------------- demo
def demo() -> None:
    out = print
    out("== 1. Trace schema, logging, validation ==")
    traces = simulate_traffic(120, seed=1, user_sim=UserSim(1), n_users=12)
    tr = traces[0]
    out(f"{len(traces)} traces logged; traces with validation problems: {sum(bool(validate_trace(t)) for t in traces)}")
    out(f"first trace {tr.trace_id}: type {tr.task_type}, harness {tr.harness}, reward {tr.reward}, {len(tr.events)} events")
    for e in tr.events:
        out(f"  {e.ts:6.1f}  {e.kind:<11} {e.text[:60]!r}" + (f"  meta={e.meta}" if e.meta else ""))
    rt = loads_jsonl(dumps_jsonl(traces))
    out(f"JSON Lines round trip equal: {rt == traces}; size {len(dumps_jsonl(traces))} bytes")
    out("span summary of the first trace: " + str(span_summary(to_span_tree(tr))))
    bad = Trace("x", "s", "u", "sum", "react", "m", 0.0, [Event("tool_result", "7", 1.0), Event("assistant", "hi", 0.5)])
    out("a broken trace: " + str(validate_trace(bad)))
    out("")
    out("== 2. Converting traces ==")
    sft = traces_to_sft(traces)
    prefs = traces_to_preferences(traces)
    edits = edit_pairs(traces)
    evals = traces_to_eval_cases(traces)
    ok = sum(1 for t in traces if t.reward == 1.0)
    out(f"traces {len(traces)}: passed {ok}, failed {len(traces) - ok}; owners allowing training: {sum(t.consent['train'] for t in traces)}")
    out(f"SFT demonstrations: {len(sft)} (passed AND consented); harness mix {harness_mix(sft)}")
    ex = sft[0]
    out(f"  example {ex['trace_id']}: learned and context words {loss_tokens(ex)}; learned roles: " + str([m['role'] for m in ex['messages'] if m['train']]))
    out(f"preference pairs: {len(prefs)} by outcome, {len(edits)} by edit")
    if prefs:
        out("  e.g. " + str({k: prefs[0][k] for k in ('prompt', 'chosen', 'rejected')}))
    out(f"eval cases from failures: {len(evals)}; per type: {count_by(evals, 'task_type')}")
    out("")
    out("== 3. Splits and leakage ==")
    items = [{"id": t.trace_id, "text": t.prompt(), "group": t.user_id, "ts": t.started} for t in traces]
    rnd = split_random(items, seed=3)
    rr = leakage_report(rnd, lambda x: x["text"], lambda x: x["group"], lambda x: x["ts"])
    out(f"random split {({k: len(v) for k, v in rnd.items()})}: exact copies {len(rr['exact'])}, near copies {len(rr['near'])}, shared users {len(rr['groups'])}, time violations {rr['time_violations']}")
    grp = split_by_group(items, lambda x: x["text"])
    gr = leakage_report(grp, lambda x: x["text"], lambda x: x["text"])
    out(f"split by prompt group {({k: len(v) for k, v in grp.items()})}: exact {len(gr['exact'])}, near {len(gr['near'])}, shared prompts {len(gr['groups'])}")
    sneaky = {**grp, "test": grp["test"] + [{"text": "again: " + x["text"]} for x in grp["train"][:5]]}
    sr = leakage_report(sneaky, lambda x: x["text"], lambda x: x["text"])
    out(f"same split plus 5 reworded copies of training prompts in test: exact {len(sr['exact'])}, near {len(sr['near'])}, shared prompts {len(sr['groups'])}, ok {sr['ok']}")
    ts = time_split(items, lambda x: x["ts"], 0.7 * 120 * 30.0, 0.85 * 120 * 30.0)
    tl = leakage_report(ts, lambda x: x["text"], None, lambda x: x["ts"])
    kept, dropped = decontaminate(ts["train"], ts["test"], lambda x: x["text"])
    out(f"time split {({k: len(v) for k, v in ts.items()})}: time violations {tl['time_violations']}, exact copies train-to-test {len(tl['exact'])}; decontaminate drops {len(dropped)} of {len(ts['train'])} training items")
    texts = {t.trace_id: t.prompt() for t in traces[:40]}
    fast, slow = near_duplicates(texts, 0.8), brute_force_near_duplicates(texts, 0.8)
    out(f"MinHash found {len(fast)} near-duplicate pairs among 40 prompts; brute force found {len(slow)}; same: {fast == slow}")
    out("")
    out("== 4. Flywheel on the mock ==")
    for label, cfg in (("base: filter failures, 50 labels per round", FlywheelConfig()),
                       ("no filter: train on failures too", FlywheelConfig(filter_failures=False)),
                       ("label noise 30 percent", FlywheelConfig(label_noise=0.3)),
                       ("no labels at all", FlywheelConfig(label_budget=0))):
        res = run_flywheel(cfg)
        out(f"{label}")
        out("  round: online/eval_all/eval_clean/overlap/sft")
        out("  " + " ".join(f"{h['round']}:{h['online']:.2f}/{h['eval_all']:.2f}/{h['eval_clean']:.2f}/{h['eval_overlap']}/{h['sft']}" for h in res['history']))
    one = run_flywheel(FlywheelConfig(harnesses=("react",), deploy_harness="json"))["history"][-1]
    mix = run_flywheel(FlywheelConfig(harnesses=("react", "json"), deploy_harness="json"))["history"][-1]
    out(f"deployed in the json harness after 6 rounds: trained on react only eval_clean {one['eval_clean']:.3f}; trained on react and json {mix['eval_clean']:.3f}")
    out("")
    out("== 5. Model collapse toy (not a language model) ==")
    probs = {"a": 0.50, "b": 0.25, "c": 0.15, "d": 0.06, "e": 0.03, "f": 0.01}
    for mix_w in (0.0, 0.2):
        rows = collapse_demo(probs, n=40, generations=12, seed=4, real_mix=mix_w)
        out(f"real_mix {mix_w}: support by generation " + str([r['support'] for r in rows]) + f"; entropy {rows[0]['entropy']} -> {rows[-1]['entropy']}")
    out("")
    out("== 6. Synthetic tasks with filters ==")
    agent = MockAgent(seed=5)
    res = generate_pipeline(400, agent, seed=2)
    out("funnel: " + " -> ".join(f"{n} {c}" for n, c in res["funnel"]))
    out(f"rejects: {res['rejects']}")
    out(f"proposed by hidden origin: {res['origin_proposed']}")
    out(f"kept by hidden origin: {res['origin_kept']}")
    out(f"diversity of kept: {diversity(res['kept'])}; of all proposed: {diversity(propose_candidates(400, 2))}")
    out("")
    out("== 7. Implicit feedback ==")
    sig = extract_signals(next(t for t in traces if t.feedback()))
    out(f"signals of one trace: {sig}")
    out(f"weak labels from signals vs the verifier: {signal_quality(traces)}")
    votes = [("a", "b")] * 6 + [("b", "a")] * 2 + [("b", "c")] * 5 + [("c", "b")] * 3 + [("a", "c")] * 7 + [("c", "a")] * 1
    bt = bradley_terry(votes)
    out("Bradley-Terry strengths from 24 made-up votes: " + str({k: round(v, 3) for k, v in bt.items()}))
    out("")
    out("== 8. PII scrubbing and consent ==")
    fake_key = "sk-" + "demo" * 5
    raw = ("Hi, I am Ada Lovelace. Mail ada@example.org or call (415) 555-0132. Card 4111 1111 1111 1111, "
           f"SSN 123-45-6789, key {fake_key}, host 10.0.0.12. Ada again: ada@example.org. Order 4111 1111 1111 1112.")
    sc = Scrubber(names=["Ada Lovelace"])
    clean = sc.scrub(raw)
    out("before: " + raw)
    out("after:  " + clean)
    out(f"counts {sc.counts}; residual PII after: {residual_pii(clean)}; a street address passes through: {Scrubber().scrub('I live at 12 Oak Street, Springfield')!r}")
    sc_traces = [scrub_trace(t, "demo-salt") for t in traces]
    dom = [t for t in traces if t.task_type == "domain"][0]
    out(f"a domain-task prompt before: {dom.prompt()!r}; after: {scrub_trace(dom, 'demo-salt').prompt()!r}")
    out(f"traces with residual PII, raw: {sum(bool(trace_residual_pii(t)) for t in traces)}; scrubbed: {sum(bool(trace_residual_pii(t)) for t in sc_traces)}")
    keep_tr, why = consent_filter(sc_traces, "train")
    keep_sh, why_sh = consent_filter(sc_traces, "share")
    raw_keep, raw_why = consent_filter(traces, "train")
    out(f"scrubbed, for training: kept {len(keep_tr)}, dropped {why}; for sharing: kept {len(keep_sh)}, dropped {why_sh}")
    out(f"raw (not scrubbed), for training: kept {len(raw_keep)}, dropped {raw_why}")
    ret, gone = retention_filter(sc_traces, now=3600.0, max_age_days=0.5 / 24)
    out(f"retention of 30 minutes, checked at second 3600: kept {len(ret)}, dropped {gone}")
    left, n_del = delete_user(sc_traces, "u03", "demo-salt")
    out(f"delete user u03 (pseudonymized): removed {n_del}, left {len(left)}")
    out("")
    out("== 9. Data card ==")
    cards_in = [{"t": t, "text": t.prompt()} for t in keep_tr]
    csplit = split_by_group(cards_in, lambda x: x["text"])
    crep = leakage_report(csplit, lambda x: x["text"], lambda x: x["text"])
    card = make_data_card("mock-agent-traces", "0.1", keep_tr, "Teach a mock agent; show the workflow.",
                          ["mock traffic from a simulator (made up)"],
                          ["owner allowed training", "scrubbed by regex", "schema validated", "no reward filter: failures kept"],
                          ["synthetic users and tasks", "regex scrubbing misses names and addresses", "six task types only"],
                          ["training a real model", "any claim about real users"],
                          splits=csplit, leakage=crep,
                          synthetic_note="All traces come from a mock agent and a user simulator.")
    out(render_card(card).rstrip())
    out(f"card problems: {validate_card(card)}")


if __name__ == "__main__":
    demo()
    sys.exit(0)

Run the demo

$ python agent_data_toolkit.py
== 1. Trace schema, logging, validation ==
120 traces logged; traces with validation problems: 0
first trace t00001: type reverse, harness react, reward 1.0, 7 events
     0.0  system      'You are a helpful assistant. Harness: react.'
     0.5  user        'Reverse the order of these words: pink red teal'  meta={'task_id': 's1-0000'}
     1.5  assistant   'Thought: This is a reverse task.'
     2.0  tool_call   'solve[pink red teal]'
     3.0  tool_result 'teal red pink'
     4.0  assistant   'Answer: teal red pink'
     6.0  feedback    ''  meta={'type': 'copy'}
JSON Lines round trip equal: True; size 121774 bytes
span summary of the first trace: {'llm_call': 2, 'tool_call': 1, 'seconds': 6.0}
a broken trace: ['first message is not from the user', 'event 0: tool_result without tool_call', 'event 1: timestamps go backwards']

== 2. Converting traces ==
traces 120: passed 87, failed 33; owners allowing training: 93
SFT demonstrations: 72 (passed AND consented); harness mix {'react': 72}
  example t00001: learned and context words (14, 19); learned roles: ['assistant', 'assistant', 'assistant']
preference pairs: 7 by outcome, 5 by edit
  e.g. {'prompt': 'Count the vowels in the word: avocado', 'chosen': 'Answer: 4', 'rejected': 'Answer: 3'}
eval cases from failures: 29; per type: {'sort': 11, 'vowels': 7, 'reverse': 6, 'domain': 3, 'upper': 1, 'sum': 1}

== 3. Splits and leakage ==
random split {'train': 96, 'dev': 12, 'test': 12}: exact copies 15, near copies 0, shared users 17, time violations 24
split by prompt group {'train': 74, 'dev': 28, 'test': 18}: exact 0, near 0, shared prompts 0
same split plus 5 reworded copies of training prompts in test: exact 0, near 5, shared prompts 0, ok False
time split {'train': 84, 'dev': 18, 'test': 18}: time violations 0, exact copies train-to-test 12; decontaminate drops 3 of 84 training items
MinHash found 1 near-duplicate pairs among 40 prompts; brute force found 1; same: True

== 4. Flywheel on the mock ==
base: filter failures, 50 labels per round
  round: online/eval_all/eval_clean/overlap/sft
  0:0.75/0.69/0.68/2/28 1:0.80/0.76/0.74/4/32 2:0.82/0.84/0.80/14/29 3:0.93/0.88/0.84/18/36 4:0.94/0.91/0.87/22/41 5:0.96/0.93/0.90/24/38
no filter: train on failures too
  round: online/eval_all/eval_clean/overlap/sft
  0:0.75/0.67/0.65/3/39 1:0.79/0.73/0.70/7/40 2:0.80/0.83/0.76/20/40 3:0.90/0.85/0.77/23/41 4:0.91/0.87/0.80/24/43 5:0.96/0.90/0.84/26/40
label noise 30 percent
  round: online/eval_all/eval_clean/overlap/sft
  0:0.75/0.68/0.67/2/24 1:0.80/0.74/0.72/5/23 2:0.82/0.81/0.76/14/23 3:0.90/0.85/0.80/18/29 4:0.93/0.89/0.83/23/30 5:0.96/0.91/0.86/25/27
no labels at all
  round: online/eval_all/eval_clean/overlap/sft
  0:0.75/0.63/0.63/0/0 1:0.75/0.63/0.63/0/0 2:0.71/0.63/0.63/0/0 3:0.72/0.63/0.63/0/0 4:0.70/0.63/0.63/0/0 5:0.71/0.63/0.63/0/0
deployed in the json harness after 6 rounds: trained on react only eval_clean 0.540; trained on react and json 0.891

== 5. Model collapse toy (not a language model) ==
real_mix 0.0: support by generation [6, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 2, 2]; entropy 1.872 -> 0.811
real_mix 0.2: support by generation [6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6]; entropy 1.872 -> 2.093

== 6. Synthetic tasks with filters ==
funnel: proposed 400 -> valid and verified 334 -> after dedupe 181 -> after difficulty 158
rejects: {'duplicate': 153, 'empty': 9, 'length': 15, 'too easy or too hard': 23, 'wrong_answer': 42}
proposed by hidden origin: {'bad_answer': 42, 'clean': 269, 'dup': 22, 'invalid': 24, 'near_dup': 43}
kept by hidden origin: {'clean': 158}
diversity of kept: {'n': 158, 'types': 6, 'type_entropy': 2.374, 'openings': 6}; of all proposed: {'n': 400, 'types': 6, 'type_entropy': 2.579, 'openings': 15}

== 7. Implicit feedback ==
signals of one trace: {'thumbs_up': 0, 'thumbs_down': 0, 'regenerate': 0, 'edit': 0, 'copy': 1, 'abandon': 0, 'edit_ratio': 0.0, 'rephrase': False, 'n_feedback': 1}
weak labels from signals vs the verifier: {'n': 120, 'coverage': 0.725, 'precision_good': 0.98, 'recall_good': 0.877, 'accuracy': 0.908, 'tp': 50, 'fp': 1, 'fn': 7, 'tn': 29}
Bradley-Terry strengths from 24 made-up votes: {'a': 2.706, 'b': 0.816, 'c': 0.453}

== 8. PII scrubbing and consent ==
before: Hi, I am Ada Lovelace. Mail ada@example.org or call (415) 555-0132. Card 4111 1111 1111 1111, SSN 123-45-6789, key sk-demodemodemodemodemo, host 10.0.0.12. Ada again: ada@example.org. Order 4111 1111 1111 1112.
after:  Hi, I am <NAME_1>. Mail <EMAIL_1> or call <PHONE_1>. Card <CARD_1>, SSN <SSN_1>, key <KEY_1>, host <IP_1>. Ada again: <EMAIL_1>. Order 4111 1111 1111 1112.
counts {'EMAIL': 2, 'KEY': 1, 'SSN': 1, 'CARD': 1, 'PHONE': 1, 'IP': 1, 'NAME': 1}; residual PII after: []; a street address passes through: 'I live at 12 Oak Street, Springfield'
a domain-task prompt before: 'What is the domain part of this email address: barbara@uni.example'; after: 'What is the domain part of this email address: <EMAIL_1>'
traces with residual PII, raw: 23; scrubbed: 0
scrubbed, for training: kept 93, dropped {'no_consent': 27, 'not_scrubbed': 0}; for sharing: kept 42, dropped {'no_consent': 78, 'not_scrubbed': 0}
raw (not scrubbed), for training: kept 0, dropped {'no_consent': 27, 'not_scrubbed': 93}
retention of 30 minutes, checked at second 3600: kept 60, dropped 60
delete user u03 (pseudonymized): removed 9, left 111

== 9. Data card ==
# Data card: mock-agent-traces (version 0.1)

**Purpose.** Teach a mock agent; show the workflow.

## Contents
- Traces: 93 from 9 users
- By task type: domain 19, reverse 14, sort 20, sum 10, upper 14, vowels 16
- Harnesses: react 93
- Labelled: 93, mean reward 0.774
- Time range (seconds from start): 0 to 3570

## Privacy and consent
- Scrubbed: 93, not scrubbed: 0
- Owners who allow training: 93, sharing: 42
- Traces with residual PII found by the patterns: 0

## Splits
- train 67, dev 19, test 7
- Leakage checks: clean (exact 0, near 0, shared groups 0, time violations 0)

## Sources and filters
- Source: mock traffic from a simulator (made up)
- Filter: owner allowed training
- Filter: scrubbed by regex
- Filter: schema validated
- Filter: no reward filter: failures kept

**Synthetic data.** All traces come from a mock agent and a user simulator.

## Known limits
- synthetic users and tasks
- regex scrubbing misses names and addresses
- six task types only

## Not for
- training a real model
- any claim about real users
card problems: []

Exercises

1. Read 20 traces. Print 20 traces with a short function of your own that prints each event on a line. Write down what you notice. Which fact would you add to the schema? Which would you remove? Compare with Husain's "remove all friction from looking at data" [3].
2. Break the validator. Build traces with an orphan tool result, a missing answer, a duplicate event time, and a feedback event before the answer. Which does validate_trace catch? Add a rule for one that it misses, with three tests.
3. Consent matrix. Build a table of 8 traces with every mix of the three consent flags and pass or fail. For each of the three products (demonstration, pair, eval case), say which traces go in. Do eval cases need consent to train? Argue for your rule.
4. Pairs from edits. Change the user simulator so that an edit sometimes makes the answer worse. How many bad pairs enter edit_pairs? Add a check that drops pairs where the "chosen" text is not verified by the verifier. What does it cost in data [27]?
5. Leak on purpose. Make a split with 5 exact copies, 5 reworded copies, and 5 same-user items. Run leakage_report. Then lower the threshold from 0.8 to 0.5. What does it find that it did not? What false alarms appear? Use the table in 2.10 to predict the miss rate first [31].
6. Flywheel knobs. Run run_flywheel with 10, 30 and 100 labels per round, and with 0%, 30% and 50% label noise. Plot eval_clean by round. At what noise level does a round make things worse? The mock "training" rule is made up, so explain which part of your answer would carry over to a real system and which would not [1].
7. Make the generator worse. Add a new kind of bad candidate (for example, an answer with the right value and extra words). Does filter_verified catch it? Change the verifier to accept it. How many bad tasks reach the final set? What does that say about the verifier as part of the data [10]?
8. Diversity after filtering. Run generate_pipeline with different k and weights. Plot the type entropy and the number of openings before and after. When does filtering reduce diversity the most? Write one rule you would add to protect it.
9. Signal weights. Change the signal weights in signals_to_score. Search for a setting that raises coverage and see what it does to precision. Why is a perfect score on the lab's mock not a good sign for a real product [26]?
10. Scrubber recall. Write 20 sentences with PII in unusual forms (a spelled-out phone number, an email with spaces, a card in groups of five). How many does the scrubber miss? Add one pattern and a test. Then explain why the residual check cannot tell you the true miss rate [32].
11. Write a data card for the output of exercise 6, with three known limits and three "not for" lines. Run validate_card. Then make the card fail in two different ways and name the rule that caught each [33].
12. Collapse with your own numbers. Run collapse_demo with 100 draws instead of 40 and the same distribution, over many seeds. The chance that the 1% item is missed in one generation is 0.99ⁿ, which is 0.67 for 40 draws and 0.37 for 100 (my arithmetic). Does the simulation agree on how often it vanishes in the first generation [25]?

The tests: test_agent_data_toolkit.py

The 266 tests check the shared helpers (hashing, words, entropy); the trace model (round trips, defaults, the scrubbed flag); the logger (order, closing, errors); the validator; JSON Lines and the span tree; the mock world (prompts, solutions, verifiers, corruption); the mock agent (skill, harness effect, memory, attempts); the user simulator and the traffic; each conversion (demonstrations, preference pairs, edit pairs, eval cases) with the consent gates; shingles, MinHash and near-duplicate search (against brute force, and on candidate counts); the splits and the leakage report; decontamination; the mock training rule; the flywheel (each variant, and clean against all); the collapse toy; the synthetic task generator and each filter; the signals, the weak labels and their quality counts; the Bradley-Terry fit; the Luhn check and the scrubber; trace scrubbing, consent, retention and deletion; the data card and its validator; an end-to-end run; and the demo itself. Writing them found real bugs in my first version, which I fixed (for example, a leakage check that missed near copies because the pairs came back in sorted order, and a generator that picked the wrong sources for its near-copy candidates). I also ran a mutation check: I made 230 small deliberate changes to the code, one at a time, and the tests caught every one. Two ideas for changes were dropped because they could not change the output (a comparison of a random number with a boundary it hits with probability near zero). The check also found one line that did nothing, a break that could never matter, and I removed it. That says the tests are sensitive. It does not say the code is correct.

$ python test_agent_data_toolkit.py -v
test_chain_of_models (test_agent_data_toolkit.TestBradleyTerry.test_chain_of_models) ... ok
test_equal_votes_equal_strength (test_agent_data_toolkit.TestBradleyTerry.test_equal_votes_equal_strength) ... ok
test_mle_conditions (test_agent_data_toolkit.TestBradleyTerry.test_mle_conditions) ... ok
test_three_models_order_and_scale (test_agent_data_toolkit.TestBradleyTerry.test_three_models_order_and_scale) ... ok
test_two_models (test_agent_data_toolkit.TestBradleyTerry.test_two_models) ... ok
test_a_single_item_stays_a_probability (test_agent_data_toolkit.TestCollapse.test_a_single_item_stays_a_probability) ... ok
test_bigger_samples_collapse_slower (test_agent_data_toolkit.TestCollapse.test_bigger_samples_collapse_slower) ... ok
test_deterministic (test_agent_data_toolkit.TestCollapse.test_deterministic) ... ok
test_full_mix_never_moves (test_agent_data_toolkit.TestCollapse.test_full_mix_never_moves) ... ok
test_real_data_keeps_the_tails (test_agent_data_toolkit.TestCollapse.test_real_data_keeps_the_tails) ... ok
test_rows_and_start (test_agent_data_toolkit.TestCollapse.test_rows_and_start) ... ok
test_tails_vanish_and_never_return (test_agent_data_toolkit.TestCollapse.test_tails_vanish_and_never_return) ... ok
test_unnormalised_input (test_agent_data_toolkit.TestCollapse.test_unnormalised_input) ... ok
test_default_consent_blocks_training (test_agent_data_toolkit.TestConsent.test_default_consent_blocks_training) ... ok
test_log_needs_no_scrubbing (test_agent_data_toolkit.TestConsent.test_log_needs_no_scrubbing) ... ok
test_share (test_agent_data_toolkit.TestConsent.test_share) ... ok
test_train (test_agent_data_toolkit.TestConsent.test_train) ... ok
test_unknown_purpose (test_agent_data_toolkit.TestConsent.test_unknown_purpose) ... ok
test_counts_come_from_the_data (test_agent_data_toolkit.TestDataCard.test_counts_come_from_the_data) ... ok
test_inputs_are_copied (test_agent_data_toolkit.TestDataCard.test_inputs_are_copied) ... ok
test_render (test_agent_data_toolkit.TestDataCard.test_render) ... ok
test_render_optional_parts (test_agent_data_toolkit.TestDataCard.test_render_optional_parts) ... ok
test_residual_pii_counted (test_agent_data_toolkit.TestDataCard.test_residual_pii_counted) ... ok
test_splits_and_leakage (test_agent_data_toolkit.TestDataCard.test_splits_and_leakage) ... ok
test_unlabelled_and_empty (test_agent_data_toolkit.TestDataCard.test_unlabelled_and_empty) ... ok
test_validate_contradictions (test_agent_data_toolkit.TestDataCard.test_validate_contradictions) ... ok
test_validate_missing_and_empty (test_agent_data_toolkit.TestDataCard.test_validate_missing_and_empty) ... ok
test_validate_ok (test_agent_data_toolkit.TestDataCard.test_validate_ok) ... ok
test_drops_exact_and_near (test_agent_data_toolkit.TestDecontaminate.test_drops_exact_and_near) ... ok
test_exact_copies_go_even_when_near_matching_is_off (test_agent_data_toolkit.TestDecontaminate.test_exact_copies_go_even_when_near_matching_is_off) ... ok
test_exact_with_case (test_agent_data_toolkit.TestDecontaminate.test_exact_with_case) ... ok
test_near_dups_inside_train_are_kept (test_agent_data_toolkit.TestDecontaminate.test_near_dups_inside_train_are_kept) ... ok
test_nothing_to_drop (test_agent_data_toolkit.TestDecontaminate.test_nothing_to_drop) ... ok
test_threshold (test_agent_data_toolkit.TestDecontaminate.test_threshold) ... ok
test_demo_is_deterministic_and_complete (test_agent_data_toolkit.TestDemo.test_demo_is_deterministic_and_complete) ... ok
test_main_guard (test_agent_data_toolkit.TestDemo.test_main_guard) ... ok
test_consent (test_agent_data_toolkit.TestEditPairs.test_consent) ... ok
test_edit_becomes_chosen (test_agent_data_toolkit.TestEditPairs.test_edit_becomes_chosen) ... ok
test_one_pair_per_trace (test_agent_data_toolkit.TestEditPairs.test_one_pair_per_trace) ... ok
test_same_text_or_empty_ignored (test_agent_data_toolkit.TestEditPairs.test_same_text_or_empty_ignored) ... ok
test_whitespace_only_difference_is_not_an_edit (test_agent_data_toolkit.TestEditPairs.test_whitespace_only_difference_is_not_an_edit) ... ok
test_generated_tasks_feed_the_agent (test_agent_data_toolkit.TestEndToEnd.test_generated_tasks_feed_the_agent) ... ok
test_raw_traces_fail_the_gates (test_agent_data_toolkit.TestEndToEnd.test_raw_traces_fail_the_gates) ... ok
test_safe_pipeline (test_agent_data_toolkit.TestEndToEnd.test_safe_pipeline)
traffic -> scrub -> consent -> group split -> leakage clean -> card valid ... ok
test_traces_to_three_kinds_of_data (test_agent_data_toolkit.TestEndToEnd.test_traces_to_three_kinds_of_data) ... ok
test_expected_score (test_agent_data_toolkit.TestEvalAndLabels.test_expected_score) ... ok
test_label_deterministic (test_agent_data_toolkit.TestEvalAndLabels.test_label_deterministic) ... ok
test_label_noise (test_agent_data_toolkit.TestEvalAndLabels.test_label_noise) ... ok
test_label_skips_unlabelled (test_agent_data_toolkit.TestEvalAndLabels.test_label_skips_unlabelled) ... ok
test_label_traces (test_agent_data_toolkit.TestEvalAndLabels.test_label_traces) ... ok
test_make_eval_set (test_agent_data_toolkit.TestEvalAndLabels.test_make_eval_set) ... ok
test_dedupe_by_prompt (test_agent_data_toolkit.TestEvalCases.test_dedupe_by_prompt) ... ok
test_failures_only_by_default (test_agent_data_toolkit.TestEvalCases.test_failures_only_by_default) ... ok
test_fields (test_agent_data_toolkit.TestEvalCases.test_fields) ... ok
test_invalid_trace_skipped (test_agent_data_toolkit.TestEvalCases.test_invalid_trace_skipped) ... ok
test_max_per_type (test_agent_data_toolkit.TestEvalCases.test_max_per_type) ... ok
test_needs_verifier_and_label (test_agent_data_toolkit.TestEvalCases.test_needs_verifier_and_label) ... ok
test_consent_limits_the_data (test_agent_data_toolkit.TestFlywheel.test_consent_limits_the_data) ... ok
test_deterministic (test_agent_data_toolkit.TestFlywheel.test_deterministic) ... ok
test_filter_failures_matters (test_agent_data_toolkit.TestFlywheel.test_filter_failures_matters) ... ok
test_harness_mismatch (test_agent_data_toolkit.TestFlywheel.test_harness_mismatch) ... ok
test_history_shape (test_agent_data_toolkit.TestFlywheel.test_history_shape) ... ok
test_init_skill_used_and_not_mutated (test_agent_data_toolkit.TestFlywheel.test_init_skill_used_and_not_mutated) ... ok
test_it_improves (test_agent_data_toolkit.TestFlywheel.test_it_improves) ... ok
test_label_noise_hurts (test_agent_data_toolkit.TestFlywheel.test_label_noise_hurts) ... ok
test_leaky_score_is_higher_than_clean (test_agent_data_toolkit.TestFlywheel.test_leaky_score_is_higher_than_clean) ... ok
test_no_labels_means_no_learning (test_agent_data_toolkit.TestFlywheel.test_no_labels_means_no_learning) ... ok
test_overlap_counts_prompts_seen_in_training (test_agent_data_toolkit.TestFlywheel.test_overlap_counts_prompts_seen_in_training) ... ok
test_zero_rounds (test_agent_data_toolkit.TestFlywheel.test_zero_rounds) ... ok
test_empty (test_agent_data_toolkit.TestJsonlAndSpans.test_empty) ... ok
test_loads_skips_blank_lines (test_agent_data_toolkit.TestJsonlAndSpans.test_loads_skips_blank_lines) ... ok
test_one_line_each_and_sorted_keys (test_agent_data_toolkit.TestJsonlAndSpans.test_one_line_each_and_sorted_keys) ... ok
test_round_trip (test_agent_data_toolkit.TestJsonlAndSpans.test_round_trip) ... ok
test_span_tree (test_agent_data_toolkit.TestJsonlAndSpans.test_span_tree) ... ok
test_span_tree_no_events (test_agent_data_toolkit.TestJsonlAndSpans.test_span_tree_no_events) ... ok
test_clean (test_agent_data_toolkit.TestLeakage.test_clean) ... ok
test_exact_copy_ignores_case_and_punctuation (test_agent_data_toolkit.TestLeakage.test_exact_copy_ignores_case_and_punctuation) ... ok
test_group_split_by_text_passes_the_check (test_agent_data_toolkit.TestLeakage.test_group_split_by_text_passes_the_check) ... ok
test_multiple_splits_are_each_checked (test_agent_data_toolkit.TestLeakage.test_multiple_splits_are_each_checked) ... ok
test_near_copies_inside_train_are_not_leaks (test_agent_data_toolkit.TestLeakage.test_near_copies_inside_train_are_not_leaks) ... ok
test_near_copy (test_agent_data_toolkit.TestLeakage.test_near_copy) ... ok
test_near_threshold (test_agent_data_toolkit.TestLeakage.test_near_threshold) ... ok
test_random_split_of_duplicated_prompts_fails_the_check (test_agent_data_toolkit.TestLeakage.test_random_split_of_duplicated_prompts_fails_the_check) ... ok
test_shared_group (test_agent_data_toolkit.TestLeakage.test_shared_group) ... ok
test_time_violation (test_agent_data_toolkit.TestLeakage.test_time_violation) ... ok
test_without_train_nothing_to_check (test_agent_data_toolkit.TestLeakage.test_without_train_nothing_to_check) ... ok
test_consent_merge (test_agent_data_toolkit.TestLogger.test_consent_merge) ... ok
test_end_checks (test_agent_data_toolkit.TestLogger.test_end_checks) ... ok
test_end_sets_fields_and_closes (test_agent_data_toolkit.TestLogger.test_end_sets_fields_and_closes) ... ok
test_first_event_not_before_start (test_agent_data_toolkit.TestLogger.test_first_event_not_before_start) ... ok
test_ids_count_up (test_agent_data_toolkit.TestLogger.test_ids_count_up) ... ok
test_log_appends (test_agent_data_toolkit.TestLogger.test_log_appends) ... ok
test_log_converts_text_to_str (test_agent_data_toolkit.TestLogger.test_log_converts_text_to_str) ... ok
test_log_time_must_not_go_back (test_agent_data_toolkit.TestLogger.test_log_time_must_not_go_back) ... ok
test_log_unknown_kind (test_agent_data_toolkit.TestLogger.test_log_unknown_kind) ... ok
test_log_unknown_trace (test_agent_data_toolkit.TestLogger.test_log_unknown_trace) ... ok
test_open_ids (test_agent_data_toolkit.TestLogger.test_open_ids) ... ok
test_luhn (test_agent_data_toolkit.TestLuhn.test_luhn) ... ok
test_shape_and_determinism (test_agent_data_toolkit.TestMinHash.test_shape_and_determinism) ... ok
test_similarity (test_agent_data_toolkit.TestMinHash.test_similarity) ... ok
test_attempt_rates (test_agent_data_toolkit.TestMockAgent.test_attempt_rates) ... ok
test_attempts_deterministic_and_seeded (test_agent_data_toolkit.TestMockAgent.test_attempts_deterministic_and_seeded) ... ok
test_attempts_perfect_and_hopeless (test_agent_data_toolkit.TestMockAgent.test_attempts_perfect_and_hopeless) ... ok
test_effective_skill (test_agent_data_toolkit.TestMockAgent.test_effective_skill) ... ok
test_memory_and_harness_together (test_agent_data_toolkit.TestMockAgent.test_memory_and_harness_together) ... ok
test_memory_floor (test_agent_data_toolkit.TestMockAgent.test_memory_floor) ... ok
test_run_consent_passed_through (test_agent_data_toolkit.TestMockAgent.test_run_consent_passed_through) ... ok
test_run_hopeless_retries_once (test_agent_data_toolkit.TestMockAgent.test_run_hopeless_retries_once) ... ok
test_run_json_harness (test_agent_data_toolkit.TestMockAgent.test_run_json_harness) ... ok
test_run_perfect_react (test_agent_data_toolkit.TestMockAgent.test_run_perfect_react) ... ok
test_run_times_increase (test_agent_data_toolkit.TestMockAgent.test_run_times_increase) ... ok
test_second_attempt_has_half_the_skill (test_agent_data_toolkit.TestMockAgent.test_second_attempt_has_half_the_skill) ... ok
test_bands_must_divide (test_agent_data_toolkit.TestNearDuplicates.test_bands_must_divide) ... ok
test_brute_force_threshold (test_agent_data_toolkit.TestNearDuplicates.test_brute_force_threshold) ... ok
test_candidates_are_a_small_superset (test_agent_data_toolkit.TestNearDuplicates.test_candidates_are_a_small_superset) ... ok
test_empty_and_single (test_agent_data_toolkit.TestNearDuplicates.test_empty_and_single) ... ok
test_identical_found (test_agent_data_toolkit.TestNearDuplicates.test_identical_found) ... ok
test_matches_brute_force (test_agent_data_toolkit.TestNearDuplicates.test_matches_brute_force) ... ok
test_one_word_changed_in_long_text (test_agent_data_toolkit.TestNearDuplicates.test_one_word_changed_in_long_text) ... ok
test_output_sorted_pairs (test_agent_data_toolkit.TestNearDuplicates.test_output_sorted_pairs) ... ok
test_partly_similar_texts_are_rarely_candidates (test_agent_data_toolkit.TestNearDuplicates.test_partly_similar_texts_are_rarely_candidates) ... ok
test_threshold_is_inclusive (test_agent_data_toolkit.TestNearDuplicates.test_threshold_is_inclusive) ... ok
test_consent_and_unlabelled (test_agent_data_toolkit.TestPreferences.test_consent_and_unlabelled) ... ok
test_different_prompts_do_not_pair (test_agent_data_toolkit.TestPreferences.test_different_prompts_do_not_pair) ... ok
test_graded_rewards (test_agent_data_toolkit.TestPreferences.test_graded_rewards) ... ok
test_invalid_trace_ignored (test_agent_data_toolkit.TestPreferences.test_invalid_trace_ignored) ... ok
test_longer_trace_is_rejected_among_fails (test_agent_data_toolkit.TestPreferences.test_longer_trace_is_rejected_among_fails) ... ok
test_no_gap_no_pair (test_agent_data_toolkit.TestPreferences.test_no_gap_no_pair) ... ok
test_pass_beats_fail (test_agent_data_toolkit.TestPreferences.test_pass_beats_fail) ... ok
test_shorter_wins_ties_among_passes (test_agent_data_toolkit.TestPreferences.test_shorter_wins_ties_among_passes) ... ok
test_single_trace_no_pair (test_agent_data_toolkit.TestPreferences.test_single_trace_no_pair) ... ok
test_sorted_by_prompt (test_agent_data_toolkit.TestPreferences.test_sorted_by_prompt) ... ok
test_delete_pseudonymized_user_needs_the_salt (test_agent_data_toolkit.TestRetentionAndDeletion.test_delete_pseudonymized_user_needs_the_salt) ... ok
test_delete_user (test_agent_data_toolkit.TestRetentionAndDeletion.test_delete_user) ... ok
test_retention_boundary (test_agent_data_toolkit.TestRetentionAndDeletion.test_retention_boundary) ... ok
test_retention_none_left (test_agent_data_toolkit.TestRetentionAndDeletion.test_retention_none_left) ... ok
test_names_and_no_verifier (test_agent_data_toolkit.TestScrubTrace.test_names_and_no_verifier) ... ok
test_original_is_untouched (test_agent_data_toolkit.TestScrubTrace.test_original_is_untouched) ... ok
test_other_fields_kept (test_agent_data_toolkit.TestScrubTrace.test_other_fields_kept) ... ok
test_scrubbed_copy (test_agent_data_toolkit.TestScrubTrace.test_scrubbed_copy) ... ok
test_simulated_domain_tasks (test_agent_data_toolkit.TestScrubTrace.test_simulated_domain_tasks) ... ok
test_trace_residual_pii (test_agent_data_toolkit.TestScrubTrace.test_trace_residual_pii) ... ok
test_card_needs_luhn (test_agent_data_toolkit.TestScrubber.test_card_needs_luhn) ... ok
test_email_same_value_same_placeholder (test_agent_data_toolkit.TestScrubber.test_email_same_value_same_placeholder) ... ok
test_email_with_several_dots (test_agent_data_toolkit.TestScrubber.test_email_with_several_dots) ... ok
test_ip (test_agent_data_toolkit.TestScrubber.test_ip) ... ok
test_key (test_agent_data_toolkit.TestScrubber.test_key) ... ok
test_known_misses (test_agent_data_toolkit.TestScrubber.test_known_misses) ... ok
test_name_with_regex_characters (test_agent_data_toolkit.TestScrubber.test_name_with_regex_characters) ... ok
test_names (test_agent_data_toolkit.TestScrubber.test_names) ... ok
test_non_string_input (test_agent_data_toolkit.TestScrubber.test_non_string_input) ... ok
test_numbering_is_per_kind (test_agent_data_toolkit.TestScrubber.test_numbering_is_per_kind) ... ok
test_phone_formats (test_agent_data_toolkit.TestScrubber.test_phone_formats) ... ok
test_pseudonymize (test_agent_data_toolkit.TestScrubber.test_pseudonymize) ... ok
test_residual (test_agent_data_toolkit.TestScrubber.test_residual) ... ok
test_ssn (test_agent_data_toolkit.TestScrubber.test_ssn) ... ok
test_state_is_kept_between_calls (test_agent_data_toolkit.TestScrubber.test_state_is_kept_between_calls) ... ok
test_feedback_not_in_messages (test_agent_data_toolkit.TestSft.test_feedback_not_in_messages) ... ok
test_fields (test_agent_data_toolkit.TestSft.test_fields) ... ok
test_loss_tokens_and_harness_mix (test_agent_data_toolkit.TestSft.test_loss_tokens_and_harness_mix) ... ok
test_max_events (test_agent_data_toolkit.TestSft.test_max_events) ... ok
test_only_passed_and_consented (test_agent_data_toolkit.TestSft.test_only_passed_and_consented) ... ok
test_partial_reward_threshold (test_agent_data_toolkit.TestSft.test_partial_reward_threshold) ... ok
test_roles_and_loss_mask (test_agent_data_toolkit.TestSft.test_roles_and_loss_mask) ... ok
test_unlabelled_and_invalid_dropped (test_agent_data_toolkit.TestSft.test_unlabelled_and_invalid_dropped) ... ok
test_count_by (test_agent_data_toolkit.TestShared.test_count_by) ... ok
test_entropy (test_agent_data_toolkit.TestShared.test_entropy) ... ok
test_mean (test_agent_data_toolkit.TestShared.test_mean) ... ok
test_rng_for_repeats (test_agent_data_toolkit.TestShared.test_rng_for_repeats) ... ok
test_stable_hash_bits (test_agent_data_toolkit.TestShared.test_stable_hash_bits) ... ok
test_stable_hash_is_stable (test_agent_data_toolkit.TestShared.test_stable_hash_is_stable) ... ok
test_stable_hash_separates_parts (test_agent_data_toolkit.TestShared.test_stable_hash_separates_parts) ... ok
test_words (test_agent_data_toolkit.TestShared.test_words) ... ok
test_exact_key (test_agent_data_toolkit.TestShingles.test_exact_key) ... ok
test_jaccard (test_agent_data_toolkit.TestShingles.test_jaccard) ... ok
test_shingles (test_agent_data_toolkit.TestShingles.test_shingles) ... ok
test_extract_counts (test_agent_data_toolkit.TestSignals.test_extract_counts) ... ok
test_extract_edit_ratio (test_agent_data_toolkit.TestSignals.test_extract_edit_ratio) ... ok
test_extract_ignores_non_feedback_events_and_unknown_types (test_agent_data_toolkit.TestSignals.test_extract_ignores_non_feedback_events_and_unknown_types) ... ok
test_extract_rephrase (test_agent_data_toolkit.TestSignals.test_extract_rephrase) ... ok
test_partial_reward_is_not_a_pass (test_agent_data_toolkit.TestSignals.test_partial_reward_is_not_a_pass) ... ok
test_score_combinations_and_clipping (test_agent_data_toolkit.TestSignals.test_score_combinations_and_clipping) ... ok
test_score_edit (test_agent_data_toolkit.TestSignals.test_score_edit) ... ok
test_score_single_signals (test_agent_data_toolkit.TestSignals.test_score_single_signals) ... ok
test_signal_quality (test_agent_data_toolkit.TestSignals.test_signal_quality) ... ok
test_signal_quality_empty (test_agent_data_toolkit.TestSignals.test_signal_quality_empty) ... ok
test_signal_quality_on_simulated_traffic (test_agent_data_toolkit.TestSignals.test_signal_quality_on_simulated_traffic) ... ok
test_weak_label_edges (test_agent_data_toolkit.TestSignals.test_weak_label_edges) ... ok
test_group_split_boundary_goes_to_the_next_split (test_agent_data_toolkit.TestSplits.test_group_split_boundary_goes_to_the_next_split) ... ok
test_group_split_keeps_groups_together (test_agent_data_toolkit.TestSplits.test_group_split_keeps_groups_together) ... ok
test_group_split_names_and_validation (test_agent_data_toolkit.TestSplits.test_group_split_names_and_validation) ... ok
test_group_split_proportions_and_salt (test_agent_data_toolkit.TestSplits.test_group_split_proportions_and_salt) ... ok
test_random_split (test_agent_data_toolkit.TestSplits.test_random_split) ... ok
test_random_split_last_gets_remainder (test_agent_data_toolkit.TestSplits.test_random_split_last_gets_remainder) ... ok
test_random_split_loses_nothing_to_rounding (test_agent_data_toolkit.TestSplits.test_random_split_loses_nothing_to_rounding) ... ok
test_time_split_boundaries (test_agent_data_toolkit.TestSplits.test_time_split_boundaries) ... ok
test_dedupe_against_existing (test_agent_data_toolkit.TestSyntheticTasks.test_dedupe_against_existing) ... ok
test_dedupe_exact_and_paraphrase (test_agent_data_toolkit.TestSyntheticTasks.test_dedupe_exact_and_paraphrase) ... ok
test_dedupe_is_per_type (test_agent_data_toolkit.TestSyntheticTasks.test_dedupe_is_per_type) ... ok
test_dedupe_threshold_edge (test_agent_data_toolkit.TestSyntheticTasks.test_dedupe_threshold_edge) ... ok
test_difficulty_filter (test_agent_data_toolkit.TestSyntheticTasks.test_difficulty_filter) ... ok
test_difficulty_is_noisy_for_easy_tasks (test_agent_data_toolkit.TestSyntheticTasks.test_difficulty_is_noisy_for_easy_tasks) ... ok
test_diversity (test_agent_data_toolkit.TestSyntheticTasks.test_diversity) ... ok
test_filter_valid (test_agent_data_toolkit.TestSyntheticTasks.test_filter_valid) ... ok
test_filter_verified (test_agent_data_toolkit.TestSyntheticTasks.test_filter_verified) ... ok
test_first_candidate_cannot_be_a_copy (test_agent_data_toolkit.TestSyntheticTasks.test_first_candidate_cannot_be_a_copy) ... ok
test_paraphrase_light (test_agent_data_toolkit.TestSyntheticTasks.test_paraphrase_light) ... ok
test_pass_count (test_agent_data_toolkit.TestSyntheticTasks.test_pass_count) ... ok
test_pass_count_depends_on_the_seed (test_agent_data_toolkit.TestSyntheticTasks.test_pass_count_depends_on_the_seed) ... ok
test_payload_words (test_agent_data_toolkit.TestSyntheticTasks.test_payload_words) ... ok
test_pipeline (test_agent_data_toolkit.TestSyntheticTasks.test_pipeline) ... ok
test_pipeline_existing_pool_shrinks_output (test_agent_data_toolkit.TestSyntheticTasks.test_pipeline_existing_pool_shrinks_output) ... ok
test_pipeline_rejects_match_hidden_origin (test_agent_data_toolkit.TestSyntheticTasks.test_pipeline_rejects_match_hidden_origin) ... ok
test_propose_basics (test_agent_data_toolkit.TestSyntheticTasks.test_propose_basics) ... ok
test_propose_origins_do_what_they_say (test_agent_data_toolkit.TestSyntheticTasks.test_propose_origins_do_what_they_say) ... ok
test_propose_rates (test_agent_data_toolkit.TestSyntheticTasks.test_propose_rates) ... ok
test_propose_without_noise_is_all_clean (test_agent_data_toolkit.TestSyntheticTasks.test_propose_without_noise_is_all_clean) ... ok
test_default_consent_is_private (test_agent_data_toolkit.TestTraceModel.test_default_consent_is_private) ... ok
test_event_from_dict_without_meta (test_agent_data_toolkit.TestTraceModel.test_event_from_dict_without_meta) ... ok
test_event_round_trip (test_agent_data_toolkit.TestTraceModel.test_event_round_trip) ... ok
test_feedback_list (test_agent_data_toolkit.TestTraceModel.test_feedback_list) ... ok
test_prompt_and_final_text (test_agent_data_toolkit.TestTraceModel.test_prompt_and_final_text) ... ok
test_scrubbed_flag_round_trips (test_agent_data_toolkit.TestTraceModel.test_scrubbed_flag_round_trips) ... ok
test_trace_from_dict_defaults (test_agent_data_toolkit.TestTraceModel.test_trace_from_dict_defaults) ... ok
test_trace_round_trip (test_agent_data_toolkit.TestTraceModel.test_trace_round_trip) ... ok
test_count_ids_and_logger (test_agent_data_toolkit.TestTraffic.test_count_ids_and_logger) ... ok
test_deterministic_and_seeded (test_agent_data_toolkit.TestTraffic.test_deterministic_and_seeded) ... ok
test_session_prefix_and_task_id (test_agent_data_toolkit.TestTraffic.test_session_prefix_and_task_id) ... ok
test_timing_users_and_consent (test_agent_data_toolkit.TestTraffic.test_timing_users_and_consent) ... ok
test_user_consent (test_agent_data_toolkit.TestTraffic.test_user_consent) ... ok
test_weights (test_agent_data_toolkit.TestTraffic.test_weights) ... ok
test_bad_data_hurts_and_even_data_does_nothing (test_agent_data_toolkit.TestTrainOn.test_bad_data_hurts_and_even_data_does_nothing) ... ok
test_bad_data_moves_a_strong_skill_down_proportionally (test_agent_data_toolkit.TestTrainOn.test_bad_data_moves_a_strong_skill_down_proportionally) ... ok
test_clamped (test_agent_data_toolkit.TestTrainOn.test_clamped) ... ok
test_input_not_mutated (test_agent_data_toolkit.TestTrainOn.test_input_not_mutated) ... ok
test_lr (test_agent_data_toolkit.TestTrainOn.test_lr) ... ok
test_no_examples_no_change (test_agent_data_toolkit.TestTrainOn.test_no_examples_no_change) ... ok
test_scale_caps_the_step (test_agent_data_toolkit.TestTrainOn.test_scale_caps_the_step) ... ok
test_step_formula (test_agent_data_toolkit.TestTrainOn.test_step_formula) ... ok
test_deterministic (test_agent_data_toolkit.TestUserSim.test_deterministic) ... ok
test_edit_carries_the_right_answer (test_agent_data_toolkit.TestUserSim.test_edit_carries_the_right_answer) ... ok
test_no_sim_no_feedback (test_agent_data_toolkit.TestUserSim.test_no_sim_no_feedback) ... ok
test_rates_on_failure (test_agent_data_toolkit.TestUserSim.test_rates_on_failure) ... ok
test_rates_on_success (test_agent_data_toolkit.TestUserSim.test_rates_on_success) ... ok
test_rephrase_repeats_the_prompt (test_agent_data_toolkit.TestUserSim.test_rephrase_repeats_the_prompt) ... ok
test_types_by_outcome (test_agent_data_toolkit.TestUserSim.test_types_by_outcome) ... ok
test_bad_feedback_type (test_agent_data_toolkit.TestValidate.test_bad_feedback_type) ... ok
test_event_before_start (test_agent_data_toolkit.TestValidate.test_event_before_start) ... ok
test_first_must_be_user (test_agent_data_toolkit.TestValidate.test_first_must_be_user) ... ok
test_good (test_agent_data_toolkit.TestValidate.test_good) ... ok
test_no_events (test_agent_data_toolkit.TestValidate.test_no_events) ... ok
test_reward_range (test_agent_data_toolkit.TestValidate.test_reward_range) ... ok
test_system_may_come_first (test_agent_data_toolkit.TestValidate.test_system_may_come_first) ... ok
test_time_backwards (test_agent_data_toolkit.TestValidate.test_time_backwards) ... ok
test_tool_result_without_call (test_agent_data_toolkit.TestValidate.test_tool_result_without_call) ... ok
test_two_calls_two_results (test_agent_data_toolkit.TestValidate.test_two_calls_two_results) ... ok
test_unanswered_call (test_agent_data_toolkit.TestValidate.test_unanswered_call) ... ok
test_unknown_kind (test_agent_data_toolkit.TestValidate.test_unknown_kind) ... ok
test_corrupt_never_returns_the_answer (test_agent_data_toolkit.TestWorld.test_corrupt_never_returns_the_answer) ... ok
test_make_prompt_unknown (test_agent_data_toolkit.TestWorld.test_make_prompt_unknown) ... ok
test_make_task_is_consistent_and_deterministic (test_agent_data_toolkit.TestWorld.test_make_task_is_consistent_and_deterministic) ... ok
test_prompt_key (test_agent_data_toolkit.TestWorld.test_prompt_key) ... ok
test_prompt_shapes (test_agent_data_toolkit.TestWorld.test_prompt_shapes) ... ok
test_solve_each_type (test_agent_data_toolkit.TestWorld.test_solve_each_type) ... ok
test_solve_errors (test_agent_data_toolkit.TestWorld.test_solve_errors) ... ok
test_verifier (test_agent_data_toolkit.TestWorld.test_verifier) ... ok

----------------------------------------------------------------------
Ran 266 tests in 2.898s

OK

5. Discussion questions

Use these in a study group or a class. Try to answer with an example and a source. Give each about 5 to 10 minutes.

1. Pick an agent you have built or used. List the data it produces in one day. For each item, say whether it could become a demonstration, a preference pair, an eval case, or nothing, and what you would need to know first (consent, quality, copies) [3] [5].
2. The slides ask you to rate each source of agent data on cost, realism and scale. Choose a domain you care about (support chat, spreadsheets, an IDE). Which source would you start with, and what would you add after the first month [5]?
3. Husain says to remove all friction from looking at data. Design the one-screen viewer for your agent's traces. What would be on it? How many traces could you read in an hour with it [3]?
4. A flywheel needs labels, and labels cost time. Compare four ways to get them: your own team, users' thumbs, an LLM judge with human edits, and a verifier. What does each one cost, and what does each get wrong [1]?
5. Shankar's loop improves prompts and retrieved examples. The lecture's loop changes weights. Which would you pick for a product with 50 users and for a product with 5 million? What would change your mind [1] [5]?
6. Self-Instruct, Alpaca and Evol-Instruct make data with a model, and Alpaca's post lists license limits. If you generated agent data with a vendor's model, what would you check in the terms before you trained on it [6] [7] [8]?
7. The lab's difficulty filter keeps tasks that the current agent sometimes passes. What kinds of valuable tasks does it throw away? How would you rescue them [10] [11]?
8. Synatra writes steps that were never run, and AgentTrek runs them and keeps verified runs. When is the cheaper method good enough, and what would you test to find out [17] [18]?
9. Simulated users give endless practice, and the slides warn that agents learn the simulator's mistakes. Name three mistakes a user simulator might have, and how you would detect each with a small amount of real data [5] [12] [24].
10. Design an implicit-feedback signal for your product (a copy, an undo, a retry). What does it mean? How would you check it against a sample that people label? What would make it unfair to some users [26] [27]?
11. Your logs contain emails, and a regex scrubber cannot find names it does not know. What would you do before training on them? Who should be able to ask for deletion, and what would deletion mean for a model that already trained on the data [32] [33]?
12. The slides say the best model is not always the best teacher, and the Nature paper warns about training on model output. Plan a data mix for a new agent that uses a strong model's traces. What share of real data would you keep, and what would you measure each round [5] [25]?

6. Quiz (10 questions)

Pick one answer for each question, then press Check answers. Check answers 0 of 10 answered.

1. For a trajectory that is used in supervised fine-tuning, what does the model learn to produce [5]?
   • The whole recording, including the system prompt, the task text and the tool outputs, since all of it is in the training sequence and so is part of what the loss rewards
   • Only the final submitted answer, because the steps before it are noise and are removed from the sequence before the training run begins
   • The agent's thoughts and actions, while the system prompt, the task and the tool outputs are context only and are not predicted
   • Only the tool outputs, because they hold the facts that the outside world supplies and a model has to learn to reproduce those facts exactly
2. The lab prints both eval_all and eval_clean for each flywheel round. Why report both [31] [34]?
   • eval_all includes eval prompts that the agent saw in training, so the gap between the two shows how much memorization inflates the score
   • eval_clean uses a harder subset of the cases, so that it measures the agent on tasks that are not typical of the traffic it normally sees in use
   • They are two names for one number, and the lab prints both only so that the columns line up with the online pass rate in the demo output
   • eval_all is measured on live users and eval_clean on the mock, so the gap shows how far the simulator is from real traffic
3. A test set holds a reworded copy of a training prompt, with the same payload. Which check finds it [31]?
   • A split by prompt group, because the reworded copy shares the group of the original and so lands on the same side of the split as the original prompt
   • A time split, because a copy that was written later than the original is always found by comparing the time stamps of the two prompts
   • An exact hash of the prompt text, since a reworded copy has the same meaning as the original and so it hashes to the same value as the original
   • A near-duplicate search on word shingles, since an exact hash or an exact group key does not match reworded text, while shingles still overlap
4. A synthetic task has a 0.9 chance of passing on each try. The difficulty filter runs k = 8 tries and drops tasks with 0 or 8 passes. About what share of such tasks does it drop (my arithmetic)?
   • About 1 in 100, because 0.9 is far from 1 and so the eight tries should almost always differ in their outcome for a task of this kind
   • About 43 in 100, because all eight tries pass with probability 0.9 to the power 8, and all eight fail almost never, which is 0.43
   • About 90 in 100, because the chance of a pass is 0.9 per try, and the filter drops a task whenever any single try passes
   • About 0 in 100, because a task that is passed nine times in ten is by definition not easy, and so the filter always keeps it
5. The lab's MinHash search uses 16 bands of 4 rows. For a pair of texts with Jaccard similarity 0.3, what is the chance that they become candidates (my arithmetic) [31]?
   • About 0.12, from 1 minus (1 minus 0.3 to the power 4) to the power 16, because a band needs all four rows to match, and there are sixteen bands
   • About 0.30, since the chance that two signature numbers match equals the Jaccard similarity, and banding leaves that value unchanged for the pair
   • About 0.9997, because any pair with a similarity above 0.2 is almost always found when a signature is cut into sixteen bands of four rows
   • About 0.0081, because that is 0.3 to the power 4, and a pair becomes a candidate only when one chosen band matches in all four rows
6. Which description of Alpaca's training data matches the Stanford post [7]?
   • 52,000 pairs written by paid crowd workers from scratch, at about $500 for each thousand pairs, and released under a licence that allows commercial use
   • 175,000 pairs made by Self-Instruct from a seed of 52 pairs, for under $500, and licensed freely for any use including commercial products
   • 52,000 pairs made by prompting text-davinci-003 from 175 human-written seed pairs, for under $500, and released for research use only, given licenses
   • 52,000 pairs recorded from real users of a public chat demo over several months, and released with the users' consent to be shared openly
7. How does the improvement step in Shankar's flywheel post differ from the setting of this week's lecture [1] [5]?
   • The post trains the model on every logged trace without labels, while the lecture says that people should label each trace before any training
   • The post mostly updates prompts, validators and retrieved few-shot examples, while the lecture's setting changes the model's weights directly
   • The post uses environments with verifiers for reinforcement learning, while the lecture covers only prompts and retrieval for a model behind an API
   • The post has no human in the loop, since LLM judges label everything, while the lecture insists on human labels for every single trace
8. In the lab's weak-label check, tp = 50, fp = 1, fn = 7 and tn = 29. What is the recall for good traces among the labelled ones (my arithmetic)?
   • About 0.98, which is 50 of 51, the share of traces called good that are truly good, and that quantity is precision and not recall
   • About 0.73, which is 87 of 120, the share of all traces that received a weak label, and that quantity is coverage and not recall
   • About 0.91, which is 79 of 87, the share of the labelled traces that the weak label gets right, and that quantity is accuracy
   • About 0.88, which is 50 of 57, the share of truly good labelled traces that the weak label calls good, so this value is recall
9. In the lab's collapse toy, each generation draws 40 items, and the rarest item has probability 0.01. What is the chance that it is missed in one generation (my arithmetic), and why does that matter [25]?
   • About 0.67, and with no real data mixed in, an item that is missed once can never return, so the tails shrink one generation at a time
   • About 0.01, because that is its probability, and so very little is lost when an item is rare and the tails of the distribution survive each generation
   • About 0.33, so most generations keep the item, and the tails of the distribution survive in the long run even without real data
   • About 0.40, because forty draws at probability 0.01 give 0.4 expected hits, and an item with fewer than one hit is dropped for good
10. Which statement about the lab's scrubber is true [32]?
    • It finds every card number, name and address in a trace, and its residual check proves that the scrubbed set holds no personal data at all
    • It replaces each value with a random token, so the same email gets a new placeholder each time it appears again in one conversation
    • It masks a long digit string as a card only if it passes the Luhn check, and it leaves a street address in the text untouched (see demo)
    • It removes the whole trace whenever any pattern matches, so scrubbing never leaves an example that has been changed but is still used

Answer key

7. Graded assignment and rubric

Title: Build, leak-test and run a data flywheel on a mock.
Time: 4 to 5 hours. Total: 100 points. Work: solo.

The task

Extend the lab code and write a short memo (three pages at most). Mock agents and toy tasks are fine. You may use a real agent or real logs if you have them, but say clearly which parts used them, get consent for any real person's data, and scrub it before you store it. Build your own task domain. Do not reuse the lab's six task types or its numbers. Label every number that comes from a mock or from made-up data. Keep the code, the data and the memo consistent with each other [1] [3].

Part A. Traces and conversion (25 points)

• Define at least four new task types in a domain that is not the lab's (for example, log parsing, unit conversion, or date arithmetic), each with a verifier. Log at least 200 mock traces with your own consent rule [3].
• Extend the trace schema with two fields you think a real product needs. Extend validate_trace with two new rules, with at least four tests each [30].
• Produce demonstrations, preference pairs and eval cases. Give a table of how many traces survive each gate (valid, passed, consented, scrubbed). Say which gate removes the most and what bias it adds [5].
• Read 20 of your traces by hand. Report three things you learned that a metric would not show [3].
• Name three limits of your traces.

Part B. Splits and leakage (25 points)

• Make a random split, a group split and a time split of your traces. Run leakage_report on each, and give a table of exact copies, near copies, shared groups and time violations [31].
• Add 10 exact copies and 10 reworded copies on purpose. Show which check finds which, and what decontaminate removes.
• Run the near-duplicate search with thresholds 0.5, 0.7 and 0.9. Predict, with the banding formula of 2.10, how often a pair of Jaccard 0.5 is a candidate, then measure it on at least 200 made pairs. Report both numbers [31].
• Write the "split and leakage" paragraph of a data card in your own words, with the date and the rule you used [33].
• Name three limits of your leak checks.

Part C. Flywheel and synthetic data (25 points)

• Run the flywheel for four variants (base, no filter, label noise at 30%, no labels) with at least three seeds each. Report the mean and the range of eval_clean by round, in a table and a plot. Say what you would and would not claim from it, since the training rule is made up [1].
• Write your own task generator for your domain, with at least one new kind of bad candidate. Run the funnel and give a table of what each filter rejected. Say which bad candidates got through and why [6] [10].
• Measure diversity before and after filtering. Change one filter setting and show what happens to the diversity [3].
• Say how Shankar's loop would differ from yours if the improvement step were a prompt change [1].
• Name three limits of your flywheel.

Part D. Feedback, privacy and the memo (25 points)

• Design two implicit signals for your domain. Hand-label 40 mock traces. Report coverage, precision and recall of the weak label against your labels, and say what would be different for real users [26] [27].
• Test the scrubber on 20 strings that hide PII in unusual ways. Report how many it misses. Add one pattern with tests. Say what your residual check can and cannot tell you [32].
• Generate a data card for your final training set with make_data_card. Make it fail validation in two ways, and name the rule each time [33].
• Write a memo of two pages at most. Your team has an agent with 10,000 sessions a week. Should it start a data flywheel? Give a clear yes or no, the data to collect, the consent and retention rules, how many labels per week, the risks (accidental learnings from 2.8), the stop criteria, and three limits of your evidence [1] [25] [35].

What to hand in

• Your agent_data_toolkit.py (or your own file names), your tasks and data, and the tests. All tests must pass.
• A memo in PDF or Markdown with the tables from Parts A to C, the data card, and the memo from Part D.
• A sources list for any reading you cite.

Rubric

Criterion	Full marks	Partial	Points
A. Traces and conversion	Four new task types with verifiers, 200 or more traces, two schema fields, two new validator rules with eight tests, the three products, a gate table with a bias note, 20 traces read with three findings, three limits.	Fewer traces or types, or no gate table, or no hand reading, or no tests for the rules.	25
B. Splits and leakage	Three splits with a leakage table, seeded exact and reworded copies found by the right checks, a threshold study with a predicted and a measured candidate rate, a data card paragraph, three limits.	Two splits only, or no prediction, or no seeded leaks, or the reworded copies are missed without comment.	25
C. Flywheel and synthetic data	Four variants with three seeds each, a table and a plot with ranges, an honest claim, an own generator with a new bad kind and a funnel table, diversity before and after, the comparison with Shankar, three limits.	One seed, or no ranges, or no new bad kind, or no diversity measure, or claims that go beyond the mock.	25
D. Feedback, privacy and memo	Two signals with measured quality against 40 hand labels, a scrubber test with a miss count and a new pattern, a card that validates and two failing variants, a two-page memo with a clear decision, rules, risks, stop criteria and limits.	No hand labels, or no miss count, or a memo without a decision or without stop criteria.	25
Total			100

Grade bands: 90 to 100 excellent. 75 to 89 good. 60 to 74 needs work. Under 60: redo and resubmit. Plain writing is rewarded: short sentences, no filler. Label your own arithmetic as your own.

8. Glossary

Term	Meaning
Accidental learning	Something a model picks up from data that you did not mean to teach, such as a reward hack or the habits of one harness [5].
Banding (LSH)	Cutting a MinHash signature into bands. Two texts are candidates if any band matches in all rows. The chance is 1 − (1 − sr)b for Jaccard s, r rows and b bands (my arithmetic, 2.10).
Binary metric	A true or false check. Shankar prefers it because it is easier to align and to judge consistently [1].
Bradley-Terry	A model of pairwise comparisons: the chance that A beats B is sA / (sA + sB) [26].
Consent flag	A per-trace yes or no for one purpose: log, train or share. The lab's defaults are my choice.
Contamination	Evaluation data that appears in training data [34].
Data card	A short document that says what a dataset is, where it came from, what was done to it and what it must not be used for. Related to datasheets for datasets [33].
Data flywheel	A loop in which a deployed system produces logs, some are labeled, and the labels improve the system, which produces better logs [1].
Decontamination	Removing training items that are exact or near copies of test items.
Demonstration	A recorded example of a task done well, used for supervised fine-tuning [5].
Difficulty filter	A filter that keeps synthetic tasks that the current agent sometimes, and not always or never, solves. It is noisy (2.10).
DPO	Direct Preference Optimization, a way to train on chosen and rejected answers with a simple loss [29].
Environment	A task, a sandbox and a way to check success, used to practice with RL [5].
Eval case	A test, with a verifier, often made from a failed run [3].
Explicit feedback	A signal the user gives on purpose: a vote, a thumb, a correction [26].
Harness	The loop and tools around a model. Training on one harness can teach its habits [5].
Implicit feedback	A signal from behaviour: copy, regenerate, rephrase, abandon. Common and noisy (my own list).
Jaccard similarity	The size of the overlap of two sets divided by the size of their union. The lab applies it to sets of word shingles [31].
Luhn check	A checksum for card numbers. The lab masks a digit string as a card only if it passes.
MinHash	A short signature of a set such that the chance two signature numbers match equals the Jaccard similarity of the sets.
Model collapse	The loss of the tails of a distribution when models are trained on the output of models [25].
PII	Personally identifiable information: names, emails, phone numbers and the like [32].
Preference pair	A prompt with a chosen and a rejected answer [29].
Retention	How long data is kept before it is deleted.
Reward hacking	Getting a high reward by exploiting a loophole in the check [35].
Scrubbing	Replacing PII with placeholders. It has false negatives.
Self-Instruct	Bootstrapping instruction data from a model's own generations, with filters [6].
Shingle	A run of k consecutive words, used to compare texts.
Synthetic data	Data made by a model, or by a program that uses a model [2].
Trace	The log of one run: messages, thoughts, tool calls, results and feedback [3].
Trajectory	A recording of an agent solving a task, used for SFT [5].
Validator	A check on an output or an input. Shankar places them at nodes of an LLM graph [1].
Weak label	A label guessed from signals, not from a person or a verifier. Measure it against a checked sample.

9. References

Entries marked (required) are the course's required reading, and (course, additional) is the course's additional reading. Entries marked (added) are my own additions. The course site and slides, [4] and [5], are the course's own pages. "Abstract level" means that I read the abstract and not the whole paper. Where an entry names more, that is what I read. Web pages were read in October 2026.

• [1] Shankar, S. "Data Flywheels for LLM Applications." Blog post, 1 July 2024. An essay with no experiments. https://www.sh-reya.com/blog/ai-engineering-flywheel/ (I read the whole post; footnotes included) (required)
• [2] Tan, Z., Li, D., Wang, S., Beigi, A., and 6 others. "Large Language Models for Data Annotation and Synthesis: A Survey." EMNLP 2024. arXiv:2402.13446 (2024). https://arxiv.org/abs/2402.13446 (I read the abstract only, not the paper) (course, additional)
• [3] Husain, H. "Your AI Product Needs Evals." Blog post, 29 March 2024. The page carries a banner for the author's paid course on evals. https://hamel.dev/blog/posts/evals/ (I read the sections on tracing, looking at traces, fine-tuning, and data synthesis and curation, and skimmed the rest) (added)
• [4] Stanford CS329Z: Engineering AI Agents, course site and schedule (Diyi Yang, Michael Ryan, John Yang). https://cs329z.stanford.edu/ (the course's own page; it lists the week's topics and readings and says the schedule is tentative)
• [5] CS329Z, Lecture 11 slides, "What Data Do Agents Need?" (Wednesday, 28 October 2026). https://web.stanford.edu/class/cs329z/slides/lecture11.pdf (the course's own slides; I used them for coverage, and where I repeat a number or a claim that the slides take from another work, I say "the slides")
• [6] Wang, Y., Kordi, Y., Mishra, S., Liu, A., and 3 others. "Self-Instruct: Aligning Language Models with Self-Generated Instructions." arXiv:2212.10560 (2022). https://arxiv.org/abs/2212.10560 (abstract level) (added)
• [7] Taori, R., Gulrajani, I., Zhang, T., and others. "Alpaca: A Strong, Replicable Instruction-Following Model." Stanford CRFM blog, 13 March 2023. https://crfm.stanford.edu/2023/03/13/alpaca.html (I read the whole post; project-authored) (added)
• [8] Xu, C., Sun, Q., Zheng, K., Geng, X., and 5 others. "WizardLM: Empowering large pre-trained language models to follow complex instructions." arXiv:2304.12244 (2023). https://arxiv.org/abs/2304.12244 (abstract level; the authors' own evaluation claims) (added)
• [9] Mitra, A., Del Corro, L., Zheng, G., Mahajan, S., and 10 others. "AgentInstruct: Toward Generative Teaching with Agentic Flows." arXiv:2407.03502 (2024). Not the same project as the AgentInstruct dataset in [13]. https://arxiv.org/abs/2407.03502 (abstract level; authors' own claims) (added)
• [10] Liu, Z., Hoang, T., Zhang, J., Zhu, M., and 13 others. "APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets." arXiv:2406.18518 (2024). https://arxiv.org/abs/2406.18518 (abstract level; authors' own claims) (added)
• [11] Liu, W., Huang, X., Zeng, X., Hao, X., and 23 others. "ToolACE: Winning the Points of LLM Function Calling." arXiv:2409.00920 (2024). https://arxiv.org/abs/2409.00920 (abstract level; authors' own claims) (added)
• [12] Prabhakar, A., Liu, Z., Zhu, M., Zhang, J., and 11 others. "APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay." arXiv:2504.03601 (2025). https://arxiv.org/abs/2504.03601 (abstract level; authors' own claims; the slides cite it too) (added)
• [13] Zeng, A., Liu, M., Lu, R., Wang, B., and 3 others. "AgentTuning: Enabling Generalized Agent Abilities for LLMs." arXiv:2310.12823 (2023). Its dataset is also called AgentInstruct; see [9]. https://arxiv.org/abs/2310.12823 (abstract level) (added)
• [14] Chen, B., Shu, C., Shareghi, E., Collier, N., and 2 others. "FireAct: Toward Language Agent Fine-tuning." arXiv:2310.05915 (2023). https://arxiv.org/abs/2310.05915 (abstract level) (added)
• [15] Deng, X., Gu, Y., Zheng, B., Chen, S., and 4 others. "Mind2Web: Towards a Generalist Agent for the Web." arXiv:2306.06070 (2023). https://arxiv.org/abs/2306.06070 (abstract level; the slides give a task count (2,350) that I did not check) (added)
• [16] Zhou, S., Xu, F. F., Zhu, H., Zhou, X., and 8 others. "WebArena: A Realistic Web Environment for Building Autonomous Agents." arXiv:2307.13854 (2023). https://arxiv.org/abs/2307.13854 (abstract level) (added)
• [17] Xu, Y., Lu, D., Shen, Z., Wang, J., and 4 others. "AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials." arXiv:2412.09605 (2024). https://arxiv.org/abs/2412.09605 (abstract level; the $0.55 figure is the authors' claim) (added)
• [18] Ou, T., Xu, F. F., Madaan, A., Liu, J., and 6 others. "Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale." arXiv:2409.15637 (2024). https://arxiv.org/abs/2409.15637 (abstract level; the $0.031 and 3% figures are the authors' claims) (added)
• [19] Murty, S., Zhu, H., Bahdanau, D., and Manning, C. D. "NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild." arXiv:2410.02907 (2024). https://arxiv.org/abs/2410.02907 (abstract level; authors' own claims) (added)
• [20] Sun, Q., Cheng, K., Ding, Z., Jin, C., and 11 others. "OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis." arXiv:2412.19723 (2024). https://arxiv.org/abs/2412.19723 (abstract level; authors' own claims) (added)
• [21] Pan, J., Wang, X., Neubig, G., Jaitly, N., and 3 others. "Training Software Engineering Agents and Verifiers with SWE-Gym." arXiv:2412.21139 (2024). https://arxiv.org/abs/2412.21139 (abstract level; authors' own claims) (added)
• [22] Yang, J., Lieret, K., Jimenez, C. E., Wettig, A., and 6 others. "SWE-smith: Scaling Data for Software Engineering Agents." arXiv:2504.21798 (2025). The first author, John Yang, teaches the course. Next week's reading. https://arxiv.org/abs/2504.21798 (abstract level; authors' own claims; the slides' "about 20 hours of human work" is not in the abstract) (added)
• [23] Wei, Y., Duchenne, O., Copet, J., Carbonneaux, Q., and 5 others. "SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution." arXiv:2502.18449 (2025). https://arxiv.org/abs/2502.18449 (abstract level; the slides' SFT comparison is not in the abstract and I did not check it) (added)
• [24] Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045 (2024). https://arxiv.org/abs/2406.12045 (abstract level) (added)
• [25] Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., and Gal, Y. "AI models collapse when trained on recursively generated data." Nature 631, 755 to 759 (24 July 2024). https://www.nature.com/articles/s41586-024-07566-y (I read the article text; I did not check its derivations or rerun anything) (added)
• [26] Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., and 7 others. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." arXiv:2403.04132 (2024). https://arxiv.org/abs/2403.04132 (abstract level) (added)
• [27] Gao, G., Taymanov, A., Salinas, E., Mineiro, P., and Misra, D. "Aligning LLM Agents by Learning Latent Preference from User Edits." arXiv:2404.15269 (2024). Introduces PRELUDE and CIPHER. https://arxiv.org/abs/2404.15269 (abstract level) (added)
• [28] Zhao, W., Ren, X., Hessel, J., Cardie, C., and 2 others. "WildChat: 1M ChatGPT Interaction Logs in the Wild." arXiv:2405.01470 (2024). https://arxiv.org/abs/2405.01470 (abstract level) (added)
• [29] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., and 2 others. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv:2305.18290 (2023). https://arxiv.org/abs/2305.18290 (abstract level; I did not check the derivation) (added)
• [30] OpenTelemetry. "Semantic Conventions for Generative AI." Repository, status Development. Reused from Week 3b. https://github.com/open-telemetry/semantic-conventions-genai (I read the client inference, agent span and metrics pages for Week 3b, and did not reread them for this week; names may change) (added)
• [31] Lee, K., Ippolito, D., Nystrom, A., Zhang, C., and 3 others. "Deduplicating Training Data Makes Language Models Better." arXiv:2107.06499 (2021). https://arxiv.org/abs/2107.06499 (abstract level) (added)
• [32] Carlini, N., Tramer, F., Wallace, E., Jagielski, M., and 8 others. "Extracting Training Data from Large Language Models." arXiv:2012.07805 (2020). https://arxiv.org/abs/2012.07805 (abstract level) (added)
• [33] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., and 3 others. "Datasheets for Datasets." arXiv:1803.09010 (2018). https://arxiv.org/abs/1803.09010 (abstract level; the lab's data card is my own simplification, not the datasheet questionnaire) (added)
• [34] OpenAI. "Why SWE-bench Verified no longer measures frontier coding capabilities" (page title "Why we no longer evaluate SWE-bench Verified"). Vendor-authored. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ (I read the opening sections as a search-result excerpt; my fetch of the page was refused, so I did not read the rest, and I did not read its date) (added)
• [35] Anthropic. "From shortcuts to sabotage: natural emergent misalignment from reward hacking." Blog, 21 November 2025. Vendor-authored. The slides cite it, with the paper by MacDiarmid et al. https://www.anthropic.com/research/emergent-misalignment-reward-hacking (I read the blog post, and not the paper) (added)
• [36] Anthropic. "Demystifying evals for AI agents." Engineering blog, 9 January 2026. Vendor-authored. Reused from Week 6a. https://www.anthropic.com/research/demystifying-evals-for-ai-agents (I reread the passage on capability and regression evals for this week) (added)