Capstone: Build an Agent Evaluation Framework
Assemble every piece into one framework for a real agent — dataset, graders, judge, reliability report, CI gate, and production feedback loop.
Assemble every piece into one framework for a real agent — dataset, graders, judge, reliability report, CI gate, and production feedback loop.
Running evals on every prompt, model, and tool change — tiered suites, baselines, statistically sound gates, and a working GitHub Actions workflow.
Exact match, schema validation, state checks, and environment assertions — the cheap, reliable graders that should carry most of your eval weight.
Designing an agent so it can be measured — traces, spans, structured events, replayable tool layers, and the seams where evaluators attach.
The four evaluation layers — component, step, trajectory, and outcome — plus the core abstractions every framework needs: Case, Run, Grader, Score, Report.
Building eval sets that actually represent production — sourcing from real traffic, synthetic generation, coverage matrices, stratification, and dataset versioning.
Rubric design, pairwise vs pointwise judging, known judge biases, and calibrating a judge against human labels until you can trust its numbers.
Simulated users, conversation-level success, memory recall tests, and attributing failures to the right agent in supervisor/worker systems.
Sampling live traces, reference-free evaluators, drift detection, user-feedback signals, and the loop that turns production failures into new test cases.
pass@k vs pass^k, variance and confidence intervals, sample sizes, and the cost and latency budgets that belong in every eval report.
Showing 10 of 60