Problem
A client says “the chatbot sometimes gives wrong answers.” That’s not actionable. What’s actionable is: retrieval recall@5 is 0.62, so the right document isn’t retrieved for 38% of questions and the generator is answering from whatever it got; faithfulness is 0.81, so 19% of answers contain at least one claim the context doesn’t support; here are the 14 cases and the exact sentences that were fabricated.
System
One interface, the adapter, is what makes the tool reusable. Anything that implements it can be audited: my own project, a client’s LangChain stack, or a REST endpoint I’ve never seen. rageval init writes a config, rageval run produces HTML and JSON, rageval calibrate checks the judge, and a baseline plus CI gate makes regressions a failing build.
Worth knowing
Four rules separate an audit tool from a scorecard generator, and each one is a design decision here:
- An unmeasurable metric is absent, never zero. If a case has no relevant-document labels, recall isn’t computed for it rather than scored 0.0 and averaged in. Every number carries its true sample size.
- The judge is calibrated, not trusted. Calibration reports Cohen’s kappa against held-out human labels, and if kappa is below 0.6 the report says so, in the report.
- Failures are diagnosable. Faithfulness decomposes each answer into atomic claims and verifies each separately, so the report points at the sentence, not just a score.
- One bad case never kills a run. Errors are captured per case, excluded from aggregates, and counted.
This is the closest thing on this site to a productized Design Sprint deliverable: the audit that tells you what to fix before anyone builds anything.