Workflows
Diffing and rerunning
Source modes, the five diff layers, verdicts and regression fixtures.
Source mode
Live web search is the enemy of a clean comparison. Pick the mode that isolates the variable you actually care about.
| Mode | What it does | Use when |
|---|---|---|
| replay | Re-serves the exact sources the original run read. The web is frozen. | Isolating a prompt change. The default for version diffs. |
| refresh | Re-runs search live for both sides, in the same minute. | Comparing versions on today's web. |
| live | A normal run. Only side B is fresh; side A is historical. | Never, for diffs. Available for honesty. |
The five layers
- Config — what differs between the two versions: prompt, tools, schema, model settings.
- Output — word-level diff for prose, field-by-field for schema'd output. An output-hash match shows a byte-identical badge and skips the diff entirely.
- Sources — a set diff over cited URLs with fetch timestamps and content hashes. Catches the answer changed because the page changed.
- Tool calls — ordered per side. Catches v5 makes four times the fetches for the same answer.
- Telemetry — latency, tokens, cost, finish reason.
Verdicts
Mark a comparison a_better, b_better, equivalent or both_bad, with a note. Verdicts roll up per version pair — v3 → v5 across 40 inputs: 31 equivalent, 6 better, 3 worse — which is the number to look at before moving the stable pointer.
Regression fixtures
An accepted run can be saved as a fixture: the input plus the accepted output hash. Any future version can be run against the fixture set, and a hash mismatch flags the row for human review rather than failing silently.