An agent that checks a flow is only useful if its behaviour is constant. The moment it carries memory across sessions, a failing check has two possible causes and you cannot tell them apart. Teams end up re-verifying agent output by hand, which is the work the agent was supposed to remove.
The agent
| system_prompt | Report validation failures you can observe: field rules, accessibility, duplicate submissions. Every finding names the field, the expected rule, the observed behaviour, and cites its source. |
| tool_config | web search, page fetch, domain allowlist on your staging host |
| output_schema | findings[] with field, expected, observed, severity — validated on every run |
The workflow
Pin the suite
Every nightly job opens its chat against site-tester v5 explicitly. The job holds a version id, so a publish mid-run changes nothing.
Save the failures as fixtures
A run you accept becomes a regression fixture: the input plus the accepted output hash.
Upgrade deliberately
Publish v6, batch-rerun the fixture set in replay mode, and read the table: 31 byte-identical, 6 better, 3 worse.
Promote or roll back
Move the stable pointer when the numbers justify it. Rollback is a pointer move, not a republish.
What you get
Pin an agent. Run it in a year. Get the same thing back.
Claude, ChatGPT and Gemini included — no provider account to open. Build one agent, publish v1, run it, then change the prompt and diff the two.