agentic verification strategy: UI

[aaronspring]

Currently I mostly implement a feature (with help of agents like claude code). Verification is mostly done manual:

• sure... claude does a lot a verification already but mostly pytest
• manual by user: starting the app, clicking around

Proposal: setup a verification system for the agent to start the app, click around and take screenshots to verify

This doesn't prevent the human from verification. Final (UX/UI) review still required. But this would help a lot.

[aaronspring]

edit: claude does this already: tmux send-keys ... clicks. However having clear instructions in PR template and AGENTS.md would help other than Claude bootstrapping each time

[clstaudt]

If I understand correctly, you mean testing via the UI. Excellent idea.

I have had good first experiments with:

• https://peekaboo.sh
• Claude Sonnet
• osascript

Nearly fully automated testing of the app via the UI was possible with enough time and tokens. Token budget needed was significant. So basically, cost was what kept me from running this all the time.

[aaronspring]

Token budget needed was significant. So basically, cost was what kept me from running this all the time.

Tokens and time bloats massively. I guess with clear instructions how the agent is supposed to verify it would go faster and cheaper. Something like what playwright is for the browser but for electron UI

[aaronspring]

There is playwright electron to be used

[clstaudt]

@aaronspring Is this still something that needs any configuration in the repo?

I noticed that Opus 5 automates UI testing quite reliably.

[aaronspring]

I think a PR adding to AGENTS.md giving guardrails of what and how to UI test and what's overkill would be helpful. Also about how to open an UI changing PR