🧪anneal

a QA engine for web apps. an agent writes the test once, then plain code replays it. github.com/rathoresids/anneal, python, MIT, built on the side.

why

there are two ways to test a web app right now and both annoy me.

hand-written playwright suites break every time the UI moves, and they break identically whether someone renamed a testid or deleted the whole feature. CI can't tell you which one happened.

the newer approach puts an ai agent in the browser on every run. that solves the brittleness and buys a worse problem: the agent adapts to whatever it finds, so real regressions sail through. i benchmarked it against an actual product redesign. the frozen replay went red in 29 seconds. the live agent worked around the same change and reported green in 252 seconds.

so anneal puts the model in the authoring and the repair, never in the run.

write once ──▸ bundle v1 ──▸ replay ──▸ red ──▸ heal or rewrite ──▸ bundle v2

how it works

one artifact, a test bundle. three things produce or consume it.

authoring is the only place an llm drives a browser. it explores the app, asks me about anything ambiguous, then writes each step out with a locator, a fallback ladder, a postcondition and the network calls it expects. then it freezes.

healing runs when a locator drifts. it walks the stored fallback ladder first with no model involved, calls one only if the ladder runs out, checks that red turns green, and ships a new version with a changelog line.

replay has no llm anywhere in it. the bundle compiles to a playwright step plan and each step's postcondition doubles as its wait. anneal export prints that same plan as a standalone playwright script, so the exported file can't drift from what ran.

three things i'd keep

• the replay path can't reach a model. anneal-executor isn't allowed to import anneal-llm, and a test asserts that in a fresh subprocess. everything smart arrives through one constructor.
• the engine knows nothing about any specific app. no routes, testids or domain words in the code or the prompts. a grep check runs before every build gate, plus a second one for a domain word smuggled into a prompt without a literal.
• it won't guess what you meant. before freezing a test it asks about the choices that change what gets asserted, and an unanswered question blocks the freeze. then it breaks the app on purpose to prove the assertion can fail.

done means ./verify.sh exits 0, nothing else. there's also a holdout gate: point it at flows it has never seen in a repo it doesn't own, and each one has to author and replay to the right verdict.