Problem
Businesses burn thousands of manual hours copying data between SaaS tools that lack APIs or gate integrations behind enterprise tiers. The workaround is a person clicking. The better workaround is an agent that navigates the UI like a person: it looks at the page, reads its accessibility tree, and acts, click by click.
System
An OBSERVE → DECIDE → ACT loop. Observe: a state parser produces numbered interactive elements, a flattened accessibility tree, and a viewport screenshot. Decide: the vision LLM returns exactly one JSON action (click, type, scroll, extract, done). Act: a Playwright primitive runs it with retries, popup dismissal, and a self-heal re-plan when the page didn’t do what the model expected. Every step is logged with the model’s reasoning, the action, and a screenshot.
Worth knowing
Combining the accessibility tree with the screenshot matters more than either alone. The tree gives precise, clickable targets and text the model can’t misread; the screenshot catches what the tree misses (visual state, overlays, layout that changes meaning). Sending only screenshots produces mis-clicks; sending only the tree produces confident actions on elements that aren’t actually visible.
Brittleness is the honest risk with any RPA approach: sites change. The self-heal re-plan and per-step logs exist so that when a workflow breaks, you can see exactly which step and why, instead of a silent failure at 3am.