Computer-Use Agents

A computer-use agent drives a real screen: screenshot → model → action (click(x, y), type(text), scroll) → new screenshot → repeat. No DOM, no API — the GUI itself is the interface, so it works on software that was never built for automation.

The hard part is grounding: mapping 'the Submit button' to pixel coordinates. Vision-language models have gotten startlingly good at it — and still wrong often enough that a 30-click flow multiplies small grounding errors into frequent misses. Every click is a coin flip you're betting the trajectory on.