Crabhaus

What it is

An AI agent that operates software the way a person does: it looks at the screen, moves the cursor, clicks, and types instead of calling an API. The model gets control of a real desktop or browser and works through the same interface you do - including apps that were never built for automation.

How it works

A screenshot-act loop. Each turn the harness captures the screen - often with the accessibility tree, the structured element list the OS exposes for screen readers - and the model replies with one action: click these coordinates, type this text, scroll. Execute it, take a fresh screenshot, repeat until done. The hard part is grounding: turning 'the blue Submit button' into exact pixels.

A concrete example

Ask Claude's computer-use tool to file an expense report: it screenshots the browser, clicks into the expenses app, types amounts into form fields, scrolls to the submit button, and reads the confirmation screen - dozens of model calls for a task a human finishes in a minute.

Why it matters

Most software has no API: internal dashboards, legacy desktop apps, government portals. Computer use is the only general route to automating that long tail. It is also far harder than API tool use - an API has a typed contract; a GUI offers ambiguous pixels and shifting layouts. On OSWorld's real desktop tasks, humans hit about 72% while the best early agents scored under 15%.