Loading patterns…
Computer Use(CU)
Agents operate graphical user interfaces (desktop, browser, or mobile) the way a person does, by reading screenshots and issuing mouse and keyboard actions instead of calling APIs. The core challenge is GUI grounding: mapping an intent to exact pixel coordinates on the screen. The agent runs a loop of screenshot, reason, act, and observe until the task is complete.
In 30 seconds
- What
- Agent reads screenshots, grounds intent to pixel coordinates, executes mouse and keyboard actions, loops until task completion.
- When to use
- Systems with no API, complex multi-step UI workflows, or when you need to automate legacy or third-party applications.
- Watch out
- GUI grounding fails on dynamic layouts, small targets, or obstructed elements, causing cascading errors that are hard to recover from.
Ask the AI expert about this pattern
Opens the assistant with your question prefilled. You review it before sending.
Computer Use: Overview
Agents operate graphical user interfaces (desktop, browser, or mobile) the way a person does, by reading screenshots and issuing mouse and keyboard actions instead of calling APIs. The core challenge is GUI grounding: mapping an intent to exact pixel coordinates on the screen. The agent runs a loop of screenshot, reason, act, and observe until the task is complete.
- Screenshot-based perception of the live screen
- GUI grounding from intent to precise pixel coordinates
- Low-level action space: click, type, scroll, drag, key press
- Perception-reason-act-observe loop with feedback after each step
- Error recovery from popups, modals, and unexpected UI states
- Works on applications and sites that expose no API
The Agent Architect
One pattern, one tradeoff, one production failure story. A short weekly briefing for people building agentic systems.
Weekly email, one-click unsubscribe. We only use your address to send the briefing.
References
The papers, specifications, and repositories this pattern is based on.
From the engineer behind this catalog
Get your agent architecture reviewed
This page documents one pattern. Your system runs dozens, and most failures live in how they fit together. Have the whole design reviewed against the 288 patterns in this catalog: architecture, reliability, evaluation and cost, every finding mapped to the pattern that fixes it.
€750 instead of €1,500, one week, written report and walkthrough call, until 30 September