"We still run on-site Excel aggregation and ERP data transcription using RPA, but the bots break constantly"—this is an increasingly frequent consultation from IT teams at companies that implemented RPA several years ago. The fragility of breaking on subtle screen layout changes, persistently high license fees, and the siloed maintenance of scenarios are driving demand for second-generation RPA replacements.
That is where Anthropic's Computer Use API makes a compelling entrance. Built on the concept of passing screenshots to an LLM and having it return coordinates and keystrokes, it operates based on screen semantics rather than pixel coordinates, enabling automations that withstand UI changes. In this article, we outline design guidelines for adopting Computer Use as a business automation engine from the perspective of custom development practitioners.
Why Computer Use Is "Second-Generation RPA"
The biggest difference compared to traditional RPA lies in how screen elements are identified.
| Dimension | Traditional RPA (UiPath, WinActor, etc.) | Computer Use |
|---|---|---|
| Screen element identification | XPath, coordinates, image matching | Screenshots + LLM visual comprehension |
| Behavior on UI changes | Stops when layouts change | Can locate "Submit buttons" by meaning |
| Scenario creation | Recorder or GUI builder | Natural language instructions |
| Exception handling | Hand-code individual branches | Convey policies via prompts |
| Unit cost | Number of workflows × annual license fee | Token-based API billing |
Token-billed Computer Use is especially suited for long-tail tasks, such as "wanting to automate year-end financial closing processes that run only a few times a year, where RPA license costs cannot be justified."
Four Design Decisions When Introducing It in Custom Development
Before putting Computer Use into production, there are four key issues that must be agreed upon prior to estimating custom development.
1. Execution Environment — Can Virtual Desktops Be Made Disposable?
Computer Use generates arbitrary clicks and keystrokes during execution. Running it directly on production workstations should be avoided because the blast radius of operational mistakes cannot be predicted. We recommend a setup using disposable instances of Docker + noVNC or cloud virtual desktops (WorkSpaces, WorkSpaces Web, Cloud Workstations).
2. Guardrails — Operating Under the Assumption That "Dangerous Actions" Cannot Be Stopped by Prompts Alone
When prompts are ambiguous, LLMs might click delete or submit buttons. Rather than relying solely on prompt-based deterrence, enforce a mandatory human review for destructive actions via an external policy layer. Specifically, implement a three-tiered defense:
- URL Whitelist — Restrict accessible domains and subpaths
- Keyword Guards — Pause execution when detecting terms like "Delete", "Submit", or "Confirm"
- Approval Queue — Convert critical actions into approval tickets, returning the next screenshot only after human approval
3. Prompt Maintenance — Managing Scenarios as "Instruction Manuals" in SCM
Teams accustomed to recorder-based workflows often overlook where prompts should be stored. Establish a workflow from the beginning where Markdown instruction sheets are managed in a repository and subjected to PR reviews. This follows the same philosophy as Practical Guide to Building In-House MCP Servers (2026 Edition): operational AI assets must be version-controlled with the same rigor as code.
4. Cost Estimation — Translating Token Billing into "Per-Job Unit Costs"
When presenting to executives, decision-making is impossible with raw token pricing. Measure number of screenshots per job × average tokens × unit price and translate it into a metric such as "8 yen per invoice issued," making ROI comparisons against RPA licensing straightforward.
Three Business Workflow Patterns to Validate in a PoC
When deciding where to begin internally, the following three patterns offer the highest PoC success rates:
- Data transcription to existing systems — Order management → accounting, CRM → marketing automation, etc. Interface change frequency is low, and token consumption is predictable
- Operating external SaaS that only provides a GUI — Legacy SaaS without APIs, municipal e-filing portals, etc.
- Automated report generation — Taking screenshots across multiple systems, pasting them into designated templates, and posting to Slack
Conversely, sub-millisecond processing or high-frequency batches exceeding 10,000 requests per day are better suited to direct API integrations or standard approaches like an edge API infrastructure using Hono + Cloudflare Workers, rather than Computer Use.
Conclusion: Computer Use Is the "Savior of Long-Tail Tasks"
Computer Use is not a silver bullet that replaces every RPA tool. However, for workflows plagued by the triple constraints of no APIs, low frequency, and specialized domain knowledge requirements, it possesses the power to make previously uneconomical automation instantly viable.
At GleamHub, we partner with clients from Computer Use PoCs to production design, guardrail implementation, and cost management systematization. If you are looking to reorganize your existing RPA or semi-automate infrequent tasks, please feel free to reach out via our contact page.
