AI infrastructure · Native automation · MCP tooling

sky-cua

sky-cua is a cross-platform computer-use platform for AI agents: a native automation layer that lets agents inspect screens, control desktop apps, drive browsers, operate Android devices, and report failure modes cleanly enough to be useful in real workflows.

The problem

Most agent automation breaks down at the boundary between the model and the real machine. Screenshots are flaky, accessibility trees vary wildly, browser state lives behind a separate protocol, Android needs ADB, and every host expects tools to be packaged slightly differently. sky-cua was built to make that messy boundary explicit, observable, and portable.

What I built

A long-lived Rust client/service architecture with platform IPC, backend abstraction, structured diagnostics, and host-portable MCP tools.
Desktop automation foundations across Linux and Windows, including AT-SPI, Wayland/X11 capture and input, Win32/UIA discovery, GDI capture, and SendInput.
A reduced MCP interface that cut the exposed tool surface from 66 tools to 34 by replacing one-off commands with a unified, schema-validated action surface targeted at the active control surface.
Browser-control tooling for tab discovery, page snapshots, screenshots, CLI integration, and native-host validation.
Android integration through ADB for discovery, pairing, connection, UI inspection, screenshots, and input control.
Packaging and smoke-test workflows for Codex, OpenClaw, Claude Code, OpenCode, and other MCP hosts.

How it works

I built sky-cua around a small but stubborn idea: the agent should get a stable set of capabilities, but the platform layer should never pretend that Linux, Windows, browsers, and Android all fail in the same way. The Rust service owns the messy native work — capture, input, accessibility, browser plumbing, device state — and exposes it through MCP tools that are boring on purpose.

Each backend reports what it can actually do before the agent starts clicking at ghosts. A desktop workflow can ask for a screenshot, inspect available UI structure, choose semantic or coordinate-based input, and get a clear failure if the compositor, accessibility layer, or input backend is the thing refusing to cooperate. Browser and Android workflows follow the same pattern: discover first, act second, report precisely.

One of the important design turns was shrinking the public tool surface from 66 tools to 34. Instead of making the agent choose between dozens of slightly different click, type, browser, phone, and desktop operations, the interface routes through a smaller set of action primitives and binds those actions to the relevant control surface. That gives the model fewer knobs to misuse while preserving the platform-specific detail underneath.

Architecturesky-cua keeps the model-facing contract stable while each native backend reports its own real capabilities and failures.

Agent action loopThe important part is the evidence loop: discover, inspect, act, and return a useful result instead of pretending the desktop is a clean API.

How it achieves the goal

The goal is not to make agents “move the mouse.” That part is easy and mostly useless. The goal is to make computer-use repeatable enough that an agent can recover from reality: missing windows, stale screenshots, native host failures, disconnected phones, unsupported capture modes, or a UI that moved between attempts. sky-cua achieves that by making readiness, capability, action, and diagnosis separate parts of the contract instead of one big optimistic “do the thing” button.

That shape makes it portable across hosts like Codex, OpenClaw, Claude Code, and OpenCode, but still honest about the machine underneath. I care about that distinction because real automation lives or dies in the gap between the clean schema and the cursed desktop it has to operate.