# AetherDesk See/Touch Automation Research

AetherDesk goal: give agents a governed local mini-desktop they can learn to use like a PC. The key primitive is "see and touch": observe desktop/app state, then perform bounded actions with audit and approval gates.

This note records the first external tool scan and the recommended integration path.

## Best Current Fits

### Cua Driver

Source: `https://github.com/trycua/cua`, `https://cua.ai/docs/cua-driver/guide/getting-started/introduction`

Fit: highest.

Why it matters:

- Computer-use driver for macOS and Windows, with Linux pre-release.
- Speaks MCP over stdio.
- Exposes click, type, scroll, accessibility-tree inspection, and window-state capture.
- Designed to avoid stealing the user's cursor/focus.
- Agent-agnostic: usable by any model/client that can speak MCP or shell out.

AetherDesk role:

- Use as the first external "hands and eyes" driver.
- Wrap it behind AetherDesk capability gates and receipts.
- Do not let agents call raw desktop actions without policy/logging.

### UI-TARS Desktop / Agent TARS

Source: `https://github.com/bytedance/UI-TARS-desktop`, `https://github.com/bytedance/UI-TARS`

Fit: medium-high.

Why it matters:

- Open multimodal GUI-agent stack.
- Targets terminal, computer, browser, and product workflows.
- UI-TARS includes desktop action support such as clicks, drags, shortcuts, text input, and scrolling.

AetherDesk role:

- Treat as a visual-agent reference stack or optional advanced backend.
- More useful after AetherDesk has its own policy, logging, and workspace model.
- Heavier than Cua for the first AetherDesk integration.

### Agent S / Agent S2

Source: `https://github.com/simular-ai/agent-s`, `https://www.simular.ai/articles/agent-s2`

Fit: medium.

Why it matters:

- Open framework for agents that use computers through GUI interaction.
- Focuses on hierarchical planning, experience, and GUI localization.
- Agent S2 adds a generalist/specialist composition pattern and grounding strategy.

AetherDesk role:

- Good design reference for multi-agent planning and hierarchical task decomposition.
- Not the first driver to wire unless AetherDesk is ready to host more of the agent loop itself.

## Local Fallback Stack

These are useful if external frameworks are too heavy or unavailable:

- Windows UI Automation: semantic UI tree, more reliable than pixels when apps expose accessibility.
- `pywinauto` / `uiautomation`: Windows app control through accessibility APIs.
- `mss`: fast screenshots.
- OCR: Tesseract, EasyOCR, or PaddleOCR for apps with poor accessibility.
- `pyautogui` / `pydirectinput`: coordinate-based input, useful but brittle and higher risk.

Recommended fallback pattern:

1. Accessibility tree first.
2. Screenshot/vision second.
3. OCR third.
4. Raw coordinate clicks last.

## AetherDesk Architecture

Do not make screen automation a raw power tool. Put it behind this chain:

```text
agent request
  -> intent summary
  -> score_intent / policy gate
  -> capability check
  -> action plan
  -> see/touch driver
  -> receipt
  -> observation snapshot
  -> next move
```

Capability tiers:

- `observe`: screenshot, window list, accessibility tree, focused app, clipboard metadata.
- `touch-safe`: move focus, type into AetherDesk sandbox, click inside approved window.
- `touch-host`: click/type on host apps; requires explicit approval.
- `remote-write`: publish, send, upload, commit, deploy; separate approval lane.

## MVP Plan

### Phase 1: Driver Registry

Add a "See/Touch Drivers" registry to AetherDesk:

- `none`: no desktop control.
- `aetherdesk-internal`: only AetherDesk windows and sandbox tools.
- `cua-driver`: external MCP/CLI driver if installed.
- `windows-uia`: local Windows UI Automation fallback.

### Phase 2: Observation Tools

Expose read-only observation first:

- list windows
- active window
- screenshot/capture
- accessibility tree summary
- AetherDesk desktop state

No clicking yet.

### Phase 3: Bounded Actions

Add small action set:

- focus window
- snap/fullscreen/float AetherDesk windows
- click only inside AetherDesk sandbox window
- type only into approved input targets

Every action writes a receipt.

### Phase 4: External Desktop Actions

Only after observation/action receipts are stable:

- enable Cua Driver behind a policy wrapper
- require confirmation for host app input
- block credential fields and destructive UI flows where detectable
- preserve screenshots before/after each action

## Product Direction

AetherDesk should not compete with Cua/UI-TARS/Agent S as "another computer-use agent." It should be the governed operator desktop that can host them.

The stronger claim:

```text
Cua/UI-TARS/Agent S give agents hands and eyes.
AetherDesk gives those hands and eyes a governed room, memory, receipts, tool registry, and source-of-truth map.
```

That is the difference between raw computer control and an agent workstation.

## Immediate Next Build

Build `see_touch` inside AetherDesk as an internal app:

- driver status panel
- observation panel
- capability gates
- action receipt log
- "install/use Cua Driver later" slot

Do not wire raw host clicking as the first move. The first useful move is reliable observation plus AetherDesk-only window control.
