# AetherDesk Upgrade Checklist - 2026-07-01

## Scope

This checklist turns the current AetherDesk/AetherBrowser systems into a staged implementation plan for the Aether Agent OS package. The rule for this pass is:

1. Pre-research local systems first.
2. Compare each item to external product/technical patterns.
3. Build only after the full update list is visible.

No delete, move, push, deploy, secret read, or external mutation is required for this checklist.

## Local Baseline

- AetherDesk standalone repo: `C:\Users\issda\AetherDesk`.
- Main server/API: `C:\Users\issda\AetherDesk\aetherdesk\server.js`.
- Main visual desktop: `C:\Users\issda\AetherDesk\aetherdesk\public\real-pc-shell.js` and `C:\Users\issda\AetherDesk\aetherdesk\public\real-pc-shell.css`.
- Agent/browser docs: `C:\Users\issda\AetherDesk\docs\PLAYWRIGHT_SIDEBAR_AGENT.md` and `C:\Users\issda\AetherDesk\docs\SEE_TOUCH_AUTOMATION_RESEARCH.md`.
- Connector/MCP docs: `C:\Users\issda\AetherDesk\docs\AETHERDESK_CONNECTOR.md` and `C:\Users\issda\AetherDesk\docs\AETHERDESK_MCP.md`.
- AI PC operating map: `C:\Users\issda\AetherDesk\docs\AETHERDESK_AI_PC_OPERATING_MAP.md`.
- Live local server: `http://127.0.0.1:5717`.
- Live read-only registries checked: `/api/health`, `/api/app-hooks`, `/api/steady-state`, `/api/browser/capabilities`, `/api/desktop-parity`, `/api/cli-systems`, `/api/ollama/status`, `/api/immersive-controls`.
- AetherBrowser repo: `C:\dev\aetherbrowser`.
- AetherBrowser headless contract: `C:\dev\aetherbrowser\tests\headless_contract.py`.
- Local system maps: `C:\dev\CODING_SYSTEMS.md` and `C:\dev\DEV_MAP.md`.

## Outside Research Anchors

- Playwright browser automation: https://playwright.dev/docs/intro, https://playwright.dev/docs/screenshots, https://playwright.dev/docs/locators, https://playwright.dev/docs/trace-viewer
- Chrome DevTools Protocol: https://chromedevtools.github.io/devtools-protocol/
- WebDriver BiDi: https://www.w3.org/TR/webdriver-bidi/
- xterm.js and node-pty terminal pattern: https://xtermjs.org/, https://github.com/microsoft/node-pty
- MCP tool protocol: https://modelcontextprotocol.io/docs
- Ollama local API: https://github.com/ollama/ollama/blob/main/docs/api.md
- OpenAI computer use: https://platform.openai.com/docs/guides/tools-computer-use
- Anthropic computer use: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool
- YouTube IFrame Player API: https://developers.google.com/youtube/iframe_api_reference
- YouTube Data API search: https://developers.google.com/youtube/v3/docs/search/list
- YouTube captions API: https://developers.google.com/youtube/v3/docs/captions
- OSWorld benchmark: https://os-world.github.io/
- WebArena benchmark: https://webarena.dev/
- WorkArena benchmark: https://github.com/ServiceNow/WorkArena
- MDN Web Speech API: https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API
- MDN WebXR Device API: https://developer.mozilla.org/en-US/docs/Web/API/WebXR_Device_API
- MDN Gamepad API: https://developer.mozilla.org/en-US/docs/Web/API/Gamepad_API
- MDN Vibration API: https://developer.mozilla.org/en-US/docs/Web/API/Vibration_API
- Microsoft Fluent design: https://fluent2.microsoft.design/
- WCAG 2.2 target size and focus guidance: https://www.w3.org/TR/WCAG22/

## Prioritized Update Checklist

## Implementation Pass 1 Status

- Completed now: AetherBrowser bridge routes, Agent Browser/Browser Workbench lane status, YouTube search API/fallback route, See/Touch driver registry, and Package Score endpoint/window.
- Live route evidence after restart: `/api/package-score` returned `84/100` and `strong alpha package`; `/api/see-touch/drivers` returned 6 drivers and 4 tiers; `/api/aetherbrowser/status` found `C:\dev\aetherbrowser` installed and backend offline at `http://127.0.0.1:8002`; `/api/youtube/search` returned `requires_api_key` plus a browser fallback URL without printing secrets.
- Visual evidence: Playwright-through-Edge smoke opened Terminal, Agent Browser, Steady State, Package Score, See/Touch Drivers, and YouTube Studio as desktop windows at `http://127.0.0.1:5717/?desktop=1&assetv=21`.
- Tests: `node --check aetherdesk/server.js`, `node --check aetherdesk/public/real-pc-shell.js`, and `npm run test:aetherdesk` passed.
- Still pending from this checklist: durable browser action receipts, provider-specific send/draft adapter policy, immersive event binding hardening, category/focus polish, broader OSWorld-style benchmark tasks, and Feature Package 2 export.

### 1. AetherBrowser Runtime Bridge

- Local evidence: AetherDesk lists AetherBrowser in `/api/cli-systems` and control surfaces, while AetherBrowser itself exposes `/headless/capabilities`, `/headless/command`, `/headless/page-context`, `/headless/browser-action`, `/headless/controller-state`, and `/headless/controller-event`.
- World evidence: Playwright/CDP/WebDriver BiDi patterns all separate visual UI from stable automation protocol.
- Update: Add AetherDesk routes that probe and summarize the running AetherBrowser backend without requiring it to be running.
- Acceptance: `/api/aetherbrowser/status` returns installed path, backend URL, online/offline state, and routes. `/api/aetherbrowser/capabilities` returns remote capabilities when online or a useful offline contract when not.

### 2. Agent Browser Source Selector

- Local evidence: Agent Browser already has Playwright session controls, screenshots, text, ARIA, guide, YouTube search, and transcripts.
- World evidence: browser-agent systems distinguish local page-control, cloud browser sessions, and visible-screen fallback.
- Update: Show whether the current browser lane is AetherDesk Playwright, AetherBrowser headless, or Computer Use fallback.
- Acceptance: Browser Workbench and Agent Browser both display lane status and recommend the right lane for repeatable browsing versus visible-screen work.

### 3. YouTube Search API Lane

- Local evidence: `/api/youtube/tools` and `/api/youtube/transcript` exist; Agent Browser currently searches YouTube by navigating to a results page and scanning links.
- World evidence: YouTube Data API search/captions are official API lanes; IFrame playback is constrained by site embed and browser policy.
- Update: Add `/api/youtube/search` with secret-safe env presence and a no-key fallback contract. It must never print API keys.
- Acceptance: Invalid/empty query is rejected. Without a key, the route returns `requires_api_key` plus the browser-search fallback URL. With a key, it returns normalized video results.

### 4. Browser Action Receipts

- Local evidence: AetherDesk has receipt lanes and browser action responses but browser actions should be more explicit as replayable evidence.
- World evidence: Playwright trace/screenshot patterns and OSWorld-style final-state scoring require durable observations.
- Update: Persist bounded browser action receipts containing session ID, action, URL, title, status, duration, screenshot flag, and no secrets.
- Acceptance: Successful and blocked browser actions appear in `/api/receipts` or a dedicated browser receipt lane.

### 5. Real Terminal Maturity

- Local evidence: Terminal routes exist: `/api/terminal/session`, `/api/terminal/input`, `/api/terminal/output`, `/api/terminal/stop`. Validation blocks destructive/chained shell text.
- World evidence: xterm.js plus node-pty is the common real web terminal pattern, but node-pty adds native dependency/host risk.
- Update: Keep bounded PowerShell first. Improve terminal UX with command history, status, session lifecycle clarity, and explicit "not raw admin shell" language in the app surface.
- Acceptance: Terminal feels like a persistent console, shows prompt/history/status, and tests still prove destructive text is blocked.

### 6. Provider Adapter Readiness

- Local evidence: `/api/providers`, `/api/app-hooks`, `/api/ollama/status`, `/api/ai/models`, and `/api/ai/chat` are live. Ollama is currently reachable.
- World evidence: MCP and multi-provider agent apps treat provider calls as adapters with explicit auth and send permissions.
- Update: Add provider adapter cards that separate `status`, `model list`, `draft prompt`, and `send`.
- Acceptance: OpenAI/Claude/Grok/Groq/Hugging Face remain secret-presence only until explicit credentials and send policy are configured. Ollama stays local-first and callable.

### 7. See/Touch Driver Registry

- Local evidence: `docs/SEE_TOUCH_AUTOMATION_RESEARCH.md` defines driver tiers but no first-class route/app yet.
- World evidence: OpenAI/Anthropic computer use and GUI-agent frameworks need observation, action, and receipt boundaries.
- Update: Add `/api/see-touch/drivers` and a small visual panel listing AetherDesk internal control, Computer Use, AetherBrowser, and future Cua/UIA lanes.
- Acceptance: The route is read-only and explains observe/touch-safe/touch-host/remote-write tiers.

### 8. Immersive Controller Binding Pass

- Local evidence: `/api/immersive-controls` exists and records events; AetherBrowser mobile includes Swift haptic handling.
- World evidence: Web Speech, WebXR, Gamepad, and Vibration APIs are browser/device capability surfaces, not guaranteed universal runtime features.
- Update: Persist hands-free profile and map gamepad/voice events to safe desktop actions only.
- Acceptance: No workspace mutation through immersive controls. Events are visible in action log with capability metadata.

### 9. Desktop Visual Information Architecture

- Local evidence: The visual polish pass improved desktop layout, but the shell is now dense because it has many first-class apps.
- World evidence: Fluent/WCAG patterns favor clear command surfaces, target size, focus visibility, and predictable taskbar/search behavior.
- Update: Add clearer grouping, app category filters, and stronger "active work mode" signal.
- Acceptance: Desktop, Start, taskbar, and mobile preview remain non-overflowing; target/focus states are visible.

### 10. Package Scoreboard

- Local evidence: `/api/desktop-parity` currently scores parity at 73/100 and `/api/cli-systems` exposes counts.
- World evidence: OSWorld/WebArena/WorkArena evaluate task completion, not vibes.
- Update: Add a package-score endpoint that combines desktop parity, route readiness, test/bench results, provider readiness, browser capability, and packaging readiness.
- Acceptance: Score is explainable and broken down by lane, not a single mystery number.

### 11. Benchmark Harness Expansion

- Local evidence: `scripts/aetherdesk-osworld-bench.js`, `scripts/aetherdesk-agent-bench.js`, and tests already exist.
- World evidence: execution-based benchmarks check final state and negative safety tasks.
- Update: Add tasks for Agent Browser search/transcript, terminal session lifecycle, AetherBrowser bridge status, and immersive event logging.
- Acceptance: Reference agent passes; deterministic instruction baseline remains separate from real Ollama agent score.

### 12. Feature Package 2

- Local evidence: `scripts/aetherdesk-feature-package.js` already exports feature lanes, routes, commands, docs, and evidence.
- World evidence: agent handoffs need machine-readable manifests plus human docs.
- Update: Include AetherBrowser status, YouTube search lane, see/touch drivers, package score, and latest benchmark paths.
- Acceptance: `npm run aetherdesk:feature-package` captures the new lanes without secrets, pushes, deploys, or external mutation.

## Build Order

1. Implement AetherBrowser Runtime Bridge.
2. Implement YouTube Search API Lane.
3. Add See/Touch Driver Registry.
4. Add Package Scoreboard.
5. Wire the new routes into App Hooks, Steady State, Browser Workbench/Agent Browser, CLI Systems, and Feature Package.
6. Add focused tests.
7. Run `node --check`, `npm run test:aetherdesk`, and the relevant benchmark/audit commands.
