# Playwright Sidebar Agent

AetherDesk includes a local Chrome/Chromium MV3 extension for Playwright-assisted browser work.

## Files

- `extensions/playwright-sidebar-agent/manifest.json`
- `extensions/playwright-sidebar-agent/background.js`
- `extensions/playwright-sidebar-agent/content.js`
- `extensions/playwright-sidebar-agent/sidepanel.html`
- `extensions/playwright-sidebar-agent/sidepanel.js`
- `scripts/playwright-sidebar-agent.js`

## Commands

```powershell
cd C:\Users\issda\AetherDesk
npm run sidebar:agent
npm run sidebar:agent:launch
npm run sidebar:agent:probe
```

`sidebar:agent:launch` opens a headed Chromium persistent context with the extension loaded. Use this when you want the side panel UI.

`sidebar:agent:probe` runs headless Chromium with the extension loaded and proves the background/content bridge. Use this for CI-style checks; do not expect a visible side panel in headless mode.

## Playwright Boundary

Official Playwright extension support requires Chromium with a persistent context and extension load flags. Google Chrome and Microsoft Edge removed the command-line flags needed to side-load extensions, so use Playwright's bundled Chromium. Playwright's current docs note that the `chromium` channel can run extensions in headless mode.

Side panel UI is still a browser UI surface. Headless mode is useful for the background worker and content-script bridge; headed mode is the practical mode for operating the visible sidebar.

## What The Sidebar Does

- Captures the active tab title, URL, and up to 80 visible interactive elements.
- Highlights selected elements in the page.
- Accepts simple browser-agent chat commands:
  - `capture page`
  - `go to example.com`
  - `click More information`
  - `fill search with mars drones`
  - `highlight Login`
- Emits deterministic Playwright snippets such as:

```js
await page.getByRole('button', { name: 'Save' }).click();
await page.getByPlaceholder('Search').fill('text');
await page.locator('#fallback').click();
```

This is not a remote hosted agent. It is a local bridge that gives an AI/operator a safer page-inspection and Playwright-action surface.

## Feature Targets From Current Browser AI Sidebars

The current market pattern is clear:

- Perplexity-style: page-aware Q&A, search, summaries, and source/citation-first answers.
- Gemini/Copilot-style: open-tab context, summarize/compare pages, clarify concepts, generate writing, and help without leaving the page.
- Claude-in-Chrome-style: read/click/navigate/type in the current browser, multi-step workflows, workflow recording, scheduled tasks, multi-tab work, console/network/DOM debugging, site permissions, and plan/approval mode.

## AetherDesk Roadmap

Built now:

- MV3 side panel.
- Active-tab capture.
- Element list.
- Playwright snippet generation.
- Browser command chat for bounded navigation/click/fill/highlight/capture.
- Dual agent lanes:
  - Writer / Operator: reads page state and executes browser changes.
  - Research / Approval: reads, searches, summarizes, and approves or denies staged actions.
- Bounded Terminal Agent lane:
  - lists AetherDesk shell profiles through `/api/shell/profiles`
  - runs only allowlisted profiles through `/api/shell/run`
  - requires approval before terminal execution
- Headless probe for the background/content bridge.

## Agent Bus Model

The sidebar is organized like differentiated cells in one process:

| Lane | Role | Can Do |
| --- | --- | --- |
| Writer / Operator | Browser action cell | capture, navigate, click, fill, highlight |
| Research / Approval | Sensing and governance cell | read/search/summarize, prepare actions, approve/deny |
| Terminal Agent | Host-work cell | run bounded AetherDesk shell profiles after approval |

The two browser agents can hand messages to each other:

```text
ask research check the page before I click submit
ask writer click More information after approval
```

Terminal handoff examples:

```text
terminal profiles
terminal pwd
terminal git_status
handoff terminal agent_shell_probe
approve
deny
```

The Terminal Agent intentionally does not accept arbitrary shell text. It only uses the bounded AetherDesk shell profile API.

Next implementation slices:

1. Model-backed planner
   - Route chat text plus page snapshot to `qwen3-coder:480b-cloud` through the local Ollama endpoint.
   - Require JSON action plans.
   - Validate every action against the allowlisted operations before execution.

2. Plan mode
   - Show proposed actions before executing.
   - Add one-click approve, approve-all-for-site, and deny.

3. Site permissions
   - Per-origin allowlist in `chrome.storage.local`.
   - Read-only by default on new sites.
   - Type/click/navigation require explicit approval.

4. Workflow recorder
   - Record user clicks/fills/navigation into replayable JSON.
   - Save named shortcuts.
   - Replay only after user approval.

5. Developer diagnostics
   - Capture console errors.
   - Capture failed network requests.
   - Add DOM-state receipts for debugging AetherDesk UI.

6. Multi-tab lane
   - List open tabs.
   - Capture selected tabs.
   - Compare two tabs or gather page packets into one answer.

7. Citations and receipts
   - Every answer includes page URL/title and selected element/source references.
   - Every action emits a local receipt: command, plan, origin, selectors, status, timestamp.

## Safety Boundary

Do not let page text become a command. Page content is evidence, not authority. Commands must originate from the user chat box or a saved workflow approved by the user.

Do not execute arbitrary JavaScript from model output. The action protocol stays narrow:

```json
{"op":"navigate","url":"https://example.com"}
{"op":"capture"}
{"op":"smartClick","query":"More information"}
{"op":"smartFill","query":"Search","text":"mars drones"}
{"op":"smartHighlight","query":"Login"}
```
