How to author a browser-automation skill

Use this when: You are writing a skill that drives a web browser — to read, interact with, or extract data from a site that has no usable API. Prerequisites: playwright-cli installed; a skill directory under packs/<pack>/.apm/skills/<name>/ with a SKILL.md; familiarity with how to author a skill. Result: A skill whose browser sessions survive across invocations, whose auth is robust to SSO and device-certificate policies, and whose data layer decouples expensive browser scans from reasoning steps.

---

Persistent profile vs. storageState

Playwright’s storageState() serialises cookies and localStorage to a JSON file — it works for cookie-based auth (most public-facing SaaS). It does not work when:

• The site uses Microsoft Entra SSO with Conditional Access (device compliance is verified against a certificate stored in the OS keychain — it cannot be exported to JSON).
• Auth state lives in the browser’s encrypted credential store rather than plain cookies or localStorage.
• You need the signed system Chrome binary to access a macOS keychain entry (bundled Playwright Chromium is unsigned and cannot reach it).

In those cases, use a persistent profile directory instead.

Method	When to use
storageState	Cookie/token auth; CI-friendly; fully headless
Persistent profile	Entra PRT, Conditional Access, encrypted credential store, keychain-bound auth

Persistent-profile auth handoff

The first-run pattern: open the system Chrome browser headed, let the user log in manually, then close. The profile directory (including PRT, cookies, device-compliance state) persists to disk. Every subsequent run opens the same profile headless.

Why system Chrome: Only /Applications/Google Chrome.app/Contents/MacOS/Google Chrome is signed with the identity that macOS Keychain trusts for credential-store access. Playwright’s bundled Chromium is unsigned.

import { chromium } from 'playwright';
import * as path from 'path';
import * as os from 'os';

const PROFILE_DIR = path.join(os.homedir(), '.agent-commander', '<pack>', 'chrome-profile');
const CHROME_PATHS = [
  '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
  path.join(os.homedir(), 'Applications/Google Chrome.app/Contents/MacOS/Google Chrome'),
];

export function findSystemChrome(): string {
  for (const p of CHROME_PATHS) {
    if (require('fs').existsSync(p)) return p;
  }
  throw new Error('Google Chrome not found. Install it from https://www.google.com/chrome/');
}

// First-run: open headed for the user to sign in
export async function authenticateInteractive(): Promise<void> {
  const context = await chromium.launchPersistentContext(PROFILE_DIR, {
    executablePath: findSystemChrome(),
    headless: false,
    args: ['--no-first-run', '--no-default-browser-check'],
  });
  const page = await context.newPage();
  await page.goto('https://example.yourorg.com/');
  // Prompt: wait for the user to complete login
  const readline = require('readline').createInterface({ input: process.stdin, output: process.stdout });
  await new Promise<void>(resolve => readline.question(
    'Log in, wait until the main screen is fully loaded, then press Enter…\n', () => { readline.close(); resolve(); }
  ));
  await context.close();
}

// Subsequent runs: reuse profile
export async function playwrightRun<T>(
  fn: (page: import('playwright').Page) => Promise<T>,
  opts: { headed?: boolean } = {}
): Promise<T> {
  const context = await chromium.launchPersistentContext(PROFILE_DIR, {
    executablePath: findSystemChrome(),
    headless: !opts.headed,
    args: ['--no-first-run'],
  });
  const page = await context.newPage();
  try { return await fn(page); }
  finally { await context.close(); }
}

Profile directory convention: always ~/.agent-commander/<pack>/chrome-profile/. Never inside the repo.

Session check

Before performing work, verify the session is still valid. On a page that requires auth, a failed session redirects to a login URL:

export async function checkSession(): Promise<'ok' | 'needs-login'> {
  return playwrightRun(async page => {
    await page.goto('https://example.yourorg.com/inbox', { waitUntil: 'domcontentloaded' });
    try {
      await page.waitForSelector('[data-app-section="main"]', { timeout: 8000 });
      return 'ok';
    } catch {
      const url = page.url();
      return url.includes('login') || url.includes('microsoftonline') ? 'needs-login' : 'ok';
    }
  });
}

Include a session check step in the skill body. On needs-login, run the office365-setup skill (or your pack’s equivalent setup skill) rather than silently failing.

Bearer token interception

Many internal portals issue a Bearer JWT on page load that you can capture to call their REST API directly — without requiring an OAuth app registration.

Critical: register the handler before page.goto(). The token arrives in the first authenticated request on page load, so a handler installed after goto will miss it.

export function captureToken(page: import('playwright').Page, hostname: string): Promise<string> {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => reject(new Error(`Token capture timeout (${hostname})`)), 20000);
    const handler = (req: import('playwright').Request): void => {
      const auth = req.headers()['authorization'] ?? '';
      if (auth.startsWith('Bearer ') && req.url().includes(hostname)) {
        clearTimeout(timer);
        page.off('requestfinished', handler);
        resolve(auth.slice(7));
      }
    };
    page.on('requestfinished', handler);
  });
}

// Usage:
const tokenPromise = captureToken(page, 'example.yourorg.com');
await page.goto('https://example.yourorg.com/inbox');
const token = await tokenPromise;

Vendor caveat — Teams: Teams uses a skypetoken_asm cookie on .asm.skype.com rather than a standard Authorization: Bearer header. The cookie is set asynchronously after page load; use polling with a deadline:

async function resolveTeamsSkypeToken(page: import('playwright').Page, timeoutMs = 10000): Promise<string> {
  const deadline = Date.now() + timeoutMs;
  while (Date.now() < deadline) {
    // do NOT filter by URL — the cookie domain is .asm.skype.com, not the Teams host
    const cookies = await page.context().cookies();
    const token = cookies.find(c => c.name === 'skypetoken_asm')?.value;
    if (token) return token;
    await page.waitForTimeout(500);
  }
  throw new Error('Teams skypetoken_asm not found within timeout');
}

Two-mode Playwright usage

The playwright-cli tool has two interaction modes. Pick based on what you need to do.

CLI commands — for single actions where one command produces one result:

playwright-cli -s=<session> open "https://…"
playwright-cli -s=<session> snapshot           # returns ARIA accessibility tree with element refs
playwright-cli -s=<session> click "e42"        # ref from snapshot — valid until next page change
playwright-cli -s=<session> fill "getByLabel('Search')" "query"   # locator — stable across page changes
playwright-cli -s=<session> close

Use refs (e42) when you want to act on exactly the element the snapshot showed. Use locators (getByLabel(…), getByRole(…)) when you want a selector that survives navigation and re-renders.

run-code — for loops, conditional logic, or structured data extraction:

playwright-cli --raw -s=<session> run-code "async page => {
  const items = await page.locator('[role=\"option\"]').evaluateAll(opts =>
    opts.slice(0, 25).map(o => ({ id: o.id, label: o.getAttribute('aria-label') || '' }))
  );
  return JSON.stringify(items);
}"

--raw returns only the return value of the async function — no metadata wrapper. Use it whenever the skill needs to parse the output as JSON.

Rule of thumb: if you’d write a for loop to drive the browser, use run-code. If you’d write a single line, use a CLI command.

ui-patterns.md — live-probe maintenance

Browser UIs change. Selectors that work today may not work after an app update. Keep a references/ui-patterns.md file in the skill directory and treat it as a living document:

## <App name> UI selectors

Last confirmed: YYYY-MM-DD against <version/build if known>

| Selector | Status | Notes |
|----------|--------|-------|
| `[role="listbox"]` | ✓ | Inbox message list |
| `[aria-label="Reply"]` | ✓ | Reply button in thread view |
| `[data-app-section="compose"]` | ✗ | Not present in current build |
| `[data-testid="send-btn"]` | [?] | Seen in dev builds; unconfirmed in prod |

## Selector update log

| Date | Change |
|------|--------|
| YYYY-MM-DD | Initial live probe (47/62 confirmed). Core inbox/toolbar confirmed. |
| YYYY-MM-DD | Post-update probe (55/62). Compose selectors changed — updated above. |

Status annotations:

• ✓ — confirmed working in a recent live probe
• ✗ — not present in the current build (may be org-specific or version-gated)
• [?] — seen in some environments but not confirmed in the canonical environment

Run a live probe (a quick snapshot pass through the main surfaces) whenever the app has a major update. Document the ceiling: 55/62 — practical ceiling; remaining 7 failures are org-build-absent.

Probe files as data layer

Browser scans are expensive — they drive a browser, navigate pages, and extract data. Decouple the scan from the reasoning step by writing scan results to timestamped JSON files:

~/.agent-commander/<pack>/probes/
  <timestamp>-senders.json
  <timestamp>-threads.json
  <timestamp>-queue.json

The skill checks probe freshness before deciding whether to re-scan:

LATEST=$(ls -t ~/.agent-commander/<pack>/probes/*-threads.json 2>/dev/null | head -1)
if [ -z "$LATEST" ]; then echo "no-probe"; exit 0; fi
AGE=$(( ($(date +%s) - $(stat -f %m "$LATEST")) / 3600 ))
[ "$AGE" -ge 4 ] && echo "stale" || echo "fresh"

Rules:

• One scan writes one file with a timestamp prefix — never overwrite the previous file.
• The skill reads the most recent file that meets the freshness threshold.
• The agent reasons from the structured JSON, never from raw HTML.
• Probe files live outside the repo (~/.agent-commander/…), never inside it.

Config and data paths

All user-specific data lives outside the repo:

~/.agent-commander/<pack>/
  config.toml         # user config (API base URLs, display prefs)
  chrome-profile/     # persistent browser profile
  probes/             # timestamped scan outputs

Scripts resolve these paths from os.homedir() or os.homedir() equivalents — never from __dirname or a repo-relative path. This keeps the repo clean and the data portable.