Skip to content

WORKFLOW_RECORDING.md

Latest commit

 

History

History
542 lines (453 loc) · 25.7 KB

WORKFLOW_RECORDING.md

File metadata and controls

542 lines (453 loc) · 25.7 KB
Raw

Workflow Recording → Skill

Status: design preview, disabled at runtime. The wire contract is drafted, but capture and interpretation are incomplete end to end.

Record a workflow the user performs anywhere on their machine, hand it to the agent, and have the agent generalize it into a reusable skill that drives the user's computer to repeat it.

Motivating example (one instance, not the ceiling): a user fills a browser form with one row of data by hand while recording; the agent turns that into a skill that fills the same form from every row of a CSV. The same machinery records a desktop app, a terminal sequence, or a flow that spans a browser and a native app.

Current implementation boundary: macOS capture code exists behind the CaptureSource seam, but the daemon intentionally advertises no recording channels on any OS. Setting enable_recording=true fails startup with EX_CONFIG. Windows/Linux capture, browser/desktop enrichment, and the full interpretation path remain incomplete. Everything below is a design contract, not an enabled product capability.

0. What changed in v0.2 (and why)

v0.1 was browser-DOM-first, which forced a browser-only MVP and shipped raw form values to the cloud for generalization. A review panel flagged both as disqualifying. v0.2 re-founds the design:

  1. The floor is a computer-use trajectory, not a DOM trace. Replay is done by our vision-capable computer-use agent re-planning against the live screen — not by deterministic selector playback. So the recording doesn't need perfect semantic selectors; it needs enough for the agent to understand and redo the task. That makes the floor universal (works in any app on any OS where computer-use works), with DOM/a11y as enrichment, not requirements.
  2. Pixels stay local by default. Raw screen frames leave the machine only for the explicit screenshots_to_cloud route. Every route may send the hygiene-redacted structured trajectory through the authenticated platform to the user's browser for review (see §3 and §6).
  3. Consent is shim-enforced, secret-detection is hygiene not a control, replay has explicit wait/assert/dedupe guards, and parameter inference must be confirmed (multi-row or asked), not guessed. (Panel must-fixes, §7–§9.)

Capability target is unchanged and full: general desktop + browser demonstration capture (the old "Tier 2") and the live co-pilot loop with on-device interpretation (the old "Tier 3"). Those are the baseline, not a phased cut. They are now framed as two interaction modes over one trajectory floor, not two releases.

0.1 v1 scope decisions (build contract)

The north star the whole build serves: a non-technical person shows the agent a repetitive task once, on their own machine, then walks away and trusts it to repeat the task correctly — the first skill, the first time, with their data staying local unless they choose otherwise. Acceptance test: a real user records the CSV-into-a-form task once and lets the resulting skill run unattended on a fresh batch; pass = correct + they didn't babysit it.

To unblock the build, the §10 open questions take these v1 defaults (revisit post-v1):

  • Capture (v1): real-machine, general — screenshot+action floor (always) + browser DOM enrichment (via the claude-in-chrome channel)
    • desktop a11y enrichment where the tree exists. Not browser-only.
  • Interpretation default: extract_then_cloud; probe-then-ask for screenshots_to_cloud (§3.1, §9.1). local_vlm when present.
  • Parameter inference: never auto-save a single-row guess — require co-pilot confirmation OR ≥2 demonstration rows (§8).
  • Skill portability: recorded skills are machine-bound (shim-local) in v1; graduating browser skills into the shared library is post-v1.
  • Compliance: content-free audit entry only in v1 (§9); a reviewable encrypted content manifest is post-v1.
  • Coverage floor: kind: none steps degrade to vision-at-replay; if

    50% of a recording's steps are kind: none and no OCR is available, warn at record time that this app records poorly.

The wire contract (§1 schema, §6 ops + error codes) is spec-locked — both repos build against it as-is. The build seam for the genuinely OS-specific part (raw input-event observation) is a CaptureSource interface with a mock producer for tests. The macOS hook is wired; additional per-OS hooks land behind the same interface without changing the contract.

1. The floor: a computer-use trajectory

Every recording is an ordered list of trajectory steps. Each step is, at minimum, what the user did and what the screen looked like when they did it — the same shape computer-use models are trained on. Richer channels (browser DOM, desktop accessibility) attach as optional enrichment on the same step; they never replace the floor.

{
  "recording_id": "rec_<uuid>",
  "version": "1.0",
  "created_at": 1712345678.0,
  "machine_id": "...",
  "interpretation_route": "extract_then_cloud",  // see §3
  "steps": [ /* TrajectoryStep[] */ ],
  "redaction_applied": true
}

TrajectoryStep

{
  "seq": 12,
  "ts": 1712345678.42,
  "actor": "human",                 // "human" | "agent"

  // --- the universal floor (always present) ---
  "action": "fill",                 // semantic verb (§1.1)
  "screenshot_ref": "stub_<uuid>",  // pre-action frame, stub id — NEVER inline bytes
  "cursor": [840, 314],             // where the action landed, display-global px
  "active_app": "Google Chrome",
  "active_window": "New customer — Acme",

  // --- enrichment: present only when the channel resolved it ---
  "enrichment": {
    "kind": "dom",                  // dom | ax | none
    "selectors": [                  // browser DOM only; most-stable-first (§7)
      {"strategy": "id",    "value": "#first_name"},
      {"strategy": "name",  "value": "first_name"},
      {"strategy": "label", "value": "First Name"},
      {"strategy": "role+text", "value": "textbox[name='First Name']"}
    ],
    "ax_path": null,                // desktop a11y only; element path within the AX tree
    "role": "textbox",
    "label": "First Name",
    "url": "https://app.example.com/new"
  },

  "value": {
    "raw": "John",                  // demo value — handled per interpretation route (§3)
    "type": "text",                 // text|email|number|date|secret|file|enum
    "is_parameter": null            // inferred + CONFIRMED during generalization (§8)
  },

  "narration": null,                // co-pilot mode: "filling the shipping address"
  "outcome": "ok",                  // ok | error | unknown
  "redacted": false
}

The key change from v0.1: screenshot_ref + action + cursor + active_app/window are the primary key and are always present. The enrichment block is additive — DOM selectors when the browser channel saw them, an AX path when the desktop channel resolved them, kind: none when neither did (pure visual step). Replay degrades gracefully: use the DOM fast-path if present, the AX path if present, else visual grounding on the screenshot.

1.1 Semantic verbs

navigate, fill, select, click, submit, upload, launch_app, focus_window, copy, paste, run (shell), file_op, plus two replay-control verbs the recorder synthesizes or the user adds (§9): wait (for a condition — a spinner clears, an element appears) and assert (a read-back check — "the confirmation banner showed").

Not captured: raw mouse movement paths, per-keystroke content (only the final committed value), scroll deltas. The recording is a sequence of intentful semantic actions over screenshots, not a raw input tape.

2. Two interaction modes (the capability baseline)

One trajectory floor, two ways the user can drive it. Both ship.

Demonstration mode (the old Tier 2). The user does the whole task by hand; the shim captures the trajectory; the agent generalizes afterward. Lowest friction, works when the task is unambiguous. Pairs with a multi-row demonstration (§8) so parameter-vs-constant is decidable.

Co-pilot mode (the old Tier 3, preferred default). The agent watches the trajectory live, narrates each step, asks clarifying questions in real time, and confirms with a dry-run before saving. Attacks the dominant failure (confident misgeneralization) directly. This is the default for anything non-trivial.

The user picks the mode per recording; the wire + schema are identical.

3. On-device interpretation (the privacy floor)

A general recording may contain screenshots of the user's screen. Raw frames stay on the machine unless the user explicitly approves screenshots_to_cloud. After STOP, however, the authenticated platform fetches the hygiene-redacted structured trajectory for browser review on every route. That trajectory can contain actions, coordinates, app/window metadata, and non-secret field values; secret detection is best-effort, not a guarantee. Interpretation happens under one of three routes, chosen per recording. The route is recorded in recording.interpretation_route.

Route What leaves the machine Capability Privacy Needs
extract_then_cloud (default) hygiene-redacted structured trajectory (OCR/a11y/DOM/window metadata and potentially non-secret values); no pixels cloud-grade reasoning over extracted structure high — pixels stay local a local extractor (OCR is cheap; a11y/DOM already captured)
local_vlm (upgrade) hygiene-redacted structured trajectory for authenticated browser review, then the abstract skill; no pixels capped by the user's local VLM highest pixel privacy; review data still crosses the shim boundary a capable local vision model (llava-class+)
screenshots_to_cloud (fallback) structured trajectory plus screenshots, behind explicit per-recording consent maximal lowest — frames leave the machine nothing; gated on a hard consent the shim enforces

Default is extract_then_cloud — it generalizes well and keeps pixels on the machine, while still sending extracted structure for review and reasoning. This is the privacy-mode pattern (local model / pipeline extracts structure, strong cloud model reasons over it) applied to screen recordings. The browser-DOM case is the happy path here: DOM is already text, so extraction is free and lossless.

local_vlm is the pixel-local upgrade for users with the hardware; the structured browser-review trajectory still crosses the shim boundary. screenshots_to_cloud exists so users with no local extractor at all aren't locked out — but it requires the shim-enforced consent gate (§9), not a platform flag.

3.1 Route selection: probe local first, only then ask

The route is detected, not assumed. On START_RECORDING the shim probes, in order:

  1. a11y / DOM already captured for the demonstrated steps → free structured text → extract_then_cloud works.
  2. OCR available on-device (cheap, bundleable) → extract_then_cloud works even for kind: none (canvas/Electron) steps.
  3. Capable local vision model present → offer local_vlm (zero cloud) as the default for this machine.
  4. None of the above → the only way to build the skill is screenshots_to_cloud. This is the only route that prompts the user (see §9.1). The other three need no special prompt — they either keep pixels local, or (for extract_then_cloud) send text/structure, which is the same trust the user already extends by letting the platform act on their machine.

Matching the prompt to the actual incremental risk is deliberate. Prompting for routes that don't cross a new line trains users to click through, which buries the one prompt that matters. Silence when local handles it; ask only when raw screen images would leave the machine.

4. Capture adapters (universal floor + enrichment)

All adapters emit TrajectorySteps into the same recording.

  • Screenshot+action floor (target universal; currently macOS). The shim's existing computer-use backends, run in reverse: instead of injecting input, the shim observes the user's action and snapshots the pre-action frame + cursor + active app/window. It is currently wired through Quartz on macOS; Windows/Linux producers have not landed and are not advertised. This is the floor every other adapter enriches.
  • Browser DOM enrichment (lights up first). A companion extension (extends the existing claude-in-chrome channel) attaches selectors + label + url to steps that happen in a browser tab, in the user's real logged-in session. Highest fidelity, and the one case where extract_then_cloud is lossless (DOM is text). Localhost-WebSocket transport to the shim (no per-OS native-host install for v1).
  • Desktop a11y enrichment. macOS AXObserver / Windows UI Automation / Linux AT-SPI attach an AX path + role + label where the tree exists. Where it doesn't (Electron, canvas/WebGL, Wayland, games), the step keeps enrichment.kind: none and replay falls back to visual grounding — not a failure, just lower fidelity. This is why the floor matters: desktop coverage gaps degrade, they don't block.

Rollout order is by tractability, not scope: the floor + browser DOM enrichment first (cheapest, most reliable, lossless privacy), desktop a11y enrichment next. Both are in the design from day one; they light up in sequence. The product is general from v1.

5. End-to-end: the motivating example (+ a desktop one)

Browser form-fill:

  1. User picks co-pilot mode, clicks Record → shim-rendered indicator shows.
  2. User fills the form once. Floor captures fill/click steps with screenshots; browser enrichment attaches #first_name etc.
  3. Agent narrates live, asks "where does the data come from?" → "this CSV", and "stop on validation error?" → "skip the row."
  4. User stops; the recording is interpreted via extract_then_cloud (DOM is already text — lossless). Agent confirms parameters (first_name/last_name/email) against the CSV columns.
  5. Multi-row dry-run on rows 1–2 with the user watching → confirmed.
  6. "Fill all 200 rows" → agent drives the browser via the DOM fast-path, with read-back asserts + per-row dedupe (§9).

Desktop app (no DOM):

  1. User records renaming + tagging files in a native asset manager.
  2. Floor captures screenshots + clicks; a11y enrichment resolves some buttons, leaves canvas regions as kind: none.
  3. extract_then_cloud: on-device OCR + a11y dump → text per step → cloud generalizes "for each file, open it, set tag = {tag}, rename to {pattern}."
  4. Replay: agent re-plans visually against the live UI, using a11y fast-paths where present, vision elsewhere.

Same machinery, fidelity scales with what the app exposes.

6. Wire ops

// platform → shim  (request/response — counts against in-flight)
{"type": "REQUEST_RECORDING_CONSENT", "payload": {
   "mode": "copilot",
   "interpretation_route": "extract_then_cloud",
   "channels": ["floor", "browser", "desktop_ax"]
}}
{"type": "RECORDING_CONSENT_RESULT", "payload": {
   "approved": true,
   "mode": "copilot",
   "interpretation_route": "extract_then_cloud",
   "consent_token": "<single-use shim-issued token>",
   "expires_at": 1760000000.0
}}

{"type": "START_RECORDING", "payload": {
   "mode": "copilot",                      // "demonstration" | "copilot"
   "interpretation_route": "extract_then_cloud",
   "channels": ["floor", "browser", "desktop_ax"],
   "consent_token": "<shim-issued, see §9>"  // REQUIRED — platform can't self-assert
}}
{"type": "RECORDING_STARTED", "payload": {"recording_id": "rec_<uuid>"}}

{"type": "STOP_RECORDING",  "payload": {"recording_id": "rec_<uuid>"}}
{"type": "RECORDING_SUMMARY","payload": {"recording_id": "...", "step_count": 14,
   "enrichment_coverage": {"dom": 11, "ax": 0, "none": 3}, "duration_seconds": 47.2}}

{"type": "RECORDING_FETCH", "payload": {"recording_id": "..."}}
{"type": "RECORDING_DATA",  "payload": { /* full WorkflowRecording, post-redaction */ }}

{"type": "APPLY_RECORDING_REVIEW", "payload": {
   "recording_id": "rec_<uuid>",
   "removed_step_seqs": [2, 5],
   "redacted_step_seqs": [3]
}}
{"type": "RECORDING_REVIEW_APPLIED", "payload": {
   "recording_id": "rec_<uuid>", "step_count": 11
}}

RECORDING_STEP is an unsolicited, non-acked, out-of-band stream (co-pilot mode only) — modeled like STATUS, exempt from max_concurrent / in-flight accounting and idempotency/retry (the panel's load-bearing protocol fix). Demonstration mode does not stream during capture. After STOP, the authenticated platform calls RECORDING_FETCH and returns the shim's hygiene-redacted structured trajectory to the user's browser for review. The browser then sends APPLY_RECORDING_REVIEW; the shim applies removals and redactions to its authoritative retained copy. Skill generation remains blocked until that apply succeeds. This is not an on-device-only review: the review gate controls generation, while raw pixels remain local unless the separate screenshots_to_cloud consent was granted.

Co-pilot interpretation reuses LOCAL_LLM_COMPLETION for the on-device model; INTERPRET_TURN / CLARIFY_PROMPT / DRY_RUN_REPLAY as in v0.1 §3.5.

New error codes: RECORDING_NOT_FOUND, RECORDING_CHANNEL_UNAVAILABLE, RECORDING_ALREADY_ACTIVE, CONSENT_REQUIRED (START without a valid shim-issued consent token), INTERPRETATION_UNAVAILABLE (route needs a local model the machine doesn't have — see §10).

7. Replay fidelity + selector strategy

Replay uses the richest available enrichment per step, falling through:

  1. DOM fast-path — selectors[] tried most-stable-first (id > name > label > role+text > xpath); if exactly one node matches, use it; if zero or >1, fall through (no guessing).
  2. AX fast-path — resolve the ax_path against the live tree; if it moved, fall through.
  3. Visual grounding — the computer-use agent locates the target on the live screenshot the way it does for any computer-use task.

Because the floor is always a screenshot+action and replay is a vision agent, a broken selector degrades to visual replay rather than failing. That's the whole reason the trajectory floor buys robustness the DOM-only design couldn't.

8. Parameter inference — confirmed, never guessed

A single demonstration can't distinguish parameter from constant. v0.2 forbids shipping a skill with unconfirmed parameters. Confirmation comes from, in preference order:

  1. Co-pilot ask — agent asks "does First Name change each run?"
  2. ≥2 demonstration rows — anything that changed is a parameter. Required for demonstration mode (no live loop to ask in).
  3. Data-source join — match fields to named CSV/sheet columns by label similarity, then confirm.

Field semantics ("a field labeled Email is probably a parameter") is a prior that seeds the question, not a basis for auto-save. A skill whose parameters were never confirmed does not save — it blocks on a clarifying question or a second row.

9. Consent, safety, security (panel must-fixes)

  • Shim-enforced consent (not platform-asserted). REQUEST_RECORDING_CONSENT renders a local shim confirmation and returns a short-lived, single-use token only after approval. The token is bound to the mode and effective interpretation route. START_RECORDING without that exact token gets CONSENT_REQUIRED; an extract-only approval cannot be reused for screenshots_to_cloud.
  • Scoped capture. Browser enrichment is origin/active-form allow-listed — it attaches to steps on the demonstrated origin, not every tab. The current macOS floor uses a session-wide, listen-only Quartz event tap while recording is active, observing mouse-down and key-down events across the desktop. It emits coarse actions and coordinates, not raw typed key content, and tears the tap down on STOP.
  • Secret-detection is best-effort hygiene, explicitly NOT a privacy control. Password fields / secret-shaped values are dropped, but the doc no longer claims "secrets never captured" as a guarantee — OTPs, account numbers, SSNs-in-generic-fields will slip a deny-list. The real control is interpretation_route (§3): it controls pixel handling. The hygiene-redacted structured trajectory is still fetched for browser review.
  • Replay safety. Every replayed step that mutates state gets a read-back assert (did the field hold the value? did the banner appear?); destructive sequences (submit) require a per-row dedupe key + resume cursor so a crash at row 150 doesn't re-file rows 1–149; multi-row dry-run before any unattended run; a hard destructive: true flag on the generated skill.
  • At-rest. The buffered recording is encrypted on disk under a distinct key in the OS keychain (not the audit-chain key — a different trust boundary). The shim retains a bounded number by recording ID for fetch/review and best-effort secure-erases an evicted buffer.
  • Audit. The audit log records that a recording happened + the channels + step count + interpretation route — never content. (Open: whether compliance needs a content manifest for data-subject requests, §10.)

9.1 The cloud-fallback consent: calibrated, not loaded

When the local probe (§3.1) comes up empty and screenshots_to_cloud is the only way to build the skill, the shim shows a consent dialog. Its copy is a spec requirement, not a UI detail, because the tone is the control: it has to give the user a real, proportionate understanding of the tradeoff and then get out of the way.

The standard is proportionate disclosure — state the actual shape of the risk and the actual shape of the benefit, in plain words, and let the user decide. Both of the obvious failure registers take the decision away from them:

  • Fear register (banned): "⚠️ HACKERS COULD STEAL YOUR DATA." A real but low-probability harm, inflated until a reasonable person declines a reasonable choice.
  • Minimizing register (banned): "This is a totally chill thing to hit yes on." Robs them of the chance to actually weigh it; trains click-through.
  • Calibrated register (required): what leaves, why it helps, the realistic scope, an honest comparison to trust already given, and the local alternative — then stop.

Reference copy (whoever builds the UI ships this tone, not necessarily these exact words):

Build this skill using the cloud?

Your computer doesn't have a local model that can read these screenshots, so building this skill needs AutoGPT's cloud. If you continue, the screen images from this recording go to our servers, a capable model reads them to write the skill, then they're deleted (or kept per your data settings).

Worth knowing:

  • The images show whatever was on your screen while recording — including anything else that was open. You decide what's visible.
  • They're handled under your AutoGPT deployment's data and retention settings.
  • It's the same trust you already place in AutoGPT to act on your computer, now with screen images for this one recording.

Prefer to keep screen images on your machine? Install a local model and re-record. The structured trajectory will still appear in your authenticated browser for review.

[ Keep it on my machine ] [ Send and build ] ☐ Remember my choice for recordings like this

Notes for implementers:

  • No warning iconography (⚠️/🔒) on the calibrated path — it biases toward the fear register before the user has read a word.
  • The remembered preference is per-kind-of-recording, not global, and is revocable in settings — so we don't re-interrupt, and don't quietly turn a one-time yes into a standing one.
  • This dialog appears only for screenshots_to_cloud. The default extract_then_cloud (text/structure, no pixels) and the local routes do not prompt — see §3.1.

10. Open questions (genuinely the author's / PM's call)

  • Default when no local extractor/VLM exists. Resolved: probe local first (§3.1); if nothing local can build the skill, fall back to screenshots_to_cloud behind the calibrated consent dialog (§9.1) — not a hard block. Reaching the user with an honest choice beats locking them out; INTERPRETATION_UNAVAILABLE is reserved for the case where the user declines the cloud fallback and no local route exists.
  • Parameter-confidence threshold. What bar forces a clarifying question vs. accepts a confirmed-by-2-rows inference? Unenforceable until set.
  • Skill portability. A recording on the user's Mac driving their local apps is machine-bound. Can a "file invoices in QuickBooks Online" (browser, portable) skill graduate into the shared library while a "rename files in Finder" stays shim-local? Two skill classes, or one?
  • Compliance manifest. Does a content-free audit entry suffice, or do we need a reviewable (encrypted, local) manifest of what was captured to service deletion/access requests?
  • Desktop-app coverage floor. For kind: none (canvas/Electron) steps, visual replay works but is slower + less certain. Is there a coverage threshold below which we tell the user "this app records poorly" up front?

11. References

  • COMPUTER_USE.md — the input-injection backends this inverts for the capture floor; the a11y access this enriches with.
  • LOCAL_LLM.md — the on-device interpretation channel (extract_then_cloud / local_vlm) routes through.
  • PRIVACY_MODE.md — the extract-on-device-reason-in-cloud pattern §3 generalizes to screen recordings.
  • AUDIT_LOG.md — records that a recording happened, never its content.
  • PROTOCOL.md — envelope, versioning, the in-flight/backpressure model RECORDING_STEP is exempted from, error-code base set.