#Replay & E2E Testing

Record a session as an .ad script, then run it again with replay or run a folder of scripts as an E2E suite with test.

replay has a 60-second wall-clock budget within a 90-second client timeout. Use replay workflow.ad --timeout 300000 for a five-minute operation budget and a five-minute, 30-second client timeout. At the budget, the running step receives the same cancellation as a canceled request. When it settles, a failed replay includes details.reason: daemon_request_timeout and details.timeoutMs. The daemon and other sessions stay available if it responds to the client. An authored close may already have ended its session. A failed replay keeps the failing script path and line. A replay that completes every step returns success even if it settles shortly after the deadline. Cancellation does not prove that native device work has stopped. If a step ignores cancellation beyond the remaining 30 seconds, the client may report a transport timeout. For a daemon that does not respond, the client's existing liveness check can reset it. Direct test runs keep their own timeout behavior.

If a resource step cannot run within replay's budget, the error includes details.reason: bounded_composite_resource_unsupported, details.shape, details.dispatched: no for that step, and details.standaloneCommandName for a known public command. This can affect perf, audio, record, trace, and some logs operations. It also affects close when it would dispose owned resources or release a provider lease, and acquisition that would recover another session's durable resources. Opening another app in the same session keeps its owned resource handles. An Android test IME alone is restored by authored close, even when restoration settles after the remaining replay time. A failed restore retires the session and retains the device claim for recovery; inspect both before acting. The name is not a complete command. Inspect the failed script path and line, then run that original operation separately with its original arguments and options, the same --session value, and the same private CLI context. If steps remain and the report offers resume.alternateFrom, use it with planDigest after completing the refused step directly. resume.from retries the same bounded step and refuses it again. A final unrecorded logs or audio step leaves no replay tail to resume. Inspect completed steps before retrying; earlier steps may have changed the app. A nested test with an attempt timeout or --record-video is refused before its first attempt. Direct test retains its own timeout behavior. If a nested test reaches a bounded resource refusal, it stops after that attempt even when you configured retries. Other test failures can still retry. An authored .ad command without a resource classification fails closed with details.shape: <command>:unclassified and the same original-input guidance.

To run Maestro YAML flows or export .ad scripts to Maestro YAML, see Run Maestro flows.

#How it works

You work in two passes:

1. Explore: discover elements and act on refs (snapshot -> click @e.. / fill @e..) while recording.
2. Replay: run the recorded .ad script with replay for a deterministic run.

#Record a replay script

Pass --save-script to open. When you close the session, the script is written:

agent-device open Settings --platform ios --session e2e --save-script
agent-device snapshot -i --session e2e
agent-device click @e13 --session e2e
agent-device close --session e2e

By default, the script goes to:

~/.agent-device/sessions/<session>-<timestamp>.ad

To choose the output file, pass a path to --save-script:

agent-device open Settings --platform ios --session e2e --save-script ./workflows/e2e-settings.ad

• Missing parent directories are created.
• The script is written on the machine running the daemon, so --save-script is rejected when you use a remote daemon.
• For a bare file name that could be read as another argument, use --save-script=workflow.ad or a path-like value such as ./workflow.ad.

#.ad line grammar

A .ad line is the CLI spelling of one command: <command> [positional ...] [flag ...]. Whitespace separates tokens, so a value with a space needs quotes.

open "com.example.app" --relaunch
scroll down --until 'id="far-button"'
press id="far-button"
wait 'label="Order summary"' 5000
close

• A token quoted with " or ' is one argument. Single quotes keep a " literal, so 'id="far-button"' and "id=\"far-button\"" are the same selector — write whichever matches how you typed the command at the shell.
• Values in double quotes are JSON strings, so escape \\, \", \t, and \n. Values in single quotes are literal, as at the shell: a backslash keeps its own character, and the only escape is \' for an apostrophe.
• A script line keeps only the flags that command records; CLI-only spellings and per-request options are not part of a step. The common flags a script does not carry are --settle, --verify, scroll --pixels/--duration-ms, and the device-selection flags (--platform, --serial, --device).
• help <command> prints the flags each command accepts. help scripting prints this grammar.

To reach an off-screen element, use a stop condition rather than a fixed scroll amount. A fixed amount passes on one screen size and fails on another:

# repeats until the element is on screen
scroll down --until 'id="checkout-submit"'
# one gesture, viewport-relative
scroll down 0.8
# run to the end of the content
scroll bottom

A # only starts a comment at the beginning of a line. A scroll line carries --until and keeps its distance as a positional (scroll down 0.8 --until <selector>). wait carries --raw, --depth <n>, and --scope <selector|@ref> to choose the capture its target is read from. Scripts use only these long spellings: the -d/-s CLI aliases are not flags in a script, so a hand-written line such as wait text -s so funny still waits for that literal text.

#Run replay

agent-device replay ~/.agent-device/sessions/e2e-2026-02-09T12-00-00-000Z.ad --session e2e-run

• Replay reads .ad scripts.

• The CLI reads script paths on your machine and sends the script, with the Maestro runFlow includes it can resolve and read, to the daemon. An include it cannot read is left out, and the run fails if it reaches that include. The same replay or test command works against a local or a remote daemon without copying files. A script that doesn't exist on your machine fails at once, naming the path you typed.

• A script that does not end in close leaves its session open. For a script that does end in close, pass --keep-session to skip only that final action and keep working in the same session:

  agent-device replay ./checkout.ad --session e2e-run --keep-session
  agent-device snapshot -i --session e2e-run

  Earlier close actions still run. test does not accept the flag because each suite attempt cleans up its own session.

• press, click, and longpress steps do not wait for a missing target to appear. If a screen needs time to load, add an explicit wait before the interaction, such as wait 'label="Continue"' 5000. Covered, off-screen, and ambiguous targets fail as they do in live commands.

• When the target is missing, replay stops with REPLAY_DIVERGENCE. For a step recorded with a target annotation (the # agent-device:target-v1 line above it), error.details.divergence.kind is selector-miss and the step is never sent. For a step without an annotation, error.details.reason is selector_not_found, as for a live command.

• A step whose read names one element (is with a predicate other than exists/absent, or get attrs) that reaches dispatch with an ambiguous selector fails with AMBIGUOUS_MATCH as the divergence cause — the same code the live command reports, so an ambiguous recorded screen is not replayed as a missing one. An annotated step can also stop earlier, before the command runs: when pre-dispatch target binding cannot confirm the recorded identity (its identity set has more than one member with no sibling or viewport signal isolating one, or no fresh snapshot can be captured to verify against), the divergence cause is IDENTITY_UNVERIFIABLE instead. Both codes say the recorded screen changed; only AMBIGUOUS_MATCH proves your selector matched more than one element.

#Run a lightweight .ad suite

agent-device test ./workflows
agent-device test "./workflows/**/*.ad" --platform android
agent-device test ./workflows --timeout 60000 --retries 1
agent-device test ./workflows --artifacts-dir ./tmp/agent-device-artifacts
agent-device test ./workflows --reporter default --reporter junit:./tmp/junit.xml

• test discovers .ad files from files, directories, or globs and runs them serially.
• Quote relative globs to expand them on the caller from its working directory, including when the directory name contains glob characters such as [ or {. A missing file input without glob characters reports an error.
• The context platform=... header inside each .ad file decides which platform that file runs on.
• --platform filters which files run; with a filter, files without a context platform= header are skipped. When filtering leaves no runnable sources, the no-match error reports how many sources were skipped for having no context platform= header versus how many declared another platform. Add the header to run a file under a filter, or omit --platform and let the selected device decide.
• Set context timeout=... and context retries=... per script; CLI flags override them. Retries are capped at 3, and duplicate keys in the context header fail fast instead of silently overriding each other.
• By default, suite artifacts are written under .agent-device/test-artifacts/<run-id>/.... Each attempt writes replay.ad, result.txt, and replay-timing.ndjson. Failed attempts also keep copied logs and artifact files when the replay produced them.
• Copied diagnostic artifacts receive numbered filenames when their names collide with another artifact, a replay source, timing trace, or attempt manifest. result.txt lists the retained names in copiedArtifacts.
• replay-timing.ndjson records attempt, cleanup, and per-step start/stop events with durations. Upload it from CI even for passing runs when comparing local and CI performance.
• When an attempt hits its timeout, it is marked failed and the replay gets a short grace period to stop before the session is cleaned up.
• The default text reporter streams live progress on stderr while a suite runs, then prints the final summary, failed tests, and passed-on-retry flaky tests. Use --verbose to include step traces in completed-test progress output.
• The default reporter prints a Warnings: section after the summary when any test accumulated composable warnings — for example a Maestro step with optional: true that was skipped — whether that test passed or failed. A failing replay run repeats the warnings it accumulated as Warning: lines after the error. --json carries the same strings in each test result's warnings array.
• --reporter is repeatable. Built-ins are default for the console summary and junit:<path> for JUnit XML. Passing any explicit reporter list replaces the implicit default reporter, so include --reporter default when you also want terminal output. --report-junit <path> is an alias for --reporter junit:<path>.
• JUnit reports preserve legal Unicode and whitespace, and replace characters forbidden by XML 1.0 (such as terminal ESC or NUL) with U+FFFD (�) so CI parsers can read the report. JSON and other reporters retain the original suite values.
• When --fail-fast and retries are both set, the current test still consumes its retries before the suite stops.

#Custom test reporters

A custom reporter formats suite output. It runs in the local CLI process, not the daemon, and can render both live progress and the final result.

agent-device test ./workflows --reporter ./scripts/replay-reporter.mjs

A reporter module can export a reporter object, reporter, createReporter, or a default factory. Factories receive a load context. Reporter hooks receive replay test objects and an IO context with stdout and stderr streams:

// scripts/replay-reporter.mjs
import fs from 'node:fs';
import path from 'node:path';

export default function createReporter(loadContext) {
  return {
    name: 'summary-file',
    onTestStep(test, context) {
      context.stderr.write(`running ${test.file} ${test.stepIndex}/${test.stepTotal}\n`);
    },
    onSuiteEnd(suite, context) {
      context.stdout.write(`finished ${suite.total} tests\n`);
      fs.mkdirSync('./tmp', { recursive: true });
      fs.writeFileSync(
        path.join('./tmp', 'report.txt'),
        JSON.stringify(
          {
            total: suite.total,
            passed: suite.passed,
            failed: suite.failed,
            modulePath: loadContext.modulePath,
          },
          null,
          2,
        ),
        'utf8',
      );
    },
    getExitCode(suite) {
      return suite.failed > 0 ? 1 : 0;
    },
  };
}

For a live terminal reporter that prints each completed test as an emoji, title, and duration:

// scripts/emoji-reporter.mjs
export default {
  name: 'emoji-status',
  onTestResult(test, context) {
    const icon = test.status === 'pass' ? '✓' : test.status === 'fail' ? '⨯' : '-';
    const title = test.title?.trim() || test.file;
    const duration =
      typeof test.durationMs === 'number' ? ` ${(test.durationMs / 1000).toFixed(2)}s` : '';

    context.stderr.write(`${icon} ${title}${duration}\n`);
  },
};

TypeScript reporters use the same object shape; compile them to JavaScript before passing them to --reporter:

const createReporter = () => ({
  name: 'typed-reporter',
  onSuiteStart(suite, context) {
    context.stderr.write(`starting ${suite.runnable} tests\n`);
  },
  onTestResult(test, context) {
    context.stderr.write(`${test.status} ${test.title ?? test.file}\n`);
  },
  onSuiteEnd(suite) {
    // Write artifacts, annotations, or summaries from suite.
  },
});

export default createReporter;

The CLI loads reporter modules with Node dynamic import(). Use .mjs or .js files at runtime; for TypeScript, compile the reporter to JavaScript before passing it to --reporter. Loading .ts files directly depends on Node's type-stripping behavior and is not part of the supported reporter contract.

The live hooks onSuiteStart, onTestStart, onTestStep, and onTestResult run while the suite is running; reporters do not receive generic command progress. Live hooks run as events arrive and are not awaited, so keep their work synchronous and move anything async to onSuiteEnd, which the CLI awaits before exiting. onSuiteEnd receives the final suite result. getExitCode can only raise the suite exit code, never lower it: the highest reporter-provided code wins and failed tests still exit with 1 when no reporter raises it further, so a reporter cannot mask a failing suite. Return an integer from 0 to 255, or undefined to leave the exit code unchanged. Other values fail with INVALID_ARGS; in particular, codes such as 256 are rejected before they can wrap to a successful process exit.

#Parametrise .ad scripts

Substitute ${VAR} tokens in .ad scripts using values from the CLI, shell env, script-local env directives, or built-ins.

context platform=android
env APP_ID=settings
env WAIT_SHORT=500

open ${APP_ID} --relaunch
wait ${WAIT_SHORT}
click "label=${APP_ID}"

#Precedence

#Built-ins

replay and test provide these built-ins in the reserved AD_* namespace.

• AD_PLATFORM - matches context platform=... or the selected platform when available
• AD_SESSION - active session name
• AD_FILENAME - path of the running .ad file
• AD_DEVICE - device identifier (when --device is set)
• AD_ARTIFACTS - attempt artifacts directory (when running under test)

User-defined keys starting with AD_ are rejected in env, -e, and shell imports such as AD_VAR_AD_FOO, so built-ins cannot be overridden.

Substitution happens inside parsed string values. It does not create extra arguments, so quote selectors or text values that contain spaces:

env SETTINGS="label=Account || label=Profile"
click "${SETTINGS}"

#Fallback and escape

wait ${WAIT_MS:-500}

${VAR:-default} yields default when VAR is unset.

echo "Price: \${APP}"

\${APP} emits a literal ${APP} with no substitution.

#Recipes

Run one flow against two app variants in CI:

agent-device test ./flows/login.ad -e APP_ID=com.example.debug
agent-device test ./flows/login.ad -e APP_ID=com.example.release

Tune timings locally without editing the script:

AD_VAR_WAIT_SHORT=2000 agent-device replay ./flow.ad

Extract a reusable selector. Before:

click "label=Account || label=Profile || label=User"
wait 500
click "label=Account || label=Profile || label=User"

After:

env SETTINGS="label=Account || label=Profile || label=User"

click "${SETTINGS}"
wait 500
click "${SETTINGS}"

Quote ${VAR} inside selector expressions so the whole expression is treated as a single argument.

#Notes

• AD_VAR_* values come from the shell that runs the CLI, so they apply the same way whether the daemon runs locally or remotely.
• Fallbacks do not nest: ${A:-${B}} is not supported.
• An unresolved ${VAR} fails with a file:line reference, so a misspelled variable name stops the run.

#Replay divergence and resume

A failing replay/test step returns a structured REPLAY_DIVERGENCE error. The report is size-bounded and redacted, and carries:

• step — the 1-based plan index and its source file/line (through Maestro runFlow includes).
• screen — a fresh post-failure snapshot digest with actionable refs, or unavailable with a reason/hint when capture failed or was sparse (never a stale tree).
• suggestions — up to 5 ranked, re-resolved candidates for the failing selector (id match ranks above role+label, which ranks above label-only), each with a basis you can inspect before acting. For a verified target-v1 annotation, equally strong candidates in the recorded scroll region rank first; document order breaks remaining ties.
• resume — whether and how to continue without re-running the script from the top.

{
  "code": "REPLAY_DIVERGENCE",
  "details": {
    "divergence": {
      "step": { "index": 4, "source": { "path": "flow.ad", "line": 6 } },
      "screen": {
        "state": "available",
        "refsGeneration": 3,
        "refs": [
          /* ... */
        ],
      },
      "suggestions": [{ "selector": "id=\"auth_continue\"", "basis": "id" }],
      "resume": { "allowed": true, "from": 4, "planDigest": "…64 hex chars…" },
    },
  },
}

Text output prints a compact summary of the same fields; --json/MCP carry the full object.

#Resume a failed replay

During a repair armed with replay --save-script, diagnostic reads stay out of the healed script. To keep a corrective read, pass --record on that invocation, or record: true in that SDK/MCP call. --record is per-invocation only: the config key record and AGENT_DEVICE_RECORD were removed, and a config file or environment still naming them fails with migration guidance. Authored replay observations are retained automatically.

replay --from <n> --plan-digest <sha256> resumes at plan step n, not after it, skipping 1..n-1 without executing them. Both flags come from a divergence report's resume field — from is the failed step, planDigest is the digest of the exact unchanged plan that produced it.

Choose one recovery workflow:

1. Change the replay script. Review the suggestion, edit the selector or include, then run a fresh full replay ./flow.ad. The old digest is intentionally invalid after any plan edit; do not combine it with the edited script. A later divergence supplies a new digest.
2. Keep the replay plan unchanged. Repair app/device state so the reported failed step can succeed when retried, then resume with the report's unchanged from and planDigest. If you manually complete the failed action itself, the reported from will execute it again; only do that when repeating the action is safe.

The unchanged-plan resume loop is:

1. Run replay ./flow.ad. On failure, read resume from the divergence.
2. Leave the script, includes, platform, and target unchanged. Repair app state yourself so the failed step can be retried safely. Resume does not restore app state; it only skips the earlier steps.
3. replay ./flow.ad --from <resume.from> --plan-digest <resume.planDigest>.

agent-device replay ./flow.ad
# ... REPLAY_DIVERGENCE, resume: { allowed: true, from: 4, planDigest: "ab12...“ }
# (repair app state on the device)
agent-device replay ./flow.ad --from 4 --plan-digest ab12...

For how --from counts steps in a Maestro flow, see Resume a failed Maestro flow.

Passing --plan-digest that no longer matches the current script — because you edited it, an include changed, or platform-conditioned expansion differs — fails INVALID_ARGS before any action; run a fresh full replay to get a new digest. --from is replay-only; test rejects it (a suite run must stay full and deterministic).

#--update/-u

--update/-u does not rewrite .ad files. The flag is accepted and does nothing: every replay divergence carries ranked suggestions whether or not you pass it. Review a suggestion, then edit the .ad file yourself if it's right. ADR 0012 explains why replay does not rewrite scripts automatically.

#Troubleshooting

• Replay fails after UI/layout changes:
  • Read the divergence report's suggestions and repair the selector by hand; there is no automated rewrite. Because the edit changes the plan digest, run a fresh full replay instead of using the old resume flags.
• Repeated re-runs are slow or the app is stateful, but the script is still correct:
  • Repair app state and resume with the unchanged --from/--plan-digest. See Resume a failed replay.
• Replay file parse error:
  • Validate quoting in .ad lines (unclosed double quotes are rejected). A selector with a space needs quotes, and help scripting states how each quote form decodes.
• A replay passes on one device size and fails on another because a target was off screen:
  • The script used a fixed scroll amount. Replace it with the stop condition, scroll down --until <selector>, which repeats until the element is actually on screen.
• A press or click step fails because its target was not found, but the element is on the screenshot:
  • A selector-miss divergence means the target was absent from the captured accessibility tree. Check the selector for this build and whether the app exposes the element to accessibility. Add a wait step for a landmark on a loading screen before the press.