如何用 Cypress Cloud CLI 的結構化 Test Replay JSON 排查不穩定 CI 測試

A flaky test gives you an unusually valuable experiment: the same test failed and then passed inside one recorded run.

Instead of rerunning it until the failure disappears, compare the attempts.

The Cypress Cloud CLI exposes Test Replay commands, network requests, and console logs as JSON. Its local replay viewer requires version 0.4.0 or newer, and the current CLI documentation requires Node.js 22.21.0 or later. Cloud CLI is currently Beta, so use the latest supported version for exploration and pin the validated version in repeatable automation.

I am a Cypress Ambassador. IDs and outputs below are synthetic, and the workflow was checked against the public Cloud CLI documentation.

Find a flaky test

npm install --global @cypress/cloud@latest
cy-cloud version
cy-cloud status

cy-cloud test list \
  --projectId <projectId> \
  --runNumber <runNumber> \
  --status passed \
  | jq '.tests[]
      | select((.attempts | length) > 1)
      | {testId, testName, attempts}'

A passing status with multiple attempts means at least one earlier attempt did not pass.

Pull both timelines

Replay attempts are one-indexed. If attempt one failed and attempt two passed:

cy-cloud replay timeline \
  --testId <testId> \
  --attempt 1 \
  --commands --network --logs \
  > failed-attempt.json

cy-cloud replay timeline \
  --testId <testId> \
  --attempt 2 \
  --commands --network --logs \
  > passing-attempt.json

Look for the first material divergence, not merely the final assertion.

Evidence	Failed	Passed
Discount response	Arrived after assertion	Arrived before assertion
HTTP status	200	200
Console error	None	None
Displayed total	`$50.00`	`$45.00`

The API did not fail. The relative ordering changed. That points toward synchronization or a product race rather than a bad selector.

Use a consistent classification so the investigation does not become “flaky” versus “not flaky”:

Class	Evidence pattern	Typical next step
Product race	DOM or state differs before the assertion	Fix application coordination
Test synchronization	Expected event occurs later, with no product defect	Wait on that observable event
Test data	Attempts receive different fixtures or records	Isolate and seed deterministically
Environment	Browser crash, unavailable dependency, resource pressure	Repair CI/runtime conditions
Unknown	Recording lacks the decisive boundary	Reproduce with more targeted evidence

The label is a hypothesis backed by cited events, not a substitute for reproduction.

Narrow to the failure window

Large timelines produce noise. Ask for only the commands around the failed assertion plus nearby browser traffic:

cy-cloud replay timeline \
  --testId <testId> \
  --attempt 1 \
  --commands \
  --aroundFailure 5 \
  --network \
  --logs

Use cy-cloud replay info --testId <testId> first when you want a summary of what the replay contains.

Inspect recorded DOM state

replay view requires Cloud CLI 0.4.0+ and a Chromium-family browser:

cy-cloud replay view open --testId <testId>
cy-cloud replay view move --attempt 1 --time failure
cy-cloud replay view inspect \
  --testId <testId> \
  --selector '[data-cy="cart-total"]'

You can also pin before and after a specific command ID found in the timeline. This helps determine whether an earlier action changed the DOM, styles, or attributes before the assertion surfaced the problem.

Give an AI agent a bounded question

Do not ask, “Make the test pass.” Ask:

Compare failed-attempt.json with passing-attempt.json.
Identify the first material difference before the assertion.
Cite the event IDs supporting the conclusion.
Classify the cause as product, synchronization, test data,
environment, or unknown.
Propose a fix, but do not edit files.
Do not add fixed waits or weaken assertions.

That prompt makes unsupported guesses visible. If the evidence is insufficient, unknown is an acceptable answer.

Keep the evidence packet smaller than the entire replay. Include the failed and passing windows, run metadata, browser/version information, and the exact test source revision. Exclude unrelated request bodies and console content. A bounded packet reduces both privacy exposure and the chance that an AI assistant anchors on irrelevant details.

Verify locally

After reviewing the proposal:

reproduce the scenario with controlled data;
apply the smallest change;
run the affected spec repeatedly;
introduce a negative control that should fail;
confirm the test still protects the business behavior.

This article deliberately covers recorded CI evidence, not a live local Cypress session. cypress tap is useful for the final local run, but it solves a different part of the workflow.

For CI-only failures, rerun the reviewed fix in a pinned Cypress Docker image before changing the test. A full image tag or digest holds Node and browser versions steady:

export CYPRESS_IMAGE='cypress/browsers@sha256:<approved-digest>'
docker compose run --rm cypress \
  npx cypress run --spec cypress/e2e/checkout.cy.ts --browser chrome

Store the image digest and Cypress/browser versions with the failed and passing JSON. Docker cannot reproduce provider load or every CI resource constraint, but it can remove silent runner upgrades from the hypothesis set.

Security boundaries

Cloud CLI is read-only and follows the authenticated user's Cypress Cloud permissions. OAuth is recommended locally; CYPRESS_CLOUD_TOKEN is intended for non-interactive environments.

Still review:

which projects the identity can read;
whether the agent can access shell secrets;
which request/response bodies Test Replay recorded;
local replay-cache retention;
whether generated JSON is safe to attach to an issue.

Structured evidence is better than a guess. It is not automatically safe evidence.

Cloud CLI requests are rate limited, so CI automation should fetch once, cache the resulting artifact within the job, and avoid letting several agents repeatedly query the same replay.

What would you learn by comparing your most recent flaky test's failed and passing attempts side by side?