以 JSON 編寫無腳本手機測試：一個檔案支援 Android 和 iOS

Short answer: your app renamed "Sign in" to "Log in", and every recorded test that taps it now fails on Android, then again on iOS. A test stored as a recording can usually only be fixed by recording it again. We store a mobile test as a JSON file instead: actions and selectors, no coordinates. You can write one by hand, have an AI agent write it, diff it in a pull request, and fix it by editing one line. With "platform": "mobile", the same file runs on an Android phone and an iPhone.

This post is the reference we wanted when we built it: what the format looks like, a snippet for every action, how the two platforms share one file, and what the validator says when something is wrong. Every snippet below was checked against the live validator before publishing.

Why recordings rot and files don't

A recorded test usually holds three things that break: a screen position for each tap, a snapshot of the screen it was recorded on, and a device model it silently assumes. Change the layout, the phone or the OS and the recording no longer matches.

A JSON flow keeps none of those. Each step names an element by what a person would call it (its text or its id), and each run looks for that element on the screen as it is right now. Nothing recorded on an Android phone is ever compared against an iPhone, because nothing is recorded at all.

There are three practical consequences:

• You can read it. A reviewer sees tap "Add to cart" in a diff, not a binary blob.
• You can fix it. A renamed button is a one-word edit, not a re-recording.
• Anyone can write it. A tester typing in an editor, a script generating it from a spreadsheet, or an AI agent that has just looked at the app all produce the same file.

Building one takes four moves

1. Start with launchApp. Name the app, so the run always begins on a known screen.
2. Add one step per thing a person does, and name each element by what is on the screen: the word on the button, the label of the field. No coordinates, no screen sizes, no waits for animations.
3. Validate. Each error names the step, the field and the fix, so you never guess what the checker wanted.
4. Import it and replay it on an Android phone and an iPhone.

If you can describe the test to a colleague, you can write it. An AI agent can write it too, from the same description.

The whole thing on one screen

Here is a complete flow that signs in, finds a product and adds it to the cart, on either phone:

{
  "name": "Sign in and add to cart",
  "platform": "mobile",
  "steps": [
    { "action": "launchApp", "appId": { "android": "com.example.shop", "ios": "com.example.Shop" } },

    { "action": "tap", "selectors": [{ "using": "id", "value": "email_input" }, { "using": "text", "value": "Email" }] },
    { "action": "type", "value": "{{secret.SHOP_EMAIL}}" },
    { "action": "tap", "selectors": [{ "using": "id", "value": "password_input" }, { "using": "text", "value": "Password" }] },
    { "action": "type", "value": "{{secret.SHOP_PASSWORD}}" },
    { "action": "tap", "selectors": [{ "using": "text", "value": "Sign in" }] },
    { "action": "assert", "assert": { "kind": "exists" }, "selectors": [{ "using": "text", "value": "Home" }], "timeoutMs": 15000 },

    { "action": "scroll", "selectors": [{ "using": "text", "value": "Running shoes" }] },
    { "action": "tap", "selectors": [{ "using": "text", "value": "Running shoes" }] },
    { "action": "tap", "selectors": [{ "using": "text", "value": "Add to cart" }] },
    { "action": "assert", "assert": { "kind": "textContains", "expected": "1 item" }, "selectors": [{ "using": "id", "value": "cart_badge" }] }
  ]
}

Eleven steps, no code, no coordinates, and no "if Android" anywhere. The rest of this post is what each of those lines can be.

The flow envelope

Selectors: how a step finds its element

A selector is { "using": ..., "value": ... }. You can list several. If one finds nothing, or matches more than one control, the next one is tried. The section after this one explains the order.

"selectors": [
  { "using": "id",   "value": "com.example.shop:id/checkout" },
  { "using": "text", "value": "Checkout" }
]

Three kinds work on both phones:

• text is the visible text or the accessibility label.
• label behaves the same way. Use whichever word reads better to you.
• id is the Android resource-id or the iOS accessibility identifier. If your developers set the same identifier on both platforms, this is the most stable selector you have.

Text matching ignores case, extra spaces and character width, but is otherwise exact. "sign IN" finds "Sign in". "Network and Internet" does not find "Network & internet".

When the wording differs between versions, regions or OS releases, make the selector a pattern. The whole text has to match, ignoring case:

{ "using": "text", "value": "(Allow|Allow While Using App|While using the app)", "regex": true }

regex works on text and label only. Keep patterns simple: the validator refuses patterns that could hang, such as nested repeats or back-references.

Four more selector kinds exist. They are valid only where the step can run on one OS, which is explained under "One file, two platforms" below.

An xpath must select by attribute, as in //*[@text='Save']. A position-only path such as //LinearLayout[3]/Button[1] is refused, because it describes where the element sat on the day you wrote it.

One rule that saves hours: the run never guesses between two controls. A "Settings" row and the "Settings" text inside it are one control, and count once. Five "Delete" buttons are five controls. If no selector narrows them to one, the step fails and says how many matched, and you add a more specific selector. A test that taps the first of five "Delete" buttons passes until the day it deletes the wrong row.

How a step runs on the phone

Nothing in the file says where an element is. Every step that has selectors finds its element on the live screen, in five moves:

1. Read the screen. The run reads the phone's live element tree, the same one accessibility services see. It reads it again every 500 ms until the step succeeds or runs out of time.
2. Rank the selectors. Each platform tries them in its own fixed order. On Android that is the visible text first, then the accessibility label, then the resource-id. On an iPhone it is the accessibility identifier first, then the label, then the value. The order you list them in only breaks ties.
3. Keep one control. A selector that matches one control wins. Copies of one control count once. Two different controls make that selector ambiguous, and the next one is tried.
4. Bring it on screen. A long list keeps rows the eye cannot see. If the element is there but off screen, the run scrolls toward it, up to five times.
5. Act at its centre. The touch lands in the middle of the element as it is drawn now, on this phone, at this screen size. A tap that would only hit a large container, such as the whole window, is refused, because it would pass without pressing anything.

If time runs out (5 seconds, or the step's timeoutMs), the step fails and says why: the element was not found, or several controls matched and it names how many. An optional step is skipped instead.

Here is what each action does once its element is known:

It runs on the phone, not through a grid. A phone flow does not go through the Selenium or Appium grid that automation suites use. The run drives the phone directly from the machine it is plugged into. There is no hub to route through, no node to wait for and no remote session queue, so nothing sits between the step and the screen. Each step is one read of the screen and one gesture, and the time between steps is the app's time, not the infrastructure's.

Every action, with a snippet

launchApp: start the app under test

{ "action": "launchApp", "appId": { "android": "com.example.shop", "ios": "com.example.Shop" } }

A package name and a bundle id are different strings, so a mobile flow names both. A single-platform flow can use a plain string, and Android can name a launch activity:

{ "action": "launchApp", "appId": "com.example.shop", "appActivity": ".MainActivity" }

Make it the first step. If you don't, the validator warns you that replay starts on whatever screen the phone happens to be showing.

tap

{ "action": "tap", "selectors": [{ "using": "text", "value": "Sign in" }] }

doubleTap

{ "action": "doubleTap", "selectors": [{ "using": "id", "value": "product_image" }] }

longPress

{ "action": "longPress", "selectors": [{ "using": "text", "value": "Size 42" }], "durationMs": 800 }

durationMs is optional, from 100 to 10,000.

type: enter text into the focused field

{ "action": "tap", "selectors": [{ "using": "text", "value": "Search" }] },
{ "action": "type", "value": "running shoes" }

type has no selector. It types into whatever field has focus, so tap the field first. This is the same order a person follows, and it means the keyboard is up before any text is sent.

Passwords and other secrets never go into the file. Save them once under Variables, and refer to them by name:

{ "action": "tap", "selectors": [{ "using": "text", "value": "Password" }] },
{ "action": "type", "value": "{{secret.SHOP_PASSWORD}}" }

If the field you just tapped looks like a password field and the value is literal text, the flow is rejected rather than stored. Phone flows have no environment variables: {{secret.KEY}} is the one reference they resolve.

scroll: bring an element into view

{ "action": "scroll", "selectors": [{ "using": "text", "value": "Running shoes" }] }

You name what you want to reach, not how far to swipe. The run scrolls until the element is on screen, so the step survives a list that grows or a taller phone.

wait

{ "action": "wait", "value": 1500 }

Milliseconds, from 1 to 120,000. Reach for it last. A fixed wait is either too short on a slow network or wasted time on a fast one. To wait for a screen, assert on something it shows and give that step a longer timeoutMs:

{ "action": "assert", "assert": { "kind": "exists" }, "selectors": [{ "using": "text", "value": "Home" }], "timeoutMs": 15000 }

assert: four kinds

{ "action": "assert", "assert": { "kind": "exists" },    "selectors": [{ "using": "text", "value": "Order placed" }] },
{ "action": "assert", "assert": { "kind": "notExists" }, "selectors": [{ "using": "text", "value": "Something went wrong" }] },
{ "action": "assert", "assert": { "kind": "textEquals",   "expected": "Payment received" }, "selectors": [{ "using": "id", "value": "order_status" }] },
{ "action": "assert", "assert": { "kind": "textContains", "expected": "1 item" }, "selectors": [{ "using": "id", "value": "cart_badge" }] }

textEquals and textContains need expected. exists and notExists must not have it. The text comparison is the same forgiving-but-exact one that selectors use.

pressButton: hardware buttons

{ "action": "pressButton", "button": "home" }

The buttons are home, volumeUp, volumeDown and lock, and all four work on both phones.

key: Android back

{ "action": "key", "key": "back" }

or any Android key code:

{ "action": "key", "keyCode": 66 }

iPhones have no back key, so key is Android-only. In a mobile flow it has to sit inside a platform block, which is the next section.

if: branch on what is on screen

{
  "action": "if",
  "condition": { "check": "exists", "timeoutMs": 3000 },
  "selectors": [{ "using": "text", "value": "(Allow|Allow While Using App|While using the app)", "regex": true }],
  "then": [
    { "action": "tap", "selectors": [{ "using": "text", "value": "(Allow|Allow While Using App|While using the app)", "regex": true }] }
  ]
}

That handles the permission dialog that appears on a fresh install and is gone on the second run. check takes the four assert kinds. then and else each hold a list of steps, and blocks nest up to five deep.

Two options every step accepts

{ "action": "tap", "selectors": [{ "using": "text", "value": "Not now" }], "optional": true, "timeoutMs": 2000 }

• optional: true skips the step when its element is absent, instead of failing the run. Use it for a promotion banner that appears sometimes.
• timeoutMs (100 to 120,000) is how long the step keeps looking for its element.

One file, two platforms

The rule is short: shared selectors by default, and a block for real differences.

Most steps never mention a platform. text and id resolve on both phones, and launchApp takes both app ids. When the two apps genuinely differ, wrap that part in platformIs:

{
  "action": "if",
  "condition": { "check": "platformIs", "expected": "android" },
  "then": [ { "action": "key", "key": "back" } ],
  "else": [ { "action": "tap", "selectors": [{ "using": "value", "value": "Back" }] } ]
}

else is the other phone. Inside each branch the flow is single-platform, so the Android-only key step and the iOS-only value selector become legal. There are no per-step platform overrides. Every difference lives in a block you can see, which tells a reviewer exactly where the two apps diverge.

The validator tracks which OS each step can run on. A branch that can never run is an error, such as an ios block inside an Android-only flow. So is a platform-specific selector outside a block.

What the validator says when you get it wrong

Here is a flow with four mistakes in it:

{
  "name": "Broken on purpose",
  "platform": "mobile",
  "steps": [
    { "action": "launchApp", "appId": "com.example.shop" },
    { "action": "tap", "selectors": [{ "using": "accessibilityId", "value": "login_button" }] },
    { "action": "tap", "selectors": [{ "using": "text", "value": "Password" }] },
    { "action": "type", "value": "hunter2" },
    { "action": "key", "key": "back" }
  ]
}

And here is the validator's reply, verbatim:

Not valid: 4 errors. Nothing was saved.
- steps[0].appId: a package name and a bundle id differ; give {"android":"com.example.app","ios":"com.example.App"}
- steps[1].selectors[0].using: "accessibilityId" is Android-only; put this step in an "if" with
  condition {"check":"platformIs","expected":"<android|ios>"}, or make the flow single-platform
- steps[3].value: this types into a password field; a literal value is not stored.
  Use {{secret.KEY}} and save the secret under Variables
- steps[4].action: "key" is Android-only; put this step in an "if" with
  condition {"check":"platformIs","expected":"<android|ios>"}, or make the flow single-platform

Every error names the step, the field and the fix. A field an action does not use is an error too, not something silently dropped, so a typo like "selector" instead of "selectors" is caught before the first run. That precision is what makes the format work for agents: the reply is specific enough to correct without a human reading it.

Uploading it: in the app, or through your AI

In the app. Open Test Manager → Tests and click Import JSON. Drop the .json file (up to 1 MB), choose it, or paste it into the box. The flow is checked before anything is saved. Errors appear as step N · field · message, and Import stays disabled until they are fixed. Warnings, such as a {{secret.KEY}} you have not saved yet, don't block it. Click Import and the flow becomes a new test with its own T### id.

Through your AI assistant. Connect Claude, Cursor or any other MCP client to RobotActions and hand it the file:

Validate sign-in-and-add-to-cart.json with flow_validate. If it is valid, save it unchanged with
flow_import.

flow_validate is the same dry run the dialog does, and flow_import is the same save. The file goes in as-is either way.

Because a flow is a plain file, everything you already do with files works here. Keep flows next to the app code, review them in pull requests, and upload the reviewed version.

Having an agent write it

An agent uses the same tools, plus one: flow_validate (a dry run), flow_import (save), and flow_replay_start (run it on a device). A prompt as plain as this one is enough:

Open the shop app on an Android phone and work out the sign-in and add-to-cart flow. Write it as
a mobile flow, validate it, save it, then replay it on an Android phone and an iPhone and tell
me which steps failed.

An agent that has the device in front of it reads the real element tree before it writes a selector, so it writes id: email_input because that id exists, not because it sounds plausible. If flow_validate sends back an error, the error tells the agent which field to change. The agent never has to guess what the validator wanted.

Fixing it when the app changes

This is where a file beats a recording. When a release renames "Sign in" to "Log in":

1. Click the test's Export JSON icon. You get <name>.json: the flow, plus its recordingId and version.
2. Change one word.
3. Click the test's Update from JSON icon and load the file. The dialog shows what will change, for example replaces 11 steps with 11 · v3 → v4. Click Replace steps.

An AI assistant does the same with flow_export, an edit, and flow_update with that recordingId and expectedVersion.

-    { "action": "tap", "selectors": [{ "using": "text", "value": "Sign in" }] },
+    { "action": "tap", "selectors": [{ "using": "text", "value": "Log in" }] },

The test keeps its T### id, its test case link and its replay history. If someone else saved the test after your export, you are told before anything is overwritten, so two people can't silently undo each other's work. Every update is versioned (the last ten are kept), and Version history → Revert in the flow editor undoes one.

If both wordings will exist for a while, say during a staged rollout, don't pick one. Match both:

{ "using": "text", "value": "(Sign|Log) in", "regex": true }

When a step fails on replay, the failure message is specific in the same way the validator is. It says the element was not found, or that several matched, and which selector it tried. Most fixes are one selector, so most fixes are one line.

Recordings and JSON are the same flow

You don't have to choose between recording and writing. A flow recorded on a real device exports to this JSON, so you can record the first draft by hand and then tidy the selectors in an editor.

One limit is worth knowing. Steps that only make sense as screen positions, such as a freeform swipe or a tap on a bare coordinate, can't be written as selectors. They are listed as left out when you export a recorded flow, and flow_update refuses to rewrite that flow from JSON rather than drop them. You keep the recording, and nothing is lost behind your back.

The same idea for a browser

Web flows use the same envelope with "platform": "web" and a browser-shaped set of actions (navigate, tap, type with a selector, hover, key, back, dialog, page-level asserts such as urlContains). They also support environment variables ({{BASE_URL}}). One flow can therefore run against staging and production.

{
  "name": "Browse skills",
  "platform": "web",
  "steps": [
    { "action": "navigate", "url": "{{BASE_URL}}" },
    { "action": "tap", "selectors": [{ "using": "role", "value": "link=Browse the skills" }] },
    { "action": "assert", "assert": { "kind": "urlContains", "expected": "/skills" } }
  ]
}

Where RobotActions fits

The format is ours, but the habits behind it work in any tool: name elements instead of positions, keep secrets out of test files, make every platform difference an explicit block, and fail loudly on ambiguous matches. On RobotActions, a flow runs on real Android phones and iPhones in the cloud, singly or as a suite, and links back to its case in TestRail, Jira or Azure DevOps.

Write your first flow in an editor and import it, or ask your agent to write it, then replay it on both phones. Try it free · No credit card required →

Related

• Why Your AI Agent Writes Appium Tests That Don't Run
• Why Your Appium Suite Goes Red Every Redesign
• Automating React Native on iOS When the Page Source Is Unreadable

Originally published at robotactions.com/blog/scriptless-mobile-tests-json-android-ios.