“Create a calendar event for the design review” sounds simple to a person. For an evaluation, it leaves important questions open: Which calendar? What date and time? How long should the event last? Is a reminder expected? What would prove that the event was saved?
If those details are left implicit, two runs can receive the same sentence yet face different tasks. A testable mobile AI case turns the user’s goal into a clear contract for one attempt. It defines what the agent is asked to achieve, the relevant starting context, the condition that counts as completion, and the evidence needed to judge that condition.
This contract describes the task, not the agent’s route through the interface. The agent may navigate differently across runs. The case should keep the intended outcome and evaluation boundary stable enough to interpret those differences.
Separate the goal from its test parameters
A natural-language request carries user intent, but a benchmark or evaluation often needs explicit parameters. Preserve the user-facing goal, then identify details that must be fixed for the case to be repeatable or assessable.
For example, “Create a calendar event for the design review” can be made specific as follows:
goal: Create a calendar event for the design review.
parameters:
title: Design review
start: 2026-11-03 14:00
end: 2026-11-03 15:00
timezone: America/Los_Angeles
calendar: Work
invitees: noneThe values are part of this illustrative test case, not an ARMARRAY API or a universal calendar-task schema. A real case should define only the parameters needed to remove ambiguity for its purpose. If the task is specifically testing whether an agent asks a clarifying question, intentionally leaving a detail unresolved may be the point; the expected behavior then belongs in the case definition.
Keep the case definition separate from its execution implementation. A UI agent might use the calendar app’s visible controls; another test framework might reach the same app through a different supported interaction path. The task contract says what outcome is required and what can be evaluated. The runner—the system that launches and observes the attempt—determines how it is executed and recorded.
Define the starting context that matters
Initial conditions should be explicit enough to make the case interpretable, but not so prescriptive that they dictate irrelevant interface details. For the calendar example, the case might require an isolated test account with the Work calendar available, the event absent before the run, and a specified device time zone. It may also identify the app and build under test.
These setup conditions are often called a test fixture: they make the task ready to run. A preparation check should confirm them before the attempt begins. If the calendar is unavailable or the event already exists, the setup is invalid or needs a declared variant; it should not silently become an agent failure.
Avoid encoding incidental screen coordinates, a preferred tap sequence, or an exact visual layout unless the evaluation is meant to test those details. Those belong to an implementation or a separate interaction-level objective. A goal-level case should remain valid when the app’s layout changes but the user’s requested result does not.
Write a completion rule that can be checked
“The agent reached the confirmation screen” is not necessarily the same as “the event exists as requested.” Choose a completion rule that matches the user goal. For this example, completion could require one event in the Work calendar with the specified title, start and end time, time zone, and no invitees.
Then state what the checker can actually observe. In an app that exposes suitable state, an evaluator may inspect the saved event fields. Where internal state is unavailable, it may inspect the visible event details after saving or reopening the event. Each method supports a particular claim; a success toast alone may show that the app acknowledged a save request, but may not establish every requested field.
Do not make the criterion depend on an exact path unless the path itself is under test. An agent can reach the same valid state through different sequences. Conversely, a visually plausible screen may not prove that the persistent result matches the request. The case should distinguish the desired state from the evidence used to assess it.
Distinguish task outcomes from setup and evidence issues
A test case benefits from more than a binary label. If the starting fixture could not be prepared, the task attempt may not have begun. If the agent completed the interaction but the result cannot be observed, the outcome may be inconclusive. If the event is observed with an incorrect time, the completion criterion was not met. These situations should not be collapsed into one generic failure.
Define the relevant end states before running the case. A practical set may include:
- Completed: the required event fields are supported by the available evidence.
- Not completed: evidence shows that one or more required conditions were not met.
- Inconclusive: the evidence is missing or conflicting, so the criterion cannot be judged.
- Setup invalid: a required starting condition was absent before execution.
The labels can differ across systems. What matters is that the case defines how they relate to the goal and that setup problems, task outcomes, and evidence gaps remain distinguishable.
Preserve the case and its evidence as a linked record
For each attempt, retain the case revision, input values, fixture result, app and device context, agent or controller version, completion rule, and the evidence used by the checker. Link observations and artifacts to the attempt so a reviewer can tell which event details supported the judgment.
This does not require collecting every possible signal. It requires preserving enough context to answer the question the case was designed to test. A case can be represented compactly, but changes to its parameters, starting conditions, or completion rule should be versioned; otherwise results from different definitions may be mixed under the same label.
Mobile benchmarks illustrate several ways to make these relationships concrete. AndroidWorld describes task-specific initialization, success checking, and cleanup logic. Android in the Wild organizes demonstrations around natural-language instructions, screen observations, and actions, while also exploring variation in instructions, applications, and Android versions. These are examples with different aims, not a single required case format. They show why a task’s prompt, context, and evaluation method should be understood together. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents and Android in the Wild: A Large-Scale Dataset for Android Device Control
For a broader view of the fields needed to interpret evaluation records, see What Should a Mobile AI Evaluation Dataset Contain?. For the collection process that connects actions and observations during an attempt, see How to Collect Mobile Interaction Trajectories on Real Android Devices.
A compact case-design check
Before adding a mobile task to an evaluation, ask:
- Can a reader identify the requested outcome and any parameters that remove ambiguity?
- Are the relevant starting conditions stated and checked before execution?
- Does the completion rule match the user goal rather than an incidental UI path?
- Can the checker observe enough evidence to apply that rule?
- Can setup invalidity, task non-completion, and inconclusive evidence be told apart?
- Are the case definition and its revisions linked to each attempt and result?
If these answers are clear, the case is ready to be implemented in a chosen runner or agent framework. If they are not, adding devices or repetitions will produce more records without resolving what the task means.
When a case requires physical Android execution, ARMARRAY provides the infrastructure layer. Teams can keep their case definitions, agent behavior, and evaluation logic in the systems they already use, while linking each device run to the conditions and evidence needed to interpret its result.