VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics

Summary

VenusBench-Mobile is a new online benchmark for mobile GUI agents built on the AndroidWorld emulator harness. The paper's central conceptual move is to reframe evaluation from an app-centric, bottom-up curation (start from apps, exercise their functionalities) to a user-intent-centric, top-down curation (start from ten user-need categories such as Function Assistance, Conflict, Vague Instructions, Multi-round Interaction, GUI State Awareness, Visual GUI Manipulation, Hard GUI Browsing, Noise Resistance, Browsecomp-like corner cases, and Stability Evaluation) and to pair each task with a five-dimensional capability annotation (Perception / Understanding / Decision / Action / Memory, each with four proficiency levels — PUDAM). The benchmark contains 149 primary tasks over 27 apps plus 80 systematic variants for stability, and adds infrastructure (multi-round dialogue, noise injection, MLLM-as-a-Judge for the ~90 semantically-verified tasks) that the base AndroidWorld harness lacks. Empirical results are striking: SOTA agents lose ~50 pp relative to prior benchmarks, GSA/GUIM categories collapse to near-floor rates, and stability pass rates hover between 0 and 15% under language / dark-mode / tablet perturbations, with capability-level breakdowns pointing at Perception, Memory and high-level Decision as the recurring bottlenecks. The reframing is a genuine contribution and the PUDAM overlay is a reasonable diagnostic. The main conceptual limitation is that the paper repeatedly promotes single-run success rates on small, hand-curated denominators (per-category cells as small as 9-10 tasks, an SE base of 20, no seeds) into causal claims about capability bottlenecks, and confounds planner quality with the shared UI-Venus-72B grounding executor on the closed-source rows. Combined with an MLLM-Judge from the same family as one of the evaluated agents and an undocumented tag-to-category mapping in the released code, the evidence supports the qualitative story (harder than AndroidWorld, brittle under perturbations, weak on state and memory) but not the more absolute headline framings.

Strengths

• Conceptual reframing from apps to intents. Explicitly moving from "cover app X's features" to "cover user need Y across whatever apps satisfy it" is a substantive rethink of what a GUI-agent benchmark should measure, and it plausibly explains why frontier agents drop so far below their AndroidWorld numbers.
• Capability annotations layered onto success rates. The PUDAM taxonomy, with four progressive levels per dimension, gives a principled way to move past aggregate SR and localise failures to specific abilities; Figure 6 and Table 8 make the L1-2 vs. L3-4 gaps concrete.
• Coverage of scenarios prior benchmarks ignore. GUI State Awareness (dynamic timing), Visual GUI Manipulation (drawing, image editing), Conflict / Refusal, Vague Instructions and Multi-round Interaction are all under-represented in existing suites; the collapse to 2.0% / 4.4% on GSA / GUIM is a real diagnostic contribution.
• Systematic stability harness. The 20-base × 5-condition SE subset (Original + Chinese + Variation + Dark + Pad) with the SPR "must pass all conditions" metric operationalises consistency more cleanly than one-off perturbation studies, and the near-zero SPRs are a striking empirical finding.
• Hybrid verification design. Distinguishing programmatic OS-state checks from MLLM-as-a-Judge (with seven task-specific prompts for BC/HGB, FA, CF, locating, video, drawing, editing) is a sensible response to categories whose success is inherently semantic, and the illustrative positive/negative examples in Appendix B.4 aid understanding.
• Efficiency accounting. Reporting Total-Tokens and Per-Step-Tokens alongside SR — and the observation that the MA3 loop costs roughly 3× the base per-step budget — is a useful practical dimension that most GUI-agent papers omit.

Weaknesses

• Internal count inconsistencies. The Table 6 per-app "Primary" column sums to ~199, and the "Subset" column to ~66 — not the paper's headline 149 and 80. Because 149 anchors Figure 3, the "90 of 149" MLLM-judged fraction, and the total-SR denominator, this discrepancy propagates.
• Single-run SR on small denominators, no variance. Per-category cells are 9-16 tasks and SE is 20 base tasks. A single trajectory can shift a cell by 5-10 pp, yet Table 3, Table 4 and Table 8 are point estimates without seeds, sampling settings, or repeat trials. Many of the comparisons the paper draws (Kimi K2.5 vs. UI-Venus-72B; the "3.0×" MA3 overhead; the 15% "best" SPR) are within plausible one-run noise.
• Confounded closed-source configuration. Gemini-3-Pro, GPT-5.1 and Seed 2.0 share the UI-Venus-72B grounding executor, whereas open-source rows do their own grounding. Their 36.9 / 26.9 / 18.1 totals therefore mix planner ability with UI-Venus-72B's grounding, inflating the gap on categories where grounding is the bottleneck (GUIM, A3-level tasks).
• Judge validity is unmeasured. The MLLM-Judge is Qwen3-VL-30B-A3B-Instruct — the same family as an evaluated agent — and no inter-rater / human-vs-judge agreement is reported for any of the seven prompts. Since 90/149 tasks depend on it, all downstream SRs inherit an unknown judge-calibration error.
• Judge prompts are internally contradictory. BC/HGB rule 1 says names must be "identical" but the worked example accepts "Expense" for "Pro Expense". CF opens with literal "same as" matching but immediately switches to "semantically affirms most of the facts". The FA user template asks whether the prediction covers "all" functionalities while the FA rubric is majority-coverage. These will silently bias the corresponding columns.
• Overclaims relative to the data. "Tablet mode causes the largest drop across all models" is contradicted by the Pad column being not the worst for either Gemini-3-Pro or GPT-5.1. "Memory as the absolute bottleneck across the board" is stronger than Table 8 supports (Perception's L3-4 average is nominally lower). Deriving "not model-scale limitations" from a single frontier model's drop is a causal leap not warranted by one data point.
• Metric definitions leave residual ambiguity. The symbol $N$ is reused with two different scopes in the Metric section; $K_j$ in SPR does not specify whether the Original condition is one of the counted variants; the SR/Acc denominators and the SE-vs-primary relationship (20 SE base subsets of the 149, or additional?) are not stated precisely. The output-token metric is framed as "directly reflecting API billing costs," but for vision-based agents image inputs typically dominate cost.
• Taxonomy underpinning is thinly evidenced. The paper does not report inter-annotator agreement on PUDAM labels (only "cross-validated by independent reviewers"), does not analyse the orthogonality of the five dimensions (several Appendix A example tasks legibly exercise two or three), and does not compare PUDAM against an alternative decomposition.
• Minor notation and description drift. ST vs. SE in Table 1; three vs. four listed noise types in the NR intro; "four systematic perturbations" vs. "all five settings" in the SPR discussion; A3/A4 defined in the appendix without a corresponding entry in the running summary of the Action dimension; the M4 example is a single multi-round dialogue but is described as "cross-session experience accumulation."
• Ecological validity is weaker than the framing suggests. Everything runs on Android emulators inherited from AndroidWorld; the paper's central "toward robust real-world deployment" pitch is not backed by any real-device sanity check, and the four perturbation types (language, paraphrase, dark, tablet) are a very narrow slice of realistic deployment noise.

Reproducibility & code

• Token metric vs. shipped metric. Eq. 4/5 define TT / PT over output tokens, but the released merge_char_stats.py / utils/stability_statistic.py pipeline aggregates characters from per-episode traces. No tokenizer specification is shipped, so the exact Table 5 values cannot be re-derived and cross-model comparisons become tokenizer-dependent.
• Missing agent implementations. The VenusBench-Mobile branch ships stubs for Qwen3-VL, GUI-Owl, MA3, AutoGLM, m3a_gemini3, and GPT/Claude proxies, but not for Kimi K2.5 or Seed 2.0 — both of which appear in Table 3. A reproducer must write the wrappers themselves.
• No sampling seeds. No API call or open-model inference path in the released code documents a seed, temperature or top-p. Combined with the small per-cell denominators, Table 3 / Table 4 / Table 8 are effectively single draws whose exact values are unreproducible.
• Judge model hosting not shipped. The config references a verification_model_url for Qwen3-VL-30B-A3B-Instruct; no endpoint / self-hosting recipe / sampling parameters are provided. The 90 semantically judged tasks therefore inherit user-specific judge calibration.
• Undocumented tag-to-category mapping. The shipped task_metadata.json tags sum to 147 (guibrowsing 28, func_assist 22, refusal 24, vague 18, noise_resist 16, GSA 10, multiround 10, browsecomplike 10, GUIM 9) — not 149 — and don't match the paper's per-category counts (e.g., HGB 34 vs. guibrowsing 28). Reproducing Figure 3 requires guessing how tasks were re-bucketed.
• External browsing assets are Google-Drive dependencies. HGB / D3 examples reference files under files_gui_browsing/… that the release routes off-repo; a GitHub-only checkout cannot run those tasks.
• Figure 4 P L4 count differs (10 in metadata, 7 in figure). The heatmap can be reproduced from task_metadata.json exactly for U/D/A/M but the P4 cell differs; the paper does not indicate that any tasks are excluded from the visualisation.
• Figures 1 and 6 have no plotting scripts. Neither the SR vs. model-size line chart / level-score distribution in Figure 1 nor the PUDAM radar in Figure 6 has shipped code, and the "capability level score" aggregation used by Figure 1's right panel is not defined in text.
• utils/pudam_stats.py has empty input/output paths and no example invocation. Even after a full run produces results.csv, the script's expected schema is undocumented.
• AndroidWorld baseline for the headline "~50 pp" drop is not shipped. The paper does not state whether the AndroidWorld side re-ran each model in the same scaffolding as VenusBench-Mobile or used prior published numbers; the comparison bundles agent-framework differences with benchmark difficulty.
• MA3 framework configuration is undocumented. run_ma3.py / infer_ma3.py are present but the planner/critic prompts, iteration cap, and summarisation budget that drive the reported 1.64M / 438.7 per-step figures are not specified.

Recommended Changes

Essential

• Reconcile the 149 / 80 headline counts with Table 6 per-app totals. Re-audit Table 6 or the primary/subset totals so that the per-app columns sum to the numbers used everywhere else, and align Figure 3, the "90 of 149" MLLM-judged fraction, and the SR denominators.
• Report per-seed variance and confidence intervals. Re-run every model in Table 3, Table 4 and Table 8 with at least 3 seeds (or documented decoding settings + repeat trials) and report either standard error or bootstrap confidence intervals per cell, so that ranking claims (Kimi vs. UI-Venus-72B; MA3 vs. GUI-Owl-7B; "best SPR = 15%"; ±3 pp Table 8 cells) are supported at the reported precision.
• Add a control that isolates planner from grounder for the closed-source rows. Report closed-source models with a weaker grounder (or with each model as its own grounder), and UI-Venus-72B planner + UI-Venus-72B grounder as the matched control, so the reader can attribute the 36.9 / 26.9 / 18.1 totals to planning capability rather than the shared UI-Venus-72B grounding executor.
• Validate the MLLM-Judge. Report human-vs-judge agreement (Cohen's $\kappa$) on a sampled subset of each of the seven judge interfaces, and run a robustness ablation using a judge from a different family (e.g. GPT-4o) on at least the FA, CF and BC categories — the 90/149 SR depends on it.
• Fix the internally contradictory judge prompts. For BC/HGB rule 1, either loosen "identical" to accommodate the "Expense"/"Pro Expense" example, or drop that example. For CF, remove the leading "same as / focus on values" sentence that contradicts the semantic-affirmation criterion. For FA, change the user-template question from "all functionalities" to "the majority of functionalities" to match the rubric.
• Ship the token pipeline and sampling settings. Replace merge_char_stats.py's character counter with a tokenizer-aware counter (documenting which tokenizer maps to which agent) that reproduces Table 5, and add seed/temperature/top-p defaults to every agent config so Table 3/4/5/8 can be regenerated.
• Ship the missing agent implementations. Add wrappers for Kimi K2.5 and Seed 2.0 (or explicitly document the API a wrapper must satisfy), so the two Table 3 rows are reproducible.

Suggested

• Soften the tablet-mode and Memory-bottleneck headlines to match the data. Rephrase "largest performance drops across all models" as "the modal worst-case perturbation, particularly for open-source agents," and rephrase Memory as "co-lowest with Perception at L3-4, with the steepest open-source drop" rather than "the absolute bottleneck across the board."
• Drop the causal claim about Gemini's single-model drop. Either remove "validating that these represent fundamental bottlenecks rather than model-scale limitations" or add a scaling-law-style ablation across at least three sizes of the same family.
• Clarify the metric definitions. Give the two $N$s distinct symbols; state whether $K_j = 4$ or $K_j = 5$ in the SPR formula and align Table 4's caption; state explicitly whether the 20 SE base tasks are drawn from or additional to the 149 primary tasks; and either add an image/input-token accounting to Table 5 or drop the "directly reflect API billing costs" claim.
• Report PUDAM annotation reliability. State how many annotators, provide a Cohen's $\kappa$ (or comparable) inter-annotator statistic, and describe the disagreement-resolution procedure so Table 8 / Figure 6 can be trusted as diagnostic.
• Ship the tag-to-category mapping and reconcile the 147 vs. 149 discrepancy. Include a small script that reproduces Figure 3 from task_metadata.json, and add the mapping (or extra tags) needed to bring the tag totals to 149. Do the same for the Figure 4 P L4 mismatch (10 vs. 7).
• Ship plotting code for Figures 1 and 6 and define the "capability level score." Add scripts that regenerate both figures from the aggregated outputs of pudam_stats.py, and give the formula used by Figure 1's right panel.
• Mirror the browsing assets in-tree. Move files_gui_browsing/… (or a pinned archive with a documented hash) into the repository so HGB / D3 tasks can be run from a GitHub checkout without Google Drive.
• Document the AndroidWorld baseline in the "~50 pp" comparison. State whether the AndroidWorld side was re-run with the same agent scaffolding and MLLM-Judge, or ship those matched-configuration numbers alongside the VenusBench-Mobile results.
• Fix the minor notation / description drift. Unify SE / ST across Table 1 and the taxonomy; align the NR intro's three-item list with the four formal noise types; add A3/A4 to the running Action-dimension summary; and either replace the M4 example with a genuinely cross-session case or rewrite the M4 definition to cover within-conversation history.
• Add a real-device sanity check or state the emulator-only limitation explicitly. A small subset of tasks run on a physical device (or a documented note that emulator generalisation is out of scope) would let readers calibrate the "toward robust real-world deployment" framing.
• Fill in the MA3 configuration and the pudam_stats.py inputs. Document MA3's planner/critic prompts and iteration cap so the 3.0× overhead figure is reproducible, and add example paths and expected results.csv schema to pudam_stats.py.