RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policies

Summary

RoboMME is a simulation benchmark for memory-augmented robotic manipulation, organized around a cognitive-science-motivated taxonomy of four memory types — temporal, spatial, object, and procedural — that map to four task suites (Counting, Permanence, Reference, Imitation). It contains 16 non-Markovian tasks with 1,600 demonstrations (770k timesteps), and is accompanied by a family of 14 memory-augmented VLA policies (the MME-VLA suite) built on the π0.5 backbone that cross three memory representations (symbolic language subgoals, perceptual visual tokens with either token dropping or frame sampling, and recurrent state via TTT or RMT) with three integration mechanisms (memory-as-context, memory-as-modulator, memory-as-expert). The core conceptual move is to hold the backbone and evaluation protocol fixed while sweeping memory representations and integration strategies, so that observed differences reflect memory-design choices rather than backbone artifacts. This is a valuable contribution: prior memory-augmented VLA work is fragmented across incomparable evaluation suites, and RoboMME plausibly becomes the standard testbed for future comparisons in this subarea.

The main empirical finding is that no single memory design dominates: perceptual memory with modulator integration (FRAMESAMP+Modul) achieves the highest average success (44.51%), symbolic memory does well on counting/short-horizon reasoning, and recurrent variants trail badly. A parallel human study (90.5% under an oracle low-level planner) and a small real-robot study (four tasks, 10 trials each) are used to argue that the benchmark is hard for humans and that simulation trends transfer.

The main conceptual limitation is that "no single representation dominates" is largely a negative result, and the constructive story around it is fragile: the best MME-VLA variant is only ~2 points above a prior baseline (MemER) at unequal run counts, recurrent methods are trained at a 4× smaller batch size, StopCube's near-universal failure may be an action-chunking artifact rather than a memory issue, and OpenVLA-OFT experiments hint that the ordering is partly backbone-specific. Structural facts about the benchmark (16 tasks, 4 suites of 4, 1,600 demonstrations) reproduce exactly, and the one qualitative signal our execution attempt did produce agrees in direction with the paper's central thesis that memory helps on RoboMME. The benchmark itself is the durable contribution; the mechanistic claims need sharper controls, and the release needs work before the headline numbers are reproducible end-to-end.

Strengths

• Conceptual contribution. Grounding the taxonomy in the Atkinson–Shiffrin framework and mapping each memory type to a task suite gives the benchmark a principled scaffold, and the explicit non-Markovian construction (identical observations from different histories requiring different actions) is exactly what has been missing from prior manipulation benchmarks.
• Standardization of memory-augmented VLA. Holding the π0.5 backbone fixed while sweeping 3 memory representations × 3 integration mechanisms and reporting all 14 variants under a single evaluation protocol is the kind of controlled comparison that this literature has needed.
• Scale and diversity. 16 tasks × 100 demonstrations × long horizons (up to 1,134 average steps for VideoPlaceOrder), plus explicit stratification into Easy/Medium/Hard configurations for most tasks, produces a testbed with real room for future improvement.
• Complementary evaluation modes. Combining the main suite with per-task-suite averages, a functional-category regrouping (Motion-Centric, Time-Sensitive, etc.), oracle-planner human evaluation, foundation-model VideoQA baselines, and a real-robot cross-check gives readers many angles on the same claims.
• Reasonable variance reporting. The 3 seeds × 3 checkpoints averaging protocol for MME-VLA variants is more careful than typical robot-learning practice and enables ± reporting in Tables 12 and 15.
• Structural reproducibility. The shipped benchmark environment loads cleanly, the 16-task / 4-suite partition and the 1,600-demonstration split reproduce exactly, and the demonstration replayer and DummyModel scoring harness run without code changes — a solid substrate for others to build on.
• Released code. Multiple public repositories cover the MME-VLA policies, the benchmark environment, the SAM2Act, OpenVLA-OFT, and MemoryVLA adaptations, and a Diffusion Policy baseline — a good starting surface for others to extend.

Weaknesses

• Headline "best" claim rests on unequal-precision estimates. FRAMESAMP+Modul (44.51 ± 0.77, 9 runs) vs MemER (42.38 ± 0.33, 3 runs) is close enough that the ranking is not clearly outside noise, and the paper reports no significance test. Since this comparison is central to the "perceptual memory performs best" finding, the current claim overreaches. The near-tie also complicates the narrative that MME-VLA designs beat prior methods.
• Recurrent-variant conclusions confounded by training budget. Symbolic and perceptual variants train at batch size 64 while TTT/RMT use batch size 16, a 4× gap, without matched-compute controls. The paper attributes recurrent underperformance to "shallow recurrent layers" and needing "stronger recurrence-oriented pretraining," but the batch-size mismatch is at least as plausible and directly weakens the recurrent-vs-non-recurrent comparison. The RMT variant additionally drops the MLP sublayer "for simplicity" and applies supervision only at the last step (vs the last eight for TTT), stacking two more asymmetries on the same comparison.
• StopCube collapse likely a control-granularity artifact. Almost every learned method scores near-zero on StopCube (even GROUNDSG+Oracle only reaches 49.67, and humans only 78). The training setup uses 20-step chunks with the first 16 executed — a granularity that could make single-timestep timing failures dominate regardless of memory design. Treating StopCube results as diagnostic of memory limitations conflates a benchmark constraint with the phenomenon of interest.
• Fair-comparison budget is really within-neural-memory only. The 512-token budget is imposed on perceptual and recurrent variants, but symbolic memory is capped only by the 512 language-token prompt length, and even for neural memory the observation-side "512" tokens are unpooled while memory tokens are pooled aggressively. Per-token information content is not comparable across representations; the fairness rhetoric attached to the 512 budget is stronger than what the control equalizes.
• Task-cognitive-dimension inference in Q4 is weaker than the sentence implies. "No method fully solves an entire task suite" is a claim about ceilings; "no one-to-one correspondence between cognitive memory types and model representations" is a claim about relative ordering. The former does not entail the latter, and the paper's regrouping into functional categories does not close the gap.
• Backbone-generality is undercut by the OpenVLA-OFT appendix. OpenVLA-OFT results (perceptual worse, symbolic better relative to π0.5) actually invert the main-body ordering of memory representations. This bounds the generality of "perceptual memory performs best" and deserves more than an appendix mention.
• Real-robot claims are under-powered. 40 total trials across 4 tasks (10/task), a memory budget of B=2048 extrapolated beyond the simulation-studied range (max 1024), and no per-trial variance make the "similar trends" statement qualitative at best.
• Human-study interpretation. A 78% StopCube score under a "flawless low-level oracle" is not clearly reconciled: the mechanism by which timing failures occur when the human only selects high-level actions is not explained. This weakens the human-upper-bound interpretation.
• Motivating prior-benchmark claim overreaches. The statement that decisions on RLBench/CALVIN/LIBERO/VLABench/RoboCerebra "can typically be made from immediate observations alone" is a load-bearing motivation for RoboMME, but is asserted uniformly for a set that includes RoboCerebra (marked Dynamic, ~2,972 avg steps). Needs empirical support or restriction.
• Wrong citation for MemoryVLA. In Appendix B.5, MemoryVLA is cited as Peller-Konrad et al. 2023 (an ArmarX cognitive-architecture paper), while elsewhere it is correctly cited as Shi et al. 2026. The surrounding chronology is impossible (Peller-Konrad 2023 cannot build on CogACT 2024a).
• Several method sub-choices are logically inconsistent with results. SAM2Act+ is said to be "sufficient for the remaining tasks" outside StopCube/InsertPeg, but it also scores 0.00 on PatternLock and RouteStick. The claim "gains become smaller beyond 512 tokens" is contradicted by TOKENDROP+Modul's marginal gain being larger at 512→1024 than at 256→512. The attention-attribution "sparse but crucial" story is drawn from four rows with a non-monotone max-attention series.
• Minor internal inconsistencies. Task names differ across sections (PatternLock vs DrawPattern, MoveCube's hooking implement called a "peg tool" here vs "hook stick" in the asset list), grounding-omission rationale covers PutFruits but is later applied to DrawPattern as well, TOKENDROP+Modul is 38.02 in Table 16 vs 38.04 in Tables 12/18, and the Short-Horizon average is bolded as exactly 48.22 for two structurally different methods with different run counts. Individually small, collectively worth polishing.

Reproducibility & code

The paper releases multiple repositories (robomme_policy_learning, robomme_benchmark, SAM2Act adaptation, OpenVLA-OFT adaptation, MemoryVLA adaptation, and a DP baseline) along with hyperparameters, prompts, and task specifications, so headline claims are broadly reproducible in principle. Under an execution attempt on a resource-constrained host, however, several concrete gaps surfaced — some environmental, several rooted in what the release actually ships.

• What reproduced. The benchmark environment loaded cleanly and the taxonomy verified exactly: BenchmarkEnvBuilder exposes 16 tasks that partition without overlap into the 4 documented suites of 4; per-task env_metadata files sum to exactly 100 train / 50 val / 50 test episodes each, matching the 1,600-demonstration claim. Demonstration replay executed to success across the tasks that could be decompressed, and the scoring harness ran without code changes. On the one policy comparison our attempt could actually run, FRAMESAMP+Modul reached 1.0 vs π0.5 baseline 0.25 on a 2-task × 2-episode subset (MoveCube, StopCube) — directionally consistent with the paper's central thesis that memory helps on RoboMME, but far too small a slice to check the specific 44.51%.
• What did not reproduce (and why). The 16-task × 50-episode × 14-variant sweep behind Table 3 could not be assembled inside the wall-clock budget: only 2 of the 14 MME-VLA variants and none of the 4 prior baselines were run to completion, so headline scalars (FRAMESAMP+Modul 44.51%, MemER 42.38%, GROUNDSG+QwenVL 32.70%, π0.5 17.93%, GROUNDSG+Oracle 84.08%) were not independently checked. Volumetric dataset totals (770k timesteps, ~481 steps/episode) came out low at 377k timesteps and 343 steps/episode because 5 of 16 tasks — exactly the longest-horizon ones (BinFill, PickXTimes, VideoRepick, VideoPlaceButton, VideoPlaceOrder) — could not be decompressed on the reproduction host; the structural counts nonetheless matched, so the shortfall is environmental, not a paper discrepancy.
• Serving-layer fragility (a real defect). The shipped policy server calls a blocking policy.infer(obs) inline in the asyncio event loop, so the ~2-minute first-inference XLA compile starves keepalive pings and the client tears the connection down before any episode completes. Getting any rollouts at all required patching the server to disable keepalive. On slow-compile or single-core hosts the code hangs silently and produces zero results out of the box — a runnable but fragile release.
• Memory-budget ablation is not code-reproducible from the release. Table 16's ablation requires evaluating each perceptual variant at B ∈ {64, 128, 256, 512, 1024}, but only the budget=512 checkpoint per variant is published on HuggingFace, and inspection shows budget is a train-time hyperparameter baked into checkpoint parameter shapes (static_image_emb.shape[1] == config.budget is asserted at eval time). Reproducing the other budgets requires retraining, which the artifact does not enable.
• Analysis scripts are absent from the release. The attention-allocation study (Table 17), the modulation-statistics study (Table 18), and the TFLOPs efficiency comparison (Figure 4) have no corresponding scripts in the released policy repo; a reader must re-author them against model internals from scratch. This makes both the specific values and the aggregation semantics unverifiable.
• Under-documented methodology for reproduction. Even inside the released code, several load-bearing details are not specified: the 5% noise distribution and application protocol in data generation; TOKENDROP boundary rules under the 512-token cap (first-frame preservation, saliency over/under-quota); whether zero-padded memory positions are attention-masked in each integration mechanism; whether "M-RoPE" is a true rotary embedding or a 768-dim concatenated positional feature; whether the shared 1024-dim projection is skipped or re-expanded for memory-as-context; and the exact aggregation for the attention-allocation table.
• Foundation-model VideoQA is effectively non-reproducible over time. Gemini-2.5-Pro and GPT-4o API versions/dates, the actual VideoQA prompt template used for Table 11, and sampling parameters are not documented; both endpoints evolve behind their names.
• Human study protocol under-documented. The 90.5% headline is single-point; no participant demographics, recruitment, familiarization procedure, per-task allocation, inter-participant agreement, or per-participant variance is reported. The GUI code shown in Figure 8 does not appear in the releases.
• Assembly burden. Reproducing anything end-to-end requires cloning and building five external repositories plus multi-GB HuggingFace checkpoints; the benchmark repo itself contains no model or evaluation code. This is documented but heavy, and would benefit from a single top-level driver script and a scoped subset-reproduction target.

Recommended Changes

Essential

• Match run counts and add a significance test for the headline "best" claim. Re-run MemER (and, ideally, π0.5 w/ past actions and SAM2Act+) at the same 3-seed × 3-checkpoint protocol used for MME-VLA and report a paired significance test between FRAMESAMP+Modul and MemER on overall AVG. If the difference is not significant, soften "achieves the highest overall success rate" to reflect the tie.
• Control for the batch-size (and MLP / supervision) confounds in the recurrent-vs-non-recurrent comparison. Add matched-batch (or matched-tokens-seen) TTT/RMT runs and an RMT ablation restoring the MLP block; alternatively, explicitly caveat that the "recurrent is worst" conclusion is conditional on the 16-vs-64 batch-size gap, the MLP-free RMT, and the asymmetric last-step vs last-eight-step supervision.
• Separate action-parameterization from memory for StopCube. Add either a smaller-chunk-size ablation (e.g., chunk 4 or 1) or a per-step control-granularity study on StopCube, and acknowledge in Q2 that the near-universal collapse partly reflects the 20/16-step action-chunk setup, not just memory-representation weakness.
• Fix the MemoryVLA citation in Appendix B.5. Replace "MemoryVLA (Peller-Konrad et al., 2023)" with the Shi et al. 2026 citation used in Tables 15 and elsewhere, and correct the surrounding chronology so the reference to CogACT (Li et al., 2024a) is temporally coherent.
• Rescope the "fair comparison" budget statement. Clarify that the 512-token budget equalizes only the perceptual and recurrent (neural) variants and does not equalize per-token information density across memory representations; then either add a matched language-token budget for symbolic variants or drop the cross-representation "fair" framing.
• Add methodology detail so that headline numbers are reproducible from the release. Specify (i) TOKENDROP's boundary rules for first-frame preservation and over/under-quota selection; (ii) whether zero-padded memory positions are masked in every integration mechanism; (iii) the 5% noise distribution/scale used in data generation; (iv) the M-RoPE concatenation vs rotation semantics; (v) whether the shared 1024-dim projection is re-expanded or skipped for memory-as-context; and (vi) exact aggregation for the attention-allocation and modulation-statistics analyses.
• Fix the serving-layer hang. Move policy.infer(obs) off the asyncio event loop (e.g., asyncio.to_thread) or ship a keepalive-off configuration by default, so the released code produces rollouts on slow-compile hosts without patching.
• Release the missing artifacts needed for the ablations. Publish per-budget checkpoints for the memory-budget ablation (or expose budget as a first-class command-line flag in a training recipe), and add scripts for the attention-allocation table, modulation-statistics table, and TFLOPs figure with a documented aggregation formula.
• Document foundation-model VideoQA reproducibility. Add API model IDs/dates for Gemini-2.5-Pro and GPT-4o, the prompt template used for the oracle-planner protocol, and sampling parameters, so Table 11 is verifiable at a later time.

Suggested

• Reframe Q4's inference. Replace the "no method fully solves an entire task suite → no one-to-one correspondence" reasoning with a per-task-suite ranking argument that shows the memory representation matched to each cognitive type is not consistently the best on the corresponding suite.
• Engage with the OpenVLA-OFT inversion. Move the OpenVLA-OFT result out of a footnote-style appendix paragraph and use it to explicitly bound the generality of the "perceptual memory performs best" conclusion.
• Expand or scope-down the real-world story. Either report more trials per task with per-seed variance and a memory budget matching the simulation ablation range, or explicitly present the real-robot section as a qualitative demonstration rather than empirical confirmation of the simulation ranking; also justify (or ablate) the B=2048 extrapolation.
• Clarify the human-study protocol. Add participant demographics/recruitment/compensation, per-task allocation, familiarization procedure, and per-participant variance; explain the mechanism by which timing failures arise on StopCube under a "flawless" low-level oracle, or restate that oracle to acknowledge residual timing uncertainty.
• Fix the SAM2Act+ "discrete actions are sufficient" claim. Include PatternLock and RouteStick in the enumeration of tasks affected by discrete action parameterization, or replace "sufficient" with a more precise scoping.
• Support the prior-benchmark motivation. Either restrict the "immediate observations alone" claim to the specific benchmarks it holds for (excluding at least RoboCerebra) or cite empirical evidence that memory-free baselines saturate the listed suites.
• Standardize task and asset naming. Use one name consistently for PatternLock/DrawPattern, standardize the hooking implement to "hook stick," and update the grounding-omission rationale to cover both PutFruits and DrawPattern.
• Reconcile small table inconsistencies. Verify the SIMPLESG-vs-GROUNDSG Oracle BinFill value 85.78 (identical to two decimals), the Short-Horizon 48.22 tie between GROUNDSG+QwenVL and MemER, and the 38.02-vs-38.04 TOKENDROP+Modul mismatch between Tables 12/18 and Table 16.
• Soften over-read statistical claims. Soften "gains become smaller beyond 512 tokens" to match the TOKENDROP+Modul curve, and soften the attention-attribution conclusion in Section C.5 to reflect the four-datapoint, non-monotone underlying data.
• Add an InsertPeg configuration table for parity with the other tasks, and clarify the direction/skipping failure criteria for RouteStick and PatternLock.
• Reduce the reproduction assembly burden. Ship a single top-level driver that stitches together the five external repositories and provide a scoped subset-reproduction target for reviewers with limited compute.