ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation

🕒 Published (v1): 2026-03-30 15:06 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; comprehensive real-world benchmark for VLA manipulation evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ManipArena is a standardized real-world evaluation framework for VLA and world models comprising 20 reasoning-intensive robot manipulation tasks, 10,812 expert trajectories, and a green-screen controlled environment with synchronized Real2Sim counterparts. It enforces a single-model-for-all-tasks rule and stratified OOD evaluation to measure genuine generalization rather than per-task overfitting. It addresses the critical absence of reproducible, fair real-world benchmarks that expose the simulation-to-reality gap.

Problem

Existing robot manipulation benchmarks (RLBench, LIBERO, CALVIN, ManiSkill2) are overwhelmingly simulator-centric, failing to capture perception noise, contact dynamics, hardware constraints, and system latency present in physical deployment. Fragmented real-world evaluations use disparate platforms, making horizontal comparison unfair and results irreproducible. No prior benchmark jointly provides reasoning-oriented tasks, controlled multi-level OOD evaluation, mobile manipulation, rich sensory diagnostics, and Real2Sim parity.

Method

ManipArena is built around five interlocking design pillars:

1. Server-side inference architecture: Participants expose a single HTTP endpoint accepting camera images and proprioception; organizers run all robot trials, removing hardware barriers and ensuring identical conditions. A one-model-for-all-tasks constraint prevents task-specific overfitting.

2. Green-screen controlled environment: A self-contained chroma-key booth with fixed studio lighting eliminates uncontrolled visual confounders (background, illumination), enabling attribution of performance differences solely to designed variation axes.

3. Systematic three-level diversity: Training diversity is treated as an experimental variable via per-task diversity guides. Level 1 varies physical attributes (material, color, size); Level 2 varies spatial configuration; Level 3 varies semantic composition (permutations, category subsets). ~500 trajectories per task (~188 hrs total) follow these guides with near-uniform distribution across dimensions.

4. Stratified OOD evaluation: 10 trials per task follow a fixed structure — T1–T4 in-domain, T5–T8 appearance-shifted (within training distribution), T9–T10 semantic OOD (unseen objects). OOD objects span varying semantic distances (near-OOD to far-OOD) and compound multi-factor shifts to measure degradation curves.

5. Real2Sim synchronization: Physical workspaces reconstructed via 3D Gaussian Splatting; task objects generated with Hunyuan3D; simulation run on IsaacLab for scalable offline evaluation and sim-to-real transfer research.

Tasks span three categories: Execution Reasoning (10 tasks — how to execute: grasp strategy, force control, bimanual coordination), Semantic Reasoning (5 tasks — what to do: category/attribute/relation understanding), and Mobile Manipulation (5 tasks — long-horizon navigation + manipulation, avg. \(4.3\times\) longer episodes than tabletop, 2–3 min, 2,878 vs. 665 avg frames).

State/action dimensionality is 56D for tabletop (end-effector poses 14D, joint positions 14D, joint velocities 14D, joint currents 14D) and 62D for mobile, providing motor current as a torque proxy for contact-rich tasks.

Key Contributions

• First standardized real-world benchmark covering VLA and world model evaluation under identical hardware, lighting, and protocol conditions across 20 tasks
• 10,812 expert trajectories (~188 hours) with designed three-level diversity across 20 tasks on a single bimanual embodiment (X2Robot)
• Stratified OOD evaluation protocol enabling full generalization profiles (in-domain → near-OOD → far-OOD → compound-OOD) from a single evaluation session
• Green-screen controlled environment enabling portable, longitudinally consistent, and cross-site reproducible evaluation
• Real2Sim pipeline (3DGS scene reconstruction + Hunyuan3D object generation + IsaacLab simulation) for scalable offline evaluation
• 56/62D sensory data including motor currents and joint velocities to enable force-aware and dynamics-aware policy research
• Server-side inference architecture that lowers participation barrier and protects participant IP

Results

The paper is primarily a benchmark/framework paper; it does not report baseline model scores in the provided text. Quantitative analysis of model performance is referenced for Section 4 but that section is not included in the excerpt. The following structural facts are reported:

• 20 tasks, 10,812 total trajectories, ~188 hours of data
• Mobile tasks are on average \(4.3\times\) longer than tabletop (2,878 vs. 665 avg frames at 20fps) and constitute 60.6% of total dataset frames despite comprising only 26.7% of trajectories
• 10 trials per task scored 0–10 per sub-task; maximum 100 points per task, 1,500 points total across 15 tabletop tasks
• Training diversity guides ensure each diversity dimension is within 10–15% of uniform distribution

Limitations

• Quantitative baseline comparisons are not included in the available text excerpt; benchmark discriminability claims cannot be verified from the provided content
• Single-embodiment design (X2Robot / Quanta X1) may limit generalizability of conclusions to other robot morphologies
• Green-screen controlled environment, while enabling reproducibility, is atypical of real deployment settings; policies trained/evaluated here may not generalize to open environments
• Server-side inference assumes low-latency network connectivity; real deployment scenarios often involve edge inference with stricter latency budgets
• Motor current and joint velocity signals are provided but current VLA/world model baselines do not use them, so their utility is aspirational at benchmark launch
• Mobile manipulation evaluation uses only a single robot unit vs. four for tabletop, potentially introducing platform-level variance in mobile results

Relevance to Vision-Language Models

ManipArena directly evaluates VLA models — the dominant paradigm that couples vision-language pretrained backbones with action generation — in real-world settings that expose failure modes invisible in simulation, making it a critical diagnostic tool for the VLA research community. The semantic reasoning task category specifically stress-tests the language-grounded understanding component of VLMs (category recognition, attribute inference, relational reasoning) under novel object instances, cleanly separating linguistic generalization from motor skill. The stratified OOD design and compound-factor trials create a structured protocol for measuring whether VLA models' language understanding genuinely drives generalization or whether they are exploiting visual shortcuts — a central open question in grounded VLM research. The Real2Sim pipeline and rich sensory data further enable the community to iterate offline before costly real-robot evaluation, lowering the barrier for VLM-based robotic policy research.