EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies

• URL: http://arxiv.org/abs/2606.18239v2
• Date: Sat, 20 Jun 2026 06:00:15 GMT
• Title: EBench: Elemental Diagnosis of Generalist Mobile Manipulation Policies
• Abstract summary: We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies.<n>EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions.
• Score: 92.63011025295123
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present EBench, a simulation benchmark that diagnoses generalist mobile manipulation policies beyond a single success-rate scalar. EBench comprises 26 diverse and challenging manipulation tasks annotated along 5 capability dimensions and 4 generalization dimensions. We evaluate state-of-the-art generalist manipulation models including $π_0$, $π_{0.5}$, XVLA, and InternVLA-A1, and reveal that models with near success rates exhibit strikingly different capability profiles: $π_{0.5}$ achieves the highest test success rate and the best train--test retention, whereas InternVLA-A1 dominates mobile manipulation but collapses on dexterous tasks, and XVLA exhibits strengths on a disjoint set of atomic skills compared to other policies. Beyond capability profiling, EBench analyzes the generalization ability from 4 representative perspectives, identifying the impact of different distribution shift factors. The results reveal strengths and weaknesses of models behind an overall score. We hope this benchmark offers a broad set of diagnostic signals to guide iteration on generalist manipulation models.

Related papers

• X-DiffVLA: X-Embodied Diffusion Action Heads for Vision-Language-Action Models [39.033717938466246]
  We propose X-DiffVLA, a diffusion-based VLA model featuring a unified cross-embodied action head.<n>X-DiffVLA can leverage the generative strengths of diffusion models to capture both the diversity and latent correlations in cross-embodied datasets.<n>We show that X-DiffVLA achieves state-of-the-art performance, with improvements of 15.3% and 12.5%, respectively.
  arXiv  Detail & Related papers  (2026-05-24T12:41:34Z)
• Attributing Emergence in Million-Agent Systems [68.53670424791751]
  Large language models (LLMs) can simulate human-like reasoning and decision-making in individual agents.<n>Such studies require attributing macro emergence to individual agents.<n>We adapt Aumann--Shapley path-integral attribution to LLM-powered MAS at million-agent scale.
  arXiv  Detail & Related papers  (2026-05-12T01:49:41Z)
• Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models [4.145971099162064]
  Large Language Models (LLMs) excel at human-like language generation but often embed and amplify implicit, intersectional biases.<n>Existing bias audits rely on static, embedding-based tests (CEAT, I-WEAT, I-SEAT) that quantify absolute association strengths.<n>We introduce the Bias Amplification Differential and Explainability Score (BADx): a novel, scalable metric that measures persona-induced bias amplification.
  arXiv  Detail & Related papers  (2026-03-16T15:57:12Z)
• From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis [7.3390265169910665]
  Hallucinations in Large Language Models (LLMs) represent a core obstacle to their reliable deployment in critical domains.<n>To address this limitation, a new research paradigm is proposed, shifting from "detection" to "diagnosis"
  arXiv  Detail & Related papers  (2025-12-31T05:06:35Z)
• Evaluation Awareness Scales Predictably in Open-Weights Large Language Models [8.165127822088499]
  We show that evaluation awareness increases predictably with model size.<n>This scaling law enables forecasting deceptive behavior in future larger models and guides the design of scale-aware evaluation strategies for AI safety.
  arXiv  Detail & Related papers  (2025-09-10T06:36:38Z)
• GRAPE: Generalizing Robot Policy via Preference Alignment [58.419992317452376]
  We present GRAPE: Generalizing Robot Policy via Preference Alignment.<n>We show GRAPE increases success rates on in-domain and unseen manipulation tasks by 51.79% and 58.20%, respectively.<n> GRAPE can be aligned with various objectives, such as safety and efficiency, reducing collision rates by 37.44% and rollout step-length by 11.15%, respectively.
  arXiv  Detail & Related papers  (2024-11-28T18:30:10Z)
• xLAM: A Family of Large Action Models to Empower AI Agent Systems [111.5719694445345]
  We release xLAM, a series of large action models designed for AI agent tasks. xLAM consistently delivers exceptional performance across multiple agent ability benchmarks.
  arXiv  Detail & Related papers  (2024-09-05T03:22:22Z)
• How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation [90.93999543169296]
  GPT-4V acts as the most advanced publicly accessible multimodal foundation model. This study rigorously evaluates GPT-4V's adaptability and generalization capabilities in dynamic environments.
  arXiv  Detail & Related papers  (2023-12-12T16:48:07Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.