λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics

• URL: http://arxiv.org/abs/2412.05313v5
• Date: Mon, 03 Feb 2025 18:54:17 GMT
• Title: λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics
• Authors: Ahmed Jaafar, Shreyas Sundara Raman, Yichen Wei, Sudarshan Harithas, Sofia Juliani, Anneke Wernerfelt, Benedict Quartey, Ifrah Idrees, Jason Xinyu Liu, Stefanie Tellex,
• Abstract summary: We benchmark several models, including learning-based models and a neuro-symbolic modular approach combining foundation models with task and motion planning. Findings highlight the need for more data-efficient learning-based MoMa approaches.
• Score: 11.901933884058021
• License:
• Abstract: Efficiently learning and executing long-horizon mobile manipulation (MoMa) tasks is crucial for advancing robotics in household and workplace settings. However, current MoMa models are data-inefficient, underscoring the need for improved models that require realistic-sized benchmarks to evaluate their efficiency, which do not exist. To address this, we introduce the LAMBDA ({\lambda}) benchmark (Long-horizon Actions for Mobile-manipulation Benchmarking of Directed Activities), which evaluates the data efficiency of models on language-conditioned, long-horizon, multi-room, multi-floor, pick-and-place tasks using a dataset of manageable size, more feasible for collection. The benchmark includes 571 human-collected demonstrations that provide realism and diversity in simulated and real-world settings. Unlike planner-generated data, these trajectories offer natural variability and replay-verifiability, ensuring robust learning and evaluation. We benchmark several models, including learning-based models and a neuro-symbolic modular approach combining foundation models with task and motion planning. Learning-based models show suboptimal success rates, even when leveraging pretrained weights, underscoring significant data inefficiencies. However, the neuro-symbolic approach performs significantly better while being more data efficient. Findings highlight the need for more data-efficient learning-based MoMa approaches. {\lambda} addresses this gap by serving as a key benchmark for evaluating the data efficiency of those future models in handling household robotics tasks.

Related papers

• Error-driven Data-efficient Large Multimodal Model Tuning [35.20400815089843]
  Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks. We propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks.
  arXiv  Detail & Related papers  (2024-12-20T08:07:11Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Active Exploration in Bayesian Model-based Reinforcement Learning for Robot Manipulation [8.940998315746684]
  We propose a model-based reinforcement learning (RL) approach for robotic arm end-tasks. We employ Bayesian neural network models to represent, in a probabilistic way, both the belief and information encoded in the dynamic model during exploration. Our experiments show the advantages of our Bayesian model-based RL approach, with similar quality in the results than relevant alternatives.
  arXiv  Detail & Related papers  (2024-04-02T11:44:37Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Value function estimation using conditional diffusion models for control [62.27184818047923]
  We propose a simple algorithm called Diffused Value Function (DVF) It learns a joint multi-step model of the environment-robot interaction dynamics using a diffusion model. We show how DVF can be used to efficiently capture the state visitation measure for multiple controllers.
  arXiv  Detail & Related papers  (2023-06-09T18:40:55Z)
• Temporal Difference Learning for Model Predictive Control [29.217382374051347]
  Data-driven model predictive control has two key advantages over model-free methods. TD-MPC achieves superior sample efficiency and performance over prior work on both state and image-based continuous control tasks.
  arXiv  Detail & Related papers  (2022-03-09T18:58:28Z)
• Model-Agnostic Multitask Fine-tuning for Few-shot Vision-Language Transfer Learning [59.38343286807997]
  We propose Model-Agnostic Multitask Fine-tuning (MAMF) for vision-language models on unseen tasks. Compared with model-agnostic meta-learning (MAML), MAMF discards the bi-level optimization and uses only first-order gradients. We show that MAMF consistently outperforms the classical fine-tuning method for few-shot transfer learning on five benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-09T17:26:53Z)
• Can Deep Learning be Applied to Model-Based Multi-Object Tracking? [25.464269324261636]
  Multi-object tracking (MOT) is the problem of tracking the state of an unknown and time-varying number of objects using noisy measurements. Deep learning (DL) has been increasingly used in MOT for improving tracking performance. In this paper, we propose a Transformer-based DL tracker and evaluate its performance in the model-based setting.
  arXiv  Detail & Related papers  (2022-02-16T07:43:08Z)
• Model-Based Visual Planning with Self-Supervised Functional Distances [104.83979811803466]
  We present a self-supervised method for model-based visual goal reaching. Our approach learns entirely using offline, unlabeled data. We find that this approach substantially outperforms both model-free and model-based prior methods.
  arXiv  Detail & Related papers  (2020-12-30T23:59:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.