Human-centered In-building Embodied Delivery Benchmark

• URL: http://arxiv.org/abs/2406.17898v1
• Date: Tue, 25 Jun 2024 19:19:10 GMT
• Title: Human-centered In-building Embodied Delivery Benchmark
• Authors: Zhuoqun Xu, Yang Liu, Xiaoqi Li, Jiyao Zhang, Hao Dong,
• Abstract summary: In this work, we propose a specific commercial scenario simulation, human-centered in-building embodied delivery. We have developed a brand-new virtual environment system from scratch, constructing a multi-level connected building space modeled after a polar research station. This environment also includes autonomous human characters and robots with grasping and mobility capabilities.
• Score: 8.079480672302424
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recently, the concept of embodied intelligence has been widely accepted and popularized, leading people to naturally consider the potential for commercialization in this field. In this work, we propose a specific commercial scenario simulation, human-centered in-building embodied delivery. Furthermore, for this scenario, we have developed a brand-new virtual environment system from scratch, constructing a multi-level connected building space modeled after a polar research station. This environment also includes autonomous human characters and robots with grasping and mobility capabilities, as well as a large number of interactive items. Based on this environment, we have built a delivery dataset containing 13k language instructions to guide robots in providing services. We simulate human behavior through human characters and sample their various needs in daily life. Finally, we proposed a method centered around a large multimodal model to serve as the baseline system for this dataset. Compared to past embodied data work, our work focuses on a virtual environment centered around human-robot interaction for commercial scenarios. We believe this will bring new perspectives and exploration angles to the embodied community.

Related papers

• Humanoid Occupancy: Enabling A Generalized Multimodal Occupancy Perception System on Humanoid Robots [50.0783429451902]
  Humanoid robot technology is advancing rapidly, with manufacturers introducing diverse visual perception modules tailored to specific scenarios.<n> occupancy-based representation has become widely recognized as particularly suitable for humanoid robots, as it provides both rich semantic and 3D geometric information essential for comprehensive environmental understanding.<n>We present Humanoid Occupancy, a generalized multimodal occupancy perception system that integrates hardware and software components, data acquisition devices, and a dedicated annotation pipeline.
  arXiv  Detail & Related papers  (2025-07-27T10:47:00Z)
• Grounding Robot Policies with Visuomotor Language Guidance [15.774237279917594]
  We propose an agent-based framework for grounding robot policies to the current context. The proposed framework is composed of a set of conversational agents designed for specific roles. We demonstrate that our approach can effectively guide manipulation policies to achieve significantly higher success rates.
  arXiv  Detail & Related papers  (2024-10-09T02:00:37Z)
• RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots [25.650235551519952]
  We present RoboCasa, a large-scale simulation framework for training generalist robots in everyday environments. We provide thousands of 3D assets across over 150 object categories and dozens of interactable furniture and appliances. Our experiments show a clear scaling trend in using synthetically generated robot data for large-scale imitation learning.
  arXiv  Detail & Related papers  (2024-06-04T17:41:31Z)
• Agent AI: Surveying the Horizons of Multimodal Interaction [83.18367129924997]
  "Agent AI" is a class of interactive systems that can perceive visual stimuli, language inputs, and other environmentally-grounded data. We envision a future where people can easily create any virtual reality or simulated scene and interact with agents embodied within the virtual environment.
  arXiv  Detail & Related papers  (2024-01-07T19:11:18Z)
• Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots [119.55240471433302]
  Habitat 3.0 is a simulation platform for studying collaborative human-robot tasks in home environments. It addresses challenges in modeling complex deformable bodies and diversity in appearance and motion. Human-in-the-loop infrastructure enables real human interaction with simulated robots via mouse/keyboard or a VR interface.
  arXiv  Detail & Related papers  (2023-10-19T17:29:17Z)
• HandMeThat: Human-Robot Communication in Physical and Social Environments [73.91355172754717]
  HandMeThat is a benchmark for a holistic evaluation of instruction understanding and following in physical and social environments. HandMeThat contains 10,000 episodes of human-robot interactions. We show that both offline and online reinforcement learning algorithms perform poorly on HandMeThat.
  arXiv  Detail & Related papers  (2023-10-05T16:14:46Z)
• Semantic-Aware Environment Perception for Mobile Human-Robot Interaction [2.309914459672557]
  We present a vision-based system for mobile robots to enable a semantic-aware environment without additional a-priori knowledge. We deploy our system on a mobile humanoid robot that enables us to test our methods in real-world applications.
  arXiv  Detail & Related papers  (2022-11-07T08:49:45Z)
• BEHAVE: Dataset and Method for Tracking Human Object Interactions [105.77368488612704]
  We present the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup.
  arXiv  Detail & Related papers  (2022-04-14T13:21:19Z)
• Spatial Computing and Intuitive Interaction: Bringing Mixed Reality and Robotics Together [68.44697646919515]
  This paper presents several human-robot systems that utilize spatial computing to enable novel robot use cases. The combination of spatial computing and egocentric sensing on mixed reality devices enables them to capture and understand human actions and translate these to actions with spatial meaning.
  arXiv  Detail & Related papers  (2022-02-03T10:04:26Z)
• HARPS: An Online POMDP Framework for Human-Assisted Robotic Planning and Sensing [1.3678064890824186]
  The Human Assisted Robotic Planning and Sensing (HARPS) framework is presented for active semantic sensing and planning in human-robot teams. This approach lets humans opportunistically impose model structure and extend the range of semantic soft data in uncertain environments. Simulations of a UAV-enabled target search application in a large-scale partially structured environment show significant improvements in time and belief state estimates.
  arXiv  Detail & Related papers  (2021-10-20T00:41:57Z)
• Language Understanding for Field and Service Robots in a Priori Unknown Environments [29.16936249846063]
  This paper provides a novel learning framework that allows field and service robots to interpret and execute natural language instructions. We use language as a "sensor" -- inferring spatial, topological, and semantic information implicit in natural language utterances. We incorporate this distribution in a probabilistic language grounding model and infer a distribution over a symbolic representation of the robot's action space.
  arXiv  Detail & Related papers  (2021-05-21T15:13:05Z)
• Few-Shot Visual Grounding for Natural Human-Robot Interaction [0.0]
  We propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. We evaluate the performance of the proposed model on real RGB-D data collected from public scene datasets.
  arXiv  Detail & Related papers  (2021-03-17T15:24:02Z)
• Human Trajectory Forecasting in Crowds: A Deep Learning Perspective [89.4600982169]
  We present an in-depth analysis of existing deep learning-based methods for modelling social interactions. We propose two knowledge-based data-driven methods to effectively capture these social interactions. We develop a large scale interaction-centric benchmark TrajNet++, a significant yet missing component in the field of human trajectory forecasting.
  arXiv  Detail & Related papers  (2020-07-07T17:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.