BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments

• URL: http://arxiv.org/abs/2108.03332v1
• Date: Fri, 6 Aug 2021 23:36:23 GMT
• Title: BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• Authors: Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Mart\'in-Mart\'in, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, Li Fei-Fei
• Abstract summary: We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation. These activities are designed to be realistic, diverse, and complex. We include 500 human demonstrations in virtual reality (VR) to serve as the human ground truth.
• Score: 70.18430114842094
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce BEHAVIOR, a benchmark for embodied AI with 100 activities in simulation, spanning a range of everyday household chores such as cleaning, maintenance, and food preparation. These activities are designed to be realistic, diverse, and complex, aiming to reproduce the challenges that agents must face in the real world. Building such a benchmark poses three fundamental difficulties for each activity: definition (it can differ by time, place, or person), instantiation in a simulator, and evaluation. BEHAVIOR addresses these with three innovations. First, we propose an object-centric, predicate logic-based description language for expressing an activity's initial and goal conditions, enabling generation of diverse instances for any activity. Second, we identify the simulator-agnostic features required by an underlying environment to support BEHAVIOR, and demonstrate its realization in one such simulator. Third, we introduce a set of metrics to measure task progress and efficiency, absolute and relative to human demonstrators. We include 500 human demonstrations in virtual reality (VR) to serve as the human ground truth. Our experiments demonstrate that even state of the art embodied AI solutions struggle with the level of realism, diversity, and complexity imposed by the activities in our benchmark. We make BEHAVIOR publicly available at behavior.stanford.edu to facilitate and calibrate the development of new embodied AI solutions.

Related papers

• EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment [38.14321677323052]
  Embodied artificial intelligence emphasizes the role of an agent's body in generating human-like behaviors. In this paper, we construct a benchmark platform for embodied intelligence evaluation in real-world city environments.
  arXiv  Detail & Related papers  (2024-10-12T17:49:26Z)
• On the Evaluation of Generative Robotic Simulations [35.8253733339539]
  We propose a comprehensive evaluation framework tailored to generative simulations. For single-task quality, we evaluate the realism of the generated task and the completeness of the generated trajectories. For task-level generalization, we assess the zero-shot generalization ability on unseen tasks of a policy trained with multiple generated tasks.
  arXiv  Detail & Related papers  (2024-10-10T17:49:25Z)
• BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation [63.42591251500825]
  We present BEHAVIOR-1K, a comprehensive simulation benchmark for human-centered robotics. The first is the definition of 1,000 everyday activities grounded in 50 scenes with more than 9,000 objects annotated with rich physical and semantic properties. The second is OMNIGIBSON, a novel simulation environment that supports these activities via realistic physics simulation and rendering of rigid bodies, deformable bodies, and liquids.
  arXiv  Detail & Related papers  (2024-03-14T09:48:36Z)
• BEHAVIOR in Habitat 2.0: Simulator-Independent Logical Task Description for Benchmarking Embodied AI Agents [31.499374840833124]
  We bring a subset of BEHAVIOR activities into Habitat 2.0 to benefit from its fast simulation speed. Inspired by the catalyzing effect that benchmarks have played in the AI fields, the community is looking for new benchmarks for embodied AI.
  arXiv  Detail & Related papers  (2022-06-13T21:37:31Z)
• Towards Autonomous Grading In The Real World [4.651327752886103]
  We aim to tackle the problem of autonomous grading, where a dozer is required to flatten an uneven area. We design both a realistic physical simulation and a scaled real prototype environment mimicking the real dozer dynamics and sensory information.
  arXiv  Detail & Related papers  (2022-06-13T12:21:20Z)
• Autonomous Reinforcement Learning: Formalism and Benchmarking [106.25788536376007]
  Real-world embodied learning, such as that performed by humans and animals, is situated in a continual, non-episodic world. Common benchmark tasks in RL are episodic, with the environment resetting between trials to provide the agent with multiple attempts. This discrepancy presents a major challenge when attempting to take RL algorithms developed for episodic simulated environments and run them on real-world platforms.
  arXiv  Detail & Related papers  (2021-12-17T16:28:06Z)
• Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments [66.83839051693695]
  Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. We propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives.
  arXiv  Detail & Related papers  (2021-09-16T10:37:21Z)
• iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes [54.04456391489063]
  iGibson is a novel simulation environment to develop robotic solutions for interactive tasks in large-scale realistic scenes. Our environment contains fifteen fully interactive home-sized scenes populated with rigid and articulated objects. iGibson features enable the generalization of navigation agents, and that the human-iGibson interface and integrated motion planners facilitate efficient imitation learning of simple human demonstrated behaviors.
  arXiv  Detail & Related papers  (2020-12-05T02:14:17Z)
• RoboTHOR: An Open Simulation-to-Real Embodied AI Platform [56.50243383294621]
  We introduce RoboTHOR to democratize research in interactive and embodied visual AI. We show there exists a significant gap between the performance of models trained in simulation when they are tested in both simulations and their carefully constructed physical analogs.
  arXiv  Detail & Related papers  (2020-04-14T20:52:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.