The Waymo Open Sim Agents Challenge

• URL: http://arxiv.org/abs/2305.12032v4
• Date: Mon, 11 Dec 2023 18:57:36 GMT
• Title: The Waymo Open Sim Agents Challenge
• Authors: Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nick Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, Brandyn White, Dragomir Anguelov
• Abstract summary: We introduce the Open Sim Agents Challenge (WOSAC) The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition.
• Score: 37.69742145084953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.

Related papers

• Promptable Closed-loop Traffic Simulation [57.36568236100507]
  ProSim is a multimodal promptable closed-loop traffic simulation framework. ProSim rolls out a traffic scenario in a closed-loop manner, modeling each agent's interaction with other traffic participants. To support research on promptable traffic simulation, we create ProSim-Instruct-520k, a multimodal prompt-scenario paired driving dataset.
  arXiv  Detail & Related papers  (2024-09-09T17:59:15Z)
• Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge [93.4434417387526]
  We propose Open Vocabulary Mobile Manipulation as a key benchmark task for robotics. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. We detail the results and methodologies used, both in simulation and real-world settings.
  arXiv  Detail & Related papers  (2024-07-09T15:15:01Z)
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking [65.24988062003096]
  We present NAVSIM, a framework for benchmarking vision-based driving policies. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights.
  arXiv  Detail & Related papers  (2024-06-21T17:59:02Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• The 2nd Place Solution for 2023 Waymo Open Sim Agents Challenge [8.821526792549648]
  We propose a simple yet effective autoregressive method for simulating multi-agent behaviors. Our submission named MTR+++ achieves 0.4697 on the Realism Meta metric in 2023 Open Sim Agents Challenge (WOSAC) Besides, a modified model based on MTR named MTR_E is proposed after the challenge, which has a better score 0.4911 and is ranked the 3rd on the leaderboard of WOSAC as of June 25, 2023.
  arXiv  Detail & Related papers  (2023-06-28T04:33:12Z)
• Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023 [3.4520774137890555]
  This report presents our 1st place solution for the Open Sim Agents Challenge (WOSAC) 2023. Our proposed MultiVerse Transformer for Agent simulation (MVTA) effectively leverages transformer-based motion prediction approaches. In order to produce simulations with a high degree of realism, we design novel training and sampling methods.
  arXiv  Detail & Related papers  (2023-06-20T20:01:07Z)
• Learn-to-Race Challenge 2022: Benchmarking Safe Learning and Cross-domain Generalisation in Autonomous Racing [12.50944966521162]
  We present the results of our autonomous racing virtual challenge, based on the newly-released Learn-to-Race (L2R) simulation framework. In this paper, we describe the new L2R Task 2.0 benchmark, with refined metrics and baseline approaches. We also provide an overview of deployment, evaluation, and rankings for the inaugural instance of the L2R Autonomous Racing Virtual Challenge.
  arXiv  Detail & Related papers  (2022-05-05T22:31:19Z)
• Reactive Long Horizon Task Execution via Visual Skill and Precondition Models [59.76233967614774]
  We describe an approach for sim-to-real training that can accomplish unseen robotic tasks using models learned in simulation to ground components of a simple task planner. We show an increase in success rate from 91.6% to 98% in simulation and from 10% to 80% success rate in the real-world as compared with naive baselines.
  arXiv  Detail & Related papers  (2020-11-17T15:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.