Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI

• URL: http://arxiv.org/abs/2308.05221v1
• Date: Wed, 9 Aug 2023 20:56:56 GMT
• Title: Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI
• Authors: Hangjie Shi, Leslie Ball, Govind Thattai, Desheng Zhang, Lucy Hu, Qiaozi Gao, Suhaila Shakiah, Xiaofeng Gao, Aishwarya Padmakumar, Bofei Yang, Cadence Chung, Dinakar Guthy, Gaurav Sukhatme, Karthika Arumugam, Matthew Wen, Osman Ipek, Patrick Lange, Rohan Khanna, Shreyas Pansare, Vasu Sharma, Chao Zhang, Cris Flagg, Daniel Pressel, Lavina Vaz, Luke Dai, Prasoon Goyal, Sattvik Sahai, Shaohua Liu, Yao Lu, Anna Gottardi, Shui Hu, Yang Liu, Dilek Hakkani-Tur, Kate Bland, Heather Rocker, James Jeun, Yadunandana Rao, Michael Johnston, Akshaya Iyengar, Arindam Mandal, Prem Natarajan, Reza Ghanadan
• Abstract summary: This paper describes the SimBot Challenge, a new challenge in which university teams compete to build robot assistants. We describe the infrastructure and support provided to the teams including Alexa Arena, the simulated environment, and the ML toolkit. We provide analysis of the performance of the competing SimBots during the competition.
• Score: 26.767216491124447
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the SocialBot Grand Challenge and the TaskBot Challenge. As conversational agents increasingly appear in multimodal and embodied contexts, it is important to explore the affordances of conversational interaction augmented with computer vision and physical embodiment. This paper describes the SimBot Challenge, a new challenge in which university teams compete to build robot assistants that complete tasks in a simulated physical environment. This paper provides an overview of the SimBot Challenge, which included both online and offline challenge phases. We describe the infrastructure and support provided to the teams including Alexa Arena, the simulated environment, and the ML toolkit provided to teams to accelerate their building of vision and language models. We summarize the approaches the participating teams took to overcome research challenges and extract key lessons learned. Finally, we provide analysis of the performance of the competing SimBots during the competition.

Related papers

• Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World [38.54892412474853]
  The Real Robot Challenge 2022 served as a bridge between the reinforcement learning and robotics communities. We asked the participants to learn two dexterous manipulation tasks involving pushing, grasping, and in-hand orientation from provided real-robot datasets. An extensive software documentation and an initial stage based on a simulation of the real set-up made the competition particularly accessible.
  arXiv  Detail & Related papers  (2023-08-15T12:40:56Z)
• RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot [56.130215236125224]
  A key challenge in robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent research in one-shot imitation learning has shown promise in transferring trained policies to new tasks based on demonstrations. This paper aims to unlock the potential for an agent to generalize to hundreds of real-world skills with multi-modal perception.
  arXiv  Detail & Related papers  (2023-07-02T15:33:31Z)
• HomeRobot: Open-Vocabulary Mobile Manipulation [107.05702777141178]
  Open-Vocabulary Mobile Manipulation (OVMM) is the problem of picking any object in any unseen environment, and placing it in a commanded location. HomeRobot has two components: a simulation component, which uses a large and diverse curated object set in new, high-quality multi-room home environments; and a real-world component, providing a software stack for the low-cost Hello Robot Stretch.
  arXiv  Detail & Related papers  (2023-06-20T14:30:32Z)
• Alexa, Let's Work Together: Introducing the First Alexa Prize TaskBot Challenge on Conversational Task Assistance [22.3267314621785]
  The Alexa Prize TaskBot challenge builds on the success of the SocialBot challenge by introducing the requirements of interactively assisting humans with real-world tasks. This paper provides an overview of the TaskBot challenge, describes the infrastructure support provided to the teams with the CoBot Toolkit, and summarizes the approaches the participating teams took to overcome the research challenges.
  arXiv  Detail & Related papers  (2022-09-13T22:01:42Z)
• Miutsu: NTU's TaskBot for the Alexa Prize [24.70443137383939]
  This paper introduces Miutsu, National Taiwan University's Alexa Prize TaskBot. It is designed to assist users in completing tasks requiring multiple steps and decisions in two different domains -- home improvement and cooking.
  arXiv  Detail & Related papers  (2022-05-16T04:56:55Z)
• Grasp and Motion Planning for Dexterous Manipulation for the Real Robot Challenge [0.05735035463793007]
  The Real Robot Challenge is a three-phase dexterous manipulation competition. Our approach combines motion planning with several motion primitives to manipulate the object. We were anonymously known as ardentstork' on the competition leaderboard.
  arXiv  Detail & Related papers  (2021-01-08T04:13:39Z)
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration [116.28433607265573]
  We introduce Watch-And-Help (WAH), a challenge for testing social intelligence in AI agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. We build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines.
  arXiv  Detail & Related papers  (2020-10-19T21:48:31Z)
• A Game AI Competition to foster Collaborative AI research and development [5.682875185620577]
  We present the Geometry Friends Game AI Competition. The concept of the game is simple, though its solving has proven to be difficult. We discuss the competition and the challenges it brings, and present an overview of the current solutions.
  arXiv  Detail & Related papers  (2020-10-17T23:03:06Z)
• ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ) [64.60303062063663]
  This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ) The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020.
  arXiv  Detail & Related papers  (2020-09-23T19:48:02Z)
• Sim2Real for Peg-Hole Insertion with Eye-in-Hand Camera [58.720142291102135]
  We use a simulator to learn the peg-hole insertion problem and then transfer the learned model to the real robot. We show that the transferred policy, which only takes RGB-D and joint information (proprioception) can perform well on the real robot.
  arXiv  Detail & Related papers  (2020-05-29T05:58:54Z)
• Analysing Affective Behavior in the First ABAW 2020 Competition [49.90617840789334]
  The Affective Behavior Analysis in-the-wild (ABAW) 2020 Competition is the first Competition aiming at automatic analysis of the three main behavior tasks. We describe this Competition, to be held in conjunction with the IEEE Conference on Face and Gesture Recognition, May 2020, in Buenos Aires, Argentina. We outline the evaluation metrics, present both the baseline system and the top-3 performing teams' methodologies per Challenge and finally present their obtained results.
  arXiv  Detail & Related papers  (2020-01-30T15:41:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.