Project Aria: A New Tool for Egocentric Multi-Modal AI Research

• URL: http://arxiv.org/abs/2308.13561v3
• Date: Sun, 1 Oct 2023 20:16:22 GMT
• Title: Project Aria: A New Tool for Egocentric Multi-Modal AI Research
• Abstract summary: Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. Our team at Meta Reality Labs Research built the Aria device, an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area.
• Score: 44.91047009314488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric, multi-modal data as available on future augmented reality (AR) devices provides unique challenges and opportunities for machine perception. These future devices will need to be all-day wearable in a socially acceptable form-factor to support always available, context-aware and personalized AI applications. Our team at Meta Reality Labs Research built the Aria device, an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area. In this paper, we describe the Aria device hardware including its sensor configuration and the corresponding software tools that enable recording and processing of such data.

Related papers

• MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes [63.12421718928034]
  We introduce MIST (the Multimodal Interactive Speech-based Tool-calling), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices.<n>We find that there is a significant gap between open-weight and closed-weight LLMs, and that even closed-weight LLMs have substantial headroom.
  arXiv  Detail & Related papers  (2026-05-07T19:57:39Z)
• Gypscie: A Cross-Platform AI Artifact Management System [0.6784745592354215]
  We introduce Gypscie, a cross-platform AI artifact management system.<n>By providing a unified view of all AI artifacts, Gypscie simplifies the development and deployment of AI applications.
  arXiv  Detail & Related papers  (2026-04-11T18:24:26Z)
• Aria Gen 2 Pilot Dataset [9.785079054502377]
  The Aria Gen 2 Pilot dataset is an egocentric multimodal open dataset captured using the state-of-the-art Aria Gen 2 glasses.<n>The initial release features Dia'ane, our primary subject, who records her daily activities alongside friends, each equipped with Aria Gen 2 glasses.<n>In each of the scenarios, we provide comprehensive raw sensor data and output data from various machine perception algorithms.<n>These data illustrate the device's ability to perceive the wearer, the surrounding environment, and interactions between the wearer and the environment, while maintaining robust performance across diverse users and conditions.
  arXiv  Detail & Related papers  (2025-10-17T18:21:11Z)
• Everyday AR through AI-in-the-Loop [16.65802031939239]
  This workshop brings together experts and practitioners from augmented reality (AR) and artificial intelligence (AI) to shape the future of AI-in-the-loop everyday AR experiences. We discuss a range of topics, including adaptive and context-aware AR, generative AR content creation, always-on AI assistants, AI-driven accessible design, and real-world-oriented AI agents. Our goal is to identify the opportunities and challenges in AI-enabled AR, focusing on creating novel AR experiences that seamlessly blend the digital and physical worlds.
  arXiv  Detail & Related papers  (2024-12-17T08:51:55Z)
• Aria-NeRF: Multimodal Egocentric View Synthesis [17.0554791846124]
  We seek to accelerate research in developing rich, multimodal scene models trained from egocentric data, based on differentiable volumetric ray-tracing inspired by Neural Radiance Fields (NeRFs) This dataset offers a comprehensive collection of sensory data, featuring RGB images, eye-tracking camera footage, audio recordings from a microphone, atmospheric pressure readings from a barometer, positional coordinates from GPS, and information from dual-frequency IMU datasets (1kHz and 800Hz) The diverse data modalities and the real-world context captured within this dataset serve as a robust foundation for furthering our understanding of human behavior and enabling more immersive and intelligent experiences in
  arXiv  Detail & Related papers  (2023-11-11T01:56:35Z)
• AI-Generated Images as Data Source: The Dawn of Synthetic Era [61.879821573066216]
  generative AI has unlocked the potential to create synthetic images that closely resemble real-world photographs. This paper explores the innovative concept of harnessing these AI-generated images as new data sources. In contrast to real data, AI-generated data exhibit remarkable advantages, including unmatched abundance and scalability.
  arXiv  Detail & Related papers  (2023-10-03T06:55:19Z)
• Autonomous Aerial Robot for High-Speed Search and Intercept Applications [86.72321289033562]
  A fully-autonomous aerial robot for high-speed object grasping has been proposed. As an additional sub-task, our system is able to autonomously pierce balloons located in poles close to the surface. Our approach has been validated in a challenging international competition and has shown outstanding results.
  arXiv  Detail & Related papers  (2021-12-10T11:49:51Z)
• How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures [7.085772863979686]
  Deep neural networks (DNNs) have led to large strides in various Artificial Intelligence (AI) inference tasks, such as object and speech recognition. deploying such AI models across commodity devices faces significant challenges. We present techniques for achieving real-time performance following a cross-stack approach.
  arXiv  Detail & Related papers  (2021-06-21T11:23:12Z)
• Towards Tool-Support for Interactive-Machine Learning Applications in the Android Ecosystem [0.0]
  We believe there is a need for tool-support for AI engineers to address the challenges of implementing, testing, and deploying machine learning models. This paper presents preliminary results of a series of inquiries, including interviews with AI engineers and experiments for an interactive machine learning use case with a Smartwatch and Smartphone.
  arXiv  Detail & Related papers  (2021-03-27T09:28:40Z)
• Reliable Fleet Analytics for Edge IoT Solutions [0.0]
  We propose a framework for facilitating machine learning at the edge for AIoT applications. The contribution is an architecture that includes services, tools, and methods for delivering fleet analytics at scale. We present a preliminary validation of the framework by performing experiments with IoT devices on a university campus's rooms.
  arXiv  Detail & Related papers  (2021-01-12T11:28:43Z)
• SensiX: A Platform for Collaborative Machine Learning on the Edge [69.1412199244903]
  We present SensiX, a personal edge platform that stays between sensor data and sensing models. We demonstrate its efficacy in developing motion and audio-based multi-device sensing systems. Our evaluation shows that SensiX offers a 7-13% increase in overall accuracy and up to 30% increase across different environment dynamics at the expense of 3mW power overhead.
  arXiv  Detail & Related papers  (2020-12-04T23:06:56Z)
• Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things [98.10037444792444]
  We show how AI can empower the IoT to make it faster, smarter, greener, and safer. First, we present progress in AI research for IoT from four perspectives: perceiving, learning, reasoning, and behaving. Finally, we summarize some promising applications of AIoT that are likely to profoundly reshape our world.
  arXiv  Detail & Related papers  (2020-11-17T13:14:28Z)
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents [61.36681529571202]
  We describe a new concept for reproducible robotics research that integrates development and benchmarking. One of the central components of this setup is the Duckietown Autolab, a standardized setup that is itself relatively low-cost and reproducible. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
  arXiv  Detail & Related papers  (2020-09-09T15:31:29Z)
• HoloLens 2 Research Mode as a Tool for Computer Vision Research [45.404600947276826]
  We present HoloLens 2 Research Mode, an API and a set of tools enabling access to the raw sensor streams. We provide an overview of the API and explain how it can be used to build mixed reality applications. We also show how to combine the Research Mode sensor data with the built-in eye and hand tracking capabilities.
  arXiv  Detail & Related papers  (2020-08-25T19:05:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.