EgoPet: Egomotion and Interaction Data from an Animal's Perspective

• URL: http://arxiv.org/abs/2404.09991v1
• Date: Mon, 15 Apr 2024 17:59:47 GMT
• Title: EgoPet: Egomotion and Interaction Data from an Animal's Perspective
• Authors: Amir Bar, Arya Bakhtiar, Danny Tran, Antonio Loquercio, Jathushan Rajasegaran, Yann LeCun, Amir Globerson, Trevor Darrell,
• Abstract summary: We introduce a dataset of pet egomotion imagery with diverse examples of simultaneous egomotion and multi-agent interaction. EgoPet offers a radically distinct perspective from existing egocentric datasets of humans or vehicles. We define two in-domain benchmark tasks that capture animal behavior, and a third benchmark to assess the utility of EgoPet as a pretraining resource to robotic quadruped locomotion.
• Score: 82.7192364237065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Animals perceive the world to plan their actions and interact with other agents to accomplish complex tasks, demonstrating capabilities that are still unmatched by AI systems. To advance our understanding and reduce the gap between the capabilities of animals and AI systems, we introduce a dataset of pet egomotion imagery with diverse examples of simultaneous egomotion and multi-agent interaction. Current video datasets separately contain egomotion and interaction examples, but rarely both at the same time. In addition, EgoPet offers a radically distinct perspective from existing egocentric datasets of humans or vehicles. We define two in-domain benchmark tasks that capture animal behavior, and a third benchmark to assess the utility of EgoPet as a pretraining resource to robotic quadruped locomotion, showing that models trained from EgoPet outperform those trained from prior datasets.

Related papers

• EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views [51.53089073920215]
  Understanding egocentric human-object interaction (HOI) is a fundamental aspect of human-centric perception. Existing methods primarily leverage observations of HOI to capture interaction regions from an exocentric view. We present EgoChoir, which links object structures with interaction contexts inherent in appearance and head motion to reveal object affordance.
  arXiv  Detail & Related papers  (2024-05-22T14:03:48Z)
• EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World [44.34800426136217]
  We introduce EgoExoLearn, a dataset that emulates the human demonstration following process. EgoExoLearn contains egocentric and demonstration video data spanning 120 hours. We present benchmarks such as cross-view association, cross-view action planning, and cross-view referenced skill assessment.
  arXiv  Detail & Related papers  (2024-03-24T15:00:44Z)
• EgoGen: An Egocentric Synthetic Data Generator [53.32942235801499]
  EgoGen is a new synthetic data generator that can produce accurate and rich ground-truth training data for egocentric perception tasks. At the heart of EgoGen is a novel human motion synthesis model that directly leverages egocentric visual inputs of a virtual human to sense the 3D environment. We demonstrate EgoGen's efficacy in three tasks: mapping and localization for head-mounted cameras, egocentric camera tracking, and human mesh recovery from egocentric views.
  arXiv  Detail & Related papers  (2024-01-16T18:55:22Z)
• Retrieval-Augmented Egocentric Video Captioning [53.2951243928289]
  EgoInstructor is a retrieval-augmented multimodal captioning model that automatically retrieves semantically relevant third-person instructional videos. We train the cross-view retrieval module with a novel EgoExoNCE loss that pulls egocentric and exocentric video features closer by aligning them to shared text features that describe similar actions.
  arXiv  Detail & Related papers  (2024-01-01T15:31:06Z)
• EgoHumans: An Egocentric 3D Multi-Human Benchmark [37.375846688453514]
  We present EgoHumans, a new multi-view multi-human video benchmark to advance the state-of-the-art of egocentric human 3D pose estimation and tracking. We propose a novel 3D capture setup to construct a comprehensive egocentric multi-human benchmark in the wild. We leverage consumer-grade wearable camera-equipped glasses for the egocentric view, which enables us to capture dynamic activities like playing tennis, fencing, volleyball, etc.
  arXiv  Detail & Related papers  (2023-05-25T21:37:36Z)
• Egocentric Video Task Translation [109.30649877677257]
  We propose EgoTask Translation (EgoT2), which takes a collection of models optimized on separate tasks and learns to translate their outputs for improved performance on any or all of them at once. Unlike traditional transfer or multi-task learning, EgoT2's flipped design entails separate task-specific backbones and a task translator shared across all tasks, which captures synergies between even heterogeneous tasks and mitigates task competition.
  arXiv  Detail & Related papers  (2022-12-13T00:47:13Z)
• Ego-Body Pose Estimation via Ego-Head Pose Estimation [22.08240141115053]
  Estimating 3D human motion from an egocentric video sequence plays a critical role in human behavior understanding and has various applications in VR/AR. We propose a new method, Ego-Body Pose Estimation via Ego-Head Pose Estimation (EgoEgo), which decomposes the problem into two stages, connected by the head motion as an intermediate representation. This disentanglement of head and body pose eliminates the need for training datasets with paired egocentric videos and 3D human motion.
  arXiv  Detail & Related papers  (2022-12-09T02:25:20Z)
• MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain [23.598727613908853]
  We present MECCANO, a dataset of egocentric videos to study humans behavior understanding in industrial-like settings. The multimodality is characterized by the presence of gaze signals, depth maps and RGB videos acquired simultaneously with a custom headset. The dataset has been explicitly labeled for fundamental tasks in the context of human behavior understanding from a first person view.
  arXiv  Detail & Related papers  (2022-09-19T00:52:42Z)
• Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos [92.38049744463149]
  We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Our experiments show that our Ego-Exo framework can be seamlessly integrated into standard video models.
  arXiv  Detail & Related papers  (2021-04-16T06:10:10Z)
• The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain [20.99718135562034]
  We introduce MECCANO, the first dataset of egocentric videos to study human-object interactions in industrial-like settings. The dataset has been explicitly labeled for the task of recognizing human-object interactions from an egocentric perspective. Baseline results show that the MECCANO dataset is a challenging benchmark to study egocentric human-object interactions in industrial-like scenarios.
  arXiv  Detail & Related papers  (2020-10-12T12:50:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.