Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environments

• URL: http://arxiv.org/abs/2410.18195v1
• Date: Wed, 23 Oct 2024 18:01:09 GMT
• Title: Personalized Instance-based Navigation Toward User-Specific Objects in Realistic Environments
• Authors: Luca Barsellotti, Roberto Bigazzi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara,
• Abstract summary: We propose a new task denominated Personalized Instance-based Navigation (PIN), in which an embodied agent is tasked with locating and reaching a specific personal object. In each episode, the target object is presented to the agent using two modalities: a set of visual reference images on a neutral background and manually annotated textual descriptions.
• Score: 44.6372390798904
• License:
• Abstract: In the last years, the research interest in visual navigation towards objects in indoor environments has grown significantly. This growth can be attributed to the recent availability of large navigation datasets in photo-realistic simulated environments, like Gibson and Matterport3D. However, the navigation tasks supported by these datasets are often restricted to the objects present in the environment at acquisition time. Also, they fail to account for the realistic scenario in which the target object is a user-specific instance that can be easily confused with similar objects and may be found in multiple locations within the environment. To address these limitations, we propose a new task denominated Personalized Instance-based Navigation (PIN), in which an embodied agent is tasked with locating and reaching a specific personal object by distinguishing it among multiple instances of the same category. The task is accompanied by PInNED, a dedicated new dataset composed of photo-realistic scenes augmented with additional 3D objects. In each episode, the target object is presented to the agent using two modalities: a set of visual reference images on a neutral background and manually annotated textual descriptions. Through comprehensive evaluations and analyses, we showcase the challenges of the PIN task as well as the performance and shortcomings of currently available methods designed for object-driven navigation, considering modular and end-to-end agents.

Related papers

• Right Place, Right Time! Towards ObjectNav for Non-Stationary Goals [55.581423861790945]
  We present a novel approach to tackle the ObjectNav task for non-stationary and potentially occluded targets in an indoor environment. We present its formulation, feasibility, and a navigation benchmark using a novel memory-enhanced LLM-based policy.
  arXiv  Detail & Related papers  (2024-03-14T22:33:22Z)
• Tracking through Containers and Occluders in the Wild [32.86030395660071]
  We introduce $textbfTCOW$, a new benchmark and model for visual tracking through heavy occlusion and containment. We create a mixture of synthetic and annotated real datasets to support both supervised learning and structured evaluation of model performance. We evaluate two recent transformer-based video models and find that while they can be surprisingly capable of tracking targets under certain settings of task variation, there remains a considerable performance gap before we can claim a tracking model to have acquired a true notion of object permanence.
  arXiv  Detail & Related papers  (2023-05-04T17:59:58Z)
• Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances [90.61897965658183]
  We consider the problem of embodied visual navigation given an image-goal (ImageNav) Unlike related navigation tasks, ImageNav does not have a standardized task definition which makes comparison across methods difficult. We present the Instance-specific ImageNav task (ImageNav) to address these limitations.
  arXiv  Detail & Related papers  (2022-11-29T02:29:35Z)
• Object Memory Transformer for Object Goal Navigation [10.359616364592075]
  This paper presents a reinforcement learning method for object goal navigation (Nav) An agent navigates in 3D indoor environments to reach a target object based on long-term observations of objects and scenes. To the best of our knowledge, this is the first work that uses a long-term memory of object semantics in a goal-oriented navigation task.
  arXiv  Detail & Related papers  (2022-03-24T09:16:56Z)
• ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations [52.226947570070784]
  We present Object, a dataset of 100 objects that addresses both challenges with two key innovations. First, Object encodes the visual, auditory, and tactile sensory data for all objects, enabling a number of multisensory object recognition tasks. Second, Object employs a uniform, object-centric simulations, and implicit representation for each object's visual textures, tactile readings, and tactile readings, making the dataset flexible to use and easy to share.
  arXiv  Detail & Related papers  (2021-09-16T14:00:59Z)
• ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments [85.81157224163876]
  We combine Vision-and-Language Navigation, assembling of collected objects, and object referring expression comprehension, to create a novel joint navigation-and-assembly task, named ArraMon. During this task, the agent is asked to find and collect different target objects one-by-one by navigating based on natural language instructions in a complex, realistic outdoor environment. We present results for several baseline models (integrated and biased) and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance gap demonstrates that our task is challenging and presents a wide scope for future work.
  arXiv  Detail & Related papers  (2020-11-15T23:30:36Z)
• Exploiting Scene-specific Features for Object Goal Navigation [9.806910643086043]
  We introduce a new reduced dataset that speeds up the training of navigation models. Our proposed dataset permits the training of models that do not exploit online-built maps in reasonable times. We propose the SMTSC model, an attention-based model capable of exploiting the correlation between scenes and objects contained in them.
  arXiv  Detail & Related papers  (2020-08-21T10:16:01Z)
• ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects [119.46959413000594]
  This document summarizes the consensus recommendations of a working group on ObjectNav. We make recommendations on subtle but important details of evaluation criteria. We provide a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020.
  arXiv  Detail & Related papers  (2020-06-23T17:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.