JIST: Joint Image and Sequence Training for Sequential Visual Place Recognition

• URL: http://arxiv.org/abs/2403.19787v1
• Date: Thu, 28 Mar 2024 19:11:26 GMT
• Title: JIST: Joint Image and Sequence Training for Sequential Visual Place Recognition
• Authors: Gabriele Berton, Gabriele Trivigno, Barbara Caputo, Carlo Masone,
• Abstract summary: Visual Place Recognition aims at recognizing previously visited places by relying on visual clues, and it is used in robotics applications for SLAM and localization. We propose a novel Joint Image and Sequence Training protocol (JIST) that leverages large uncurated sets of images through a multi-task learning framework. We show that our model is able to outperform previous state of the art while being faster, using 8 times smaller descriptors, having a lighter architecture and allowing to process sequences of various lengths.
• Score: 21.039399444257807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Place Recognition aims at recognizing previously visited places by relying on visual clues, and it is used in robotics applications for SLAM and localization. Since typically a mobile robot has access to a continuous stream of frames, this task is naturally cast as a sequence-to-sequence localization problem. Nevertheless, obtaining sequences of labelled data is much more expensive than collecting isolated images, which can be done in an automated way with little supervision. As a mitigation to this problem, we propose a novel Joint Image and Sequence Training protocol (JIST) that leverages large uncurated sets of images through a multi-task learning framework. With JIST we also introduce SeqGeM, an aggregation layer that revisits the popular GeM pooling to produce a single robust and compact embedding from a sequence of single-frame embeddings. We show that our model is able to outperform previous state of the art while being faster, using 8 times smaller descriptors, having a lighter architecture and allowing to process sequences of various lengths. Code is available at https://github.com/ga1i13o/JIST

Related papers

• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences. We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries. We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• Made to Order: Discovering monotonic temporal changes via self-supervised video ordering [89.0660110757949]
  We exploit a simple proxy task of ordering a shuffled image sequence, with time' serving as a supervisory signal. We introduce a transformer-based model for ordering of image sequences of arbitrary length with built-in attribution maps.
  arXiv  Detail & Related papers  (2024-04-25T17:59:56Z)
• UnLoc: A Unified Framework for Video Localization Tasks [82.59118972890262]
  UnLoc is a new approach for temporal localization in untrimmed videos. It uses pretrained image and text towers, and feeds tokens to a video-text fusion model. We achieve state of the art results on all three different localization tasks with a unified approach.
  arXiv  Detail & Related papers  (2023-08-21T22:15:20Z)
• ASIC: Aligning Sparse in-the-wild Image Collections [86.66498558225625]
  We present a method for joint alignment of sparse in-the-wild image collections of an object category. We use pairwise nearest neighbors obtained from deep features of a pre-trained vision transformer (ViT) model as noisy and sparse keypoint matches. Experiments on CUB and SPair-71k benchmarks demonstrate that our method can produce globally consistent and higher quality correspondences.
  arXiv  Detail & Related papers  (2023-03-28T17:59:28Z)
• Learning Sequential Descriptors for Sequence-based Visual Place Recognition [14.738954189759156]
  In robotics, Visual Place Recognition is a continuous process that receives as input a video stream to produce a hypothesis of the robot's current position. This work proposes a detailed taxonomy of techniques using sequential descriptors, highlighting different mechanism to fuse the information from the individual images.
  arXiv  Detail & Related papers  (2022-07-08T12:52:04Z)
• Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework [83.82026345508334]
  We propose OFA, a unified multimodal pretrained model that unifies modalities (i.e., cross-modality, vision, language) and tasks (e.g., image generation, visual grounding, image captioning, image classification, text generation, etc.) OFA achieves new state-of-the-arts on a series of multimodal tasks, including image captioning (COCO test CIDEr: 149.6), text-to-image generation (COCO test FID: 10.5), VQA (test-std encoder acc.: 80.02), SNLI-VE (test acc.: 90.
  arXiv  Detail & Related papers  (2022-02-07T10:38:21Z)
• Few-shot Weakly-Supervised Object Detection via Directional Statistics [55.97230224399744]
  We propose a probabilistic multiple instance learning approach for few-shot Common Object Localization (COL) and few-shot Weakly Supervised Object Detection (WSOD) Our model simultaneously learns the distribution of the novel objects and localizes them via expectation-maximization steps. Our experiments show that the proposed method, despite being simple, outperforms strong baselines in few-shot COL and WSOD, as well as large-scale WSOD tasks.
  arXiv  Detail & Related papers  (2021-03-25T22:34:16Z)
• SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognition [31.714928102950594]
  We present a novel hybrid system that creates a high performance initial match hypothesis generator. Sequence descriptors are generated using a temporal convolutional network dubbed SeqNet. We then perform selective sequential score aggregation using shortlisted single image learnt descriptors to produce an overall place match hypothesis.
  arXiv  Detail & Related papers  (2021-02-23T10:32:10Z)
• Revisiting Sequence-to-Sequence Video Object Segmentation with Multi-Task Loss and Skip-Memory [4.343892430915579]
  Video Object (VOS) is an active research area of the visual domain. Current approaches lose objects in longer sequences, especially when the object is small or briefly occluded. We build upon a sequence-to-sequence approach that employs an encoder-decoder architecture together with a memory module for exploiting the sequential data.
  arXiv  Detail & Related papers  (2020-04-25T15:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.