Leveraging Transformers for Weakly Supervised Object Localization in Unconstrained Videos

• URL: http://arxiv.org/abs/2407.06018v1
• Date: Mon, 8 Jul 2024 15:08:41 GMT
• Title: Leveraging Transformers for Weakly Supervised Object Localization in Unconstrained Videos
• Authors: Shakeeb Murtaza, Marco Pedersoli, Aydin Sarraf, Eric Granger,
• Abstract summary: State-of-the-art WSVOL methods rely on class activation mapping (CAM) Our TrCAM-V method allows training a localization network by sampling pseudo-pixels on the fly from these regions. During inference, the model can process individual frames for real-time localization applications.
• Score: 12.762698438702854
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weakly-Supervised Video Object Localization (WSVOL) involves localizing an object in videos using only video-level labels, also referred to as tags. State-of-the-art WSVOL methods like Temporal CAM (TCAM) rely on class activation mapping (CAM) and typically require a pre-trained CNN classifier. However, their localization accuracy is affected by their tendency to minimize the mutual information between different instances of a class and exploit temporal information during training for downstream tasks, e.g., detection and tracking. In the absence of bounding box annotation, it is challenging to exploit precise information about objects from temporal cues because the model struggles to locate objects over time. To address these issues, a novel method called transformer based CAM for videos (TrCAM-V), is proposed for WSVOL. It consists of a DeiT backbone with two heads for classification and localization. The classification head is trained using standard classification loss (CL), while the localization head is trained using pseudo-labels that are extracted using a pre-trained CLIP model. From these pseudo-labels, the high and low activation values are considered to be foreground and background regions, respectively. Our TrCAM-V method allows training a localization network by sampling pseudo-pixels on the fly from these regions. Additionally, a conditional random field (CRF) loss is employed to align the object boundaries with the foreground map. During inference, the model can process individual frames for real-time localization applications. Extensive experiments on challenging YouTube-Objects unconstrained video datasets show that our TrCAM-V method achieves new state-of-the-art performance in terms of classification and localization accuracy.

Related papers

• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  Vision-Language Models (VLMs) have shown remarkable capabilities across diverse visual tasks. Current VLMs lack a fundamental cognitive ability: learning to localize objects in a scene by taking into account the context. This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts [57.01985221057047]
  This paper introduces a novel method that learnstemporal prompt embeddings for weakly supervised video anomaly detection and localization (WSVADL) based on pre-trained vision-language models (VLMs) Our method achieves state-of-theart performance on three public benchmarks for the WSVADL task.
  arXiv  Detail & Related papers  (2024-08-12T03:31:29Z)
• Harnessing Large Language Models for Training-free Video Anomaly Detection [34.76811491190446]
  Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Training-based methods are prone to be domain-specific, thus being costly for practical deployment. We propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm.
  arXiv  Detail & Related papers  (2024-04-01T09:34:55Z)
• CoLo-CAM: Class Activation Mapping for Object Co-Localization in Weakly-Labeled Unconstrained Videos [23.447026400051772]
  Co-Localization-CAM method exploitstemporal information in activation maps during training without constraining an object's position. Co-Localization improves localization performance because the joint learning creates direct communication among pixels across all image locations.
  arXiv  Detail & Related papers  (2023-03-16T02:29:53Z)
• Location-Aware Self-Supervised Transformers [74.76585889813207]
  We propose to pretrain networks for semantic segmentation by predicting the relative location of image parts. We control the difficulty of the task by masking a subset of the reference patch features visible to those of the query. Our experiments show that this location-aware pretraining leads to representations that transfer competitively to several challenging semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T16:24:29Z)
• TCAM: Temporal Class Activation Maps for Object Localization in Weakly-Labeled Unconstrained Videos [22.271760669551817]
  Weakly supervised object localization (WSVOL) allows object locating in videos using only global video tags as such object class. In this paper, we leverage the successful class activation mapping (CAM) methods, designed for WSOL based on still images. A new Temporal CAM (TCAM) method is introduced to train ariminant deep learning (DL) model to exploittemporal information in videos.
  arXiv  Detail & Related papers  (2022-08-30T21:20:34Z)
• PreViTS: Contrastive Pretraining with Video Tracking Supervision [53.73237606312024]
  PreViTS is an unsupervised SSL framework for selecting clips containing the same object. PreViTS spatially constrains the frame regions to learn from and trains the model to locate meaningful objects. We train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS.
  arXiv  Detail & Related papers  (2021-12-01T19:49:57Z)
• TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization [112.46381729542658]
  Weakly supervised object localization (WSOL) is a challenging problem when given image category labels. We introduce the token semantic coupled attention map (TS-CAM) to take full advantage of the self-attention mechanism in visual transformer for long-range dependency extraction.
  arXiv  Detail & Related papers  (2021-03-27T09:43:16Z)
• Instance Localization for Self-supervised Detection Pretraining [68.24102560821623]
  We propose a new self-supervised pretext task, called instance localization. We show that integration of bounding boxes into pretraining promotes better task alignment and architecture alignment for transfer learning. Experimental results demonstrate that our approach yields state-of-the-art transfer learning results for object detection.
  arXiv  Detail & Related papers  (2021-02-16T17:58:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.