On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes

• URL: http://arxiv.org/abs/2410.19553v1
• Date: Fri, 25 Oct 2024 13:27:55 GMT
• Title: On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
• Authors: Rajat Modi, Vibhav Vineet, Yogesh Singh Rawat,
• Abstract summary: This paper explores the impact of occlusions in video action detection. We introduce five new benchmark datasets namely O-UCF, O-JHMDB, OVIS-UCF, OVIS-JHMDB and Real-OUCF. Models leveraging these recipes outperform existing video action-detectors under occlusion by 32.3% on O-UCF, 32.7% on O-JHMDB & 2.6% on Real-OUCF in terms of the vMAP metric.
• Score: 14.464718780172582
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper explores the impact of occlusions in video action detection. We facilitate this study by introducing five new benchmark datasets namely O-UCF and O-JHMDB consisting of synthetically controlled static/dynamic occlusions, OVIS-UCF and OVIS-JHMDB consisting of occlusions with realistic motions and Real-OUCF for occlusions in realistic-world scenarios. We formally confirm an intuitive expectation: existing models suffer a lot as occlusion severity is increased and exhibit different behaviours when occluders are static vs when they are moving. We discover several intriguing phenomenon emerging in neural nets: 1) transformers can naturally outperform CNN models which might have even used occlusion as a form of data augmentation during training 2) incorporating symbolic-components like capsules to such backbones allows them to bind to occluders never even seen during training and 3) Islands of agreement can emerge in realistic images/videos without instance-level supervision, distillation or contrastive-based objectives2(eg. video-textual training). Such emergent properties allow us to derive simple yet effective training recipes which lead to robust occlusion models inductively satisfying the first two stages of the binding mechanism (grouping/segregation). Models leveraging these recipes outperform existing video action-detectors under occlusion by 32.3% on O-UCF, 32.7% on O-JHMDB & 2.6% on Real-OUCF in terms of the vMAP metric. The code for this work has been released at https://github.com/rajatmodi62/OccludedActionBenchmark.

Related papers

• EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization [17.622013322533423]
  We introduce EVA02-AT, a suite of EVA02-based video-language foundation models tailored to egocentric video understanding tasks.<n> EVA02-AT efficiently transfers an image-based CLIP model into a unified video encoder via a single-stage pretraining.<n>We introduce the Symmetric Multi-Similarity (SMS) loss and a novel training framework that advances all soft labels for both positive and negative pairs.
  arXiv  Detail & Related papers  (2025-06-17T09:51:51Z)
• OccludeNet: A Causal Journey into Mixed-View Actor-Centric Video Action Recognition under Occlusions [37.79525665359017]
  OccludeNet is a large-scale occluded video dataset that includes both real-world and synthetic occlusion scene videos. We propose a structural causal model for occluded scenes and introduce the Causal Action Recognition framework, which employs backdoor adjustment and counterfactual reasoning.
  arXiv  Detail & Related papers  (2024-11-24T06:10:05Z)
• Distilling Aggregated Knowledge for Weakly-Supervised Video Anomaly Detection [11.250490586786878]
  Video anomaly detection aims to develop automated models capable of identifying abnormal events in surveillance videos. We show that distilling knowledge from aggregated representations into a relatively simple model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-06-05T00:44:42Z)
• Mixed Diffusion for 3D Indoor Scene Synthesis [55.94569112629208]
  We present MiDiffusion, a novel mixed discrete-continuous diffusion model architecture. We represent a scene layout by a 2D floor plan and a set of objects, each defined by its category, location, size, and orientation. Our experimental results demonstrate that MiDiffusion substantially outperforms state-of-the-art autoregressive and diffusion models in floor-conditioned 3D scene synthesis.
  arXiv  Detail & Related papers  (2024-05-31T17:54:52Z)
• Adversarial Augmentation Training Makes Action Recognition Models More Robust to Realistic Video Distribution Shifts [13.752169303624147]
  Action recognition models often lack robustness when faced with natural distribution shifts between training and test data. We propose two novel evaluation methods to assess model resilience to such distribution disparity. We experimentally demonstrate the superior performance of the proposed adversarial augmentation approach over baselines across three state-of-the-art action recognition models.
  arXiv  Detail & Related papers  (2024-01-21T05:50:39Z)
• Unsupervised Video Anomaly Detection with Diffusion Models Conditioned on Compact Motion Representations [17.816344808780965]
  unsupervised video anomaly detection (VAD) problem involves classifying each frame in a video as normal or abnormal, without any access to labels. To accomplish this, proposed method employs conditional diffusion models, where the input data is features extracted from pre-trained network. Our method utilizes a data-driven threshold and considers a high reconstruction error as an indicator of anomalous events.
  arXiv  Detail & Related papers  (2023-07-04T07:36:48Z)
• Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors [117.61449210940955]
  We propose an efficient abnormal event detection model based on a lightweight masked auto-encoder (AE) applied at the video frame level. We introduce an approach to weight tokens based on motion gradients, thus shifting the focus from the static background scene to the foreground objects. We generate synthetic abnormal events to augment the training videos, and task the masked AE model to jointly reconstruct the original frames.
  arXiv  Detail & Related papers  (2023-06-21T06:18:05Z)
• Re-Evaluating LiDAR Scene Flow for Autonomous Driving [80.37947791534985]
  Popular benchmarks for self-supervised LiDAR scene flow have unrealistic rates of dynamic motion, unrealistic correspondences, and unrealistic sampling patterns. We evaluate a suite of top methods on a suite of real-world datasets. We show that despite the emphasis placed on learning, most performance gains are caused by pre- and post-processing steps.
  arXiv  Detail & Related papers  (2023-04-04T22:45:50Z)
• You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos [56.676761067861236]
  Given an untrimmed video, temporal sentence grounding aims to locate a target moment semantically according to a sentence query. Previous respectable works have made decent success, but they only focus on high-level visual features extracted from decoded frames. We propose a new setting, compressed-domain TSG, which directly utilizes compressed videos rather than fully-decompressed frames as the visual input.
  arXiv  Detail & Related papers  (2023-03-14T12:53:27Z)
• Boosting Low-Data Instance Segmentation by Unsupervised Pre-training with Saliency Prompt [103.58323875748427]
  This work offers a novel unsupervised pre-training solution for low-data regimes. Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models. Experimental results show that our method significantly boosts several QEIS models on three datasets.
  arXiv  Detail & Related papers  (2023-02-02T15:49:03Z)
• Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking [19.772968520292345]
  In this paper, we study whether 3D synthetic data can replace real-world videos for association training. Specifically, we introduce a large-scale synthetic data engine named MOTX, where the motion characteristics of cameras and objects are manually configured to be similar to those in real-world datasets. We show that compared with real data, association knowledge obtained from synthetic data can achieve very similar performance on real-world test sets without domain adaption techniques.
  arXiv  Detail & Related papers  (2021-06-30T14:46:36Z)
• ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency [62.38914747727636]
  We study self-supervised video representation learning, which is a challenging task due to 1) a lack of labels for explicit supervision and 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other. In this paper, we observe that the consistency between positive samples is the key to learn robust video representations.
  arXiv  Detail & Related papers  (2021-06-04T08:44:50Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.