OccludeNet: A Causal Journey into Mixed-View Actor-Centric Video Action Recognition under Occlusions

• URL: http://arxiv.org/abs/2411.15729v2
• Date: Mon, 09 Jun 2025 08:55:43 GMT
• Title: OccludeNet: A Causal Journey into Mixed-View Actor-Centric Video Action Recognition under Occlusions
• Abstract summary: We build OccludeNet, a large-scale occluded video dataset including both real and synthetic occlusion scenes.<n>Our analysis shows that actions with low scene relevance and partial body visibility see larger drops in accuracy.<n>We propose a structural causal model for occluded scenes and introduce the Causal Action Recognition (CAR) method, which uses backdoor adjustment and counterfactual reasoning.
• Score: 37.79525665359017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The lack of occlusion data in common action recognition video datasets limits model robustness and hinders consistent performance gains. We build OccludeNet, a large-scale occluded video dataset including both real and synthetic occlusion scenes in different natural settings. OccludeNet includes dynamic occlusion, static occlusion, and multi-view interactive occlusion, addressing gaps in current datasets. Our analysis shows occlusion affects action classes differently: actions with low scene relevance and partial body visibility see larger drops in accuracy. To overcome the limits of existing occlusion-aware methods, we propose a structural causal model for occluded scenes and introduce the Causal Action Recognition (CAR) method, which uses backdoor adjustment and counterfactual reasoning. This approach strengthens key actor information and improves model robustness to occlusion. We hope the challenges of OccludeNet will encourage more study of causal links in occluded scenes and lead to a fresh look at class relations, ultimately leading to lasting performance improvements. Our code and data is availibale at: https://github.com/The-Martyr/OccludeNet-Dataset

Related papers

• BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal [27.338971504674983]
  We introduce BeyondMasks, a paired benchmark for causally consistent video object removal.<n>The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions.<n>We propose CORE, a structured vision language model based evaluation protocol.
  arXiv  Detail & Related papers  (2026-08-20T14:37:40Z)
• LIBERO-Occ: Evaluating and Improving Vision-Language-Action Models under Scene-Induced Occlusion via Viewpoint Imagination [66.06027569507403]
  textbfViewpoint Imagination (VIM) generates a complementary view from an occluded primary observation and conditions action prediction on both observed and imagined evidence.<n>VIM improves robustness across task suites, occlusion types, and severity levels without requiring additional cameras at deployment time.
  arXiv  Detail & Related papers  (2026-06-09T13:39:49Z)
• JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generation [13.168628936598367]
  JointTuner is a novel adaptive joint training framework. We develop Adaptive LoRA, which incorporates a context-aware gating mechanism. Appearance-independent Temporal Loss is introduced to decouple motion patterns from intrinsic appearance.
  arXiv  Detail & Related papers  (2025-03-31T11:04:07Z)
• Object-Centric Latent Action Learning [70.3173534658611]
  We propose a novel object-centric latent action learning approach, based on VideoSaur and LAPO. This method effectively disentangles causal agent-object interactions from irrelevant background noise and reduces the performance degradation caused by distractors. Our preliminary experiments with the Distracting Control Suite show that latent action pretraining based on object decompositions improve the quality of inferred latent actions by x2.7 and efficiency of downstream fine-tuning with a small set of labeled actions, increasing return by x2.6 on average.
  arXiv  Detail & Related papers  (2025-02-13T11:27:05Z)
• On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes [14.464718780172582]
  This paper explores the impact of occlusions in video action detection. We introduce five new benchmark datasets namely O-UCF, O-JHMDB, OVIS-UCF, OVIS-JHMDB and Real-OUCF. Models leveraging these recipes outperform existing video action-detectors under occlusion by 32.3% on O-UCF, 32.7% on O-JHMDB & 2.6% on Real-OUCF in terms of the vMAP metric.
  arXiv  Detail & Related papers  (2024-10-25T13:27:55Z)
• Deep Generative Adversarial Network for Occlusion Removal from a Single Image [3.5639148953570845]
  We propose a fully automatic, two-stage convolutional neural network for fence segmentation and occlusion completion. We leverage generative adversarial networks (GANs) to synthesize realistic content, including both structure and texture, in a single shot for inpainting.
  arXiv  Detail & Related papers  (2024-09-20T06:00:45Z)
• Object-level Scene Deocclusion [92.39886029550286]
  We present a new self-supervised PArallel visible-to-COmplete diffusion framework, named PACO, for object-level scene deocclusion. To train PACO, we create a large-scale dataset with 500k samples to enable self-supervised learning. Experiments on COCOA and various real-world scenes demonstrate the superior capability of PACO for scene deocclusion, surpassing the state of the arts by a large margin.
  arXiv  Detail & Related papers  (2024-06-11T20:34:10Z)
• Imagine the Unseen: Occluded Pedestrian Detection via Adversarial Feature Completion [31.488897675973657]
  We propose to complete features for occluded regions so as to align the features of pedestrians across different occlusion patterns. In order to narrow down the gap between completed features and real fully visible ones, we propose an adversarial learning method. We report experimental results on the CityPersons, Caltech and CrowdHuman datasets.
  arXiv  Detail & Related papers  (2024-05-02T14:20:20Z)
• CMU-Flownet: Exploring Point Cloud Scene Flow Estimation in Occluded Scenario [10.852258389804984]
  Occlusions hinder point cloud frame alignment in LiDAR data, a challenge inadequately addressed by scene flow models. We introduce the Correlation Matrix Upsampling Flownet (CMU-Flownet), incorporating an occlusion estimation module within its cost volume layer. CMU-Flownet establishes state-of-the-art performance within the realms of occluded Flyingthings3D and KITTY datasets.
  arXiv  Detail & Related papers  (2024-04-16T13:47:21Z)
• Appearance-Based Refinement for Object-Centric Motion Segmentation [85.2426540999329]
  We introduce an appearance-based refinement method that leverages temporal consistency in video streams to correct inaccurate flow-based proposals. Our approach involves a sequence-level selection mechanism that identifies accurate flow-predicted masks as exemplars. Its performance is evaluated on multiple video segmentation benchmarks, including DAVIS, YouTube, SegTrackv2, and FBMS-59.
  arXiv  Detail & Related papers  (2023-12-18T18:59:51Z)
• Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements [83.5820690348833]
  We present a framework for low-level vision tasks that does not require any external training data corpus. Our approach learns neural modules by optimizing over a corrupted sequence, leveraging the weights of the coherence-temporal test and statistics internal statistics.
  arXiv  Detail & Related papers  (2023-12-13T01:57:11Z)
• TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models [75.20168902300166]
  We propose TrackDiffusion, a novel video generation framework affording fine-grained trajectory-conditioned motion control. A pivotal component of TrackDiffusion is the instance enhancer, which explicitly ensures inter-frame consistency of multiple objects. generated video sequences by our TrackDiffusion can be used as training data for visual perception models.
  arXiv  Detail & Related papers  (2023-12-01T15:24:38Z)
• Now You See Me: Robust approach to Partial Occlusions [0.15229257192293202]
  Occlusions of objects is one of the indispensable problems in Computer vision. This paper introduces our own synthetically created dataset by utilising Stanford Car dataset. We conduct a comprehensive analysis using various state of the art CNN models such as VGG-19, ResNet 50/101, GoogleNet, DenseNet 121.
  arXiv  Detail & Related papers  (2023-04-24T00:31:49Z)
• Boundary-Denoising for Video Activity Localization [57.9973253014712]
  We study the video activity localization problem from a denoising perspective. Specifically, we propose an encoder-decoder model named DenoiseLoc. Experiments show that DenoiseLoc advances %in several video activity understanding tasks.
  arXiv  Detail & Related papers  (2023-04-06T08:48:01Z)
• OTPose: Occlusion-Aware Transformer for Pose Estimation in Sparsely-Labeled Videos [21.893572076171527]
  We propose a method that leverages an attention mask for occluded joints and encodes temporal dependency between frames using transformers. We achieve state-of-the-art pose estimation results for PoseTrack 2017 and PoseTrack 2018 datasets.
  arXiv  Detail & Related papers  (2022-07-20T08:06:06Z)
• RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream [27.600873320989276]
  High-quality 4D reconstruction of human performance with complex interactions to various objects is essential in real-world scenarios. Recent advances still fail to provide reliable performance reconstruction. We propose RobustFusion, a robust volumetric performance reconstruction system for human-object interaction scenarios.
  arXiv  Detail & Related papers  (2021-04-30T08:41:45Z)
• Generative Partial Visual-Tactile Fused Object Clustering [81.17645983141773]
  We propose a Generative Partial Visual-Tactile Fused (i.e., GPVTF) framework for object clustering. A conditional cross-modal clustering generative adversarial network is then developed to synthesize one modality conditioning on the other modality. To the end, two pseudo-label based KL-divergence losses are employed to update the corresponding modality-specific encoders.
  arXiv  Detail & Related papers  (2020-12-28T02:37:03Z)
• Robust Instance Segmentation through Reasoning about Multi-Object Occlusion [9.536947328412198]
  We propose a deep network for multi-object instance segmentation that is robust to occlusion. Our work builds on Compositional Networks, which learn a generative model of neural feature activations to locate occluders. In particular, we obtain feed-forward predictions of the object classes and their instance and occluder segmentations.
  arXiv  Detail & Related papers  (2020-12-03T17:41:55Z)
• FenceMask: A Data Augmentation Approach for Pre-extracted Image Features [18.299882139724684]
  We propose a novel data augmentation method named 'FenceMask' It exhibits outstanding performance in various computer vision tasks. Our method achieved significant performance improvement on Fine-Grained Visual Categorization task and VisDrone dataset.
  arXiv  Detail & Related papers  (2020-06-14T12:16:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.