Don't Judge by the Look: Towards Motion Coherent Video Representation

• URL: http://arxiv.org/abs/2403.09506v2
• Date: Mon, 25 Mar 2024 02:45:35 GMT
• Title: Don't Judge by the Look: Towards Motion Coherent Video Representation
• Authors: Yitian Zhang, Yue Bai, Huan Wang, Yizhou Wang, Yun Fu,
• Abstract summary: Motion Coherent Augmentation (MCA) is a data augmentation method for video understanding. MCA introduces appearance variation in videos and implicitly encourages the model to prioritize motion patterns, rather than static appearances.
• Score: 56.09346222721583
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current training pipelines in object recognition neglect Hue Jittering when doing data augmentation as it not only brings appearance changes that are detrimental to classification, but also the implementation is inefficient in practice. In this study, we investigate the effect of hue variance in the context of video understanding and find this variance to be beneficial since static appearances are less important in videos that contain motion information. Based on this observation, we propose a data augmentation method for video understanding, named Motion Coherent Augmentation (MCA), that introduces appearance variation in videos and implicitly encourages the model to prioritize motion patterns, rather than static appearances. Concretely, we propose an operation SwapMix to efficiently modify the appearance of video samples, and introduce Variation Alignment (VA) to resolve the distribution shift caused by SwapMix, enforcing the model to learn appearance invariant representations. Comprehensive empirical evaluation across various architectures and different datasets solidly validates the effectiveness and generalization ability of MCA, and the application of VA in other augmentation methods. Code is available at https://github.com/BeSpontaneous/MCA-pytorch.

Related papers

• Stem-OB: Generalizable Visual Imitation Learning with Stem-Like Convergent Observation through Diffusion Inversion [18.990678061962825]
  We propose Stem-OB that utilizes pretrained image diffusion models to suppress low-level visual differences. This image inversion process is akin to transforming the observation into a shared representation. Our method is a simple yet highly effective plug-and-play solution.
  arXiv  Detail & Related papers  (2024-11-07T17:56:16Z)
• RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability [25.943330238941602]
  We propose a visual model-based RL method that learns a latent representation resilient to spurious variations. Our training objective encourages the representation to be maximally predictive of dynamics and reward. Our effort is a step towards making model-based RL a practical and useful tool for dynamic, diverse domains.
  arXiv  Detail & Related papers  (2023-08-31T18:43:04Z)
• Unsupervised Video Anomaly Detection with Diffusion Models Conditioned on Compact Motion Representations [17.816344808780965]
  unsupervised video anomaly detection (VAD) problem involves classifying each frame in a video as normal or abnormal, without any access to labels. To accomplish this, proposed method employs conditional diffusion models, where the input data is features extracted from pre-trained network. Our method utilizes a data-driven threshold and considers a high reconstruction error as an indicator of anomalous events.
  arXiv  Detail & Related papers  (2023-07-04T07:36:48Z)
• EAN: Event Adaptive Network for Enhanced Action Recognition [66.81780707955852]
  We propose a unified action recognition framework to investigate the dynamic nature of video content. First, when extracting local cues, we generate the spatial-temporal kernels of dynamic-scale to adaptively fit the diverse events. Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer.
  arXiv  Detail & Related papers  (2021-07-22T15:57:18Z)
• Consistency Regularization for Variational Auto-Encoders [14.423556966548544]
  Variational auto-encoders (VAEs) are a powerful approach to unsupervised learning. We propose a regularization method to enforce consistency in VAEs.
  arXiv  Detail & Related papers  (2021-05-31T10:26:32Z)
• Composable Augmentation Encoding for Video Representation Learning [94.2358972764708]
  We focus on contrastive methods for self-supervised video representation learning. A common paradigm in contrastive learning is to construct positive pairs by sampling different data views for the same instance, with different data instances as negatives. We propose an 'augmentation aware' contrastive learning framework, where we explicitly provide a sequence of augmentation parameterisations. We show that our method encodes valuable information about specified spatial or temporal augmentation, and in doing so also achieve state-of-the-art performance on a number of video benchmarks.
  arXiv  Detail & Related papers  (2021-04-01T16:48:53Z)
• Learning Representational Invariances for Data-Efficient Action Recognition [52.23716087656834]
  We show that our data augmentation strategy leads to promising performance on the Kinetics-100, UCF-101, and HMDB-51 datasets. We also validate our data augmentation strategy in the fully supervised setting and demonstrate improved performance.
  arXiv  Detail & Related papers  (2021-03-30T17:59:49Z)
• DecAug: Augmenting HOI Detection via Decomposition [54.65572599920679]
  Current algorithms suffer from insufficient training samples and category imbalance within datasets. We propose an efficient and effective data augmentation method called DecAug for HOI detection. Experiments show that our method brings up to 3.3 mAP and 1.6 mAP improvements on V-COCO and HICODET dataset.
  arXiv  Detail & Related papers  (2020-10-02T13:59:05Z)
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases [34.02639091680309]
  Recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet.
  arXiv  Detail & Related papers  (2020-07-28T00:11:31Z)
• TransMoMo: Invariance-Driven Unsupervised Video Motion Retargeting [107.39743751292028]
  TransMoMo is capable of transferring motion of a person in a source video realistically to another video of a target person. We exploit invariance properties of three factors of variation including motion, structure, and view-angle. We demonstrate the effectiveness of our method over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-31T17:49:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.