Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture

• URL: http://arxiv.org/abs/2403.07347v2
• Date: Sun, 24 Mar 2024 09:22:13 GMT
• Title: Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture
• Authors: Fei Wang, Dan Guo, Kun Li, Zhun Zhong, Meng Wang,
• Abstract summary: Video Motion Magnification aims to reveal subtle and imperceptible motion information of objects in the macroscopic world. We present FD4MM, a new paradigm of Frequency Decoupling for Motion Magnification with a Multi-level Isomorphic Architecture. We show that FD4MM reduces FLOPs by 1.63$times$ and boosts inference speed by 1.68$times$ than the latest method.
• Score: 42.51987004849891
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Motion Magnification (VMM) aims to reveal subtle and imperceptible motion information of objects in the macroscopic world. Prior methods directly model the motion field from the Eulerian perspective by Representation Learning that separates shape and texture or Multi-domain Learning from phase fluctuations. Inspired by the frequency spectrum, we observe that the low-frequency components with stable energy always possess spatial structure and less noise, making them suitable for modeling the subtle motion field. To this end, we present FD4MM, a new paradigm of Frequency Decoupling for Motion Magnification with a Multi-level Isomorphic Architecture to capture multi-level high-frequency details and a stable low-frequency structure (motion field) in video space. Since high-frequency details and subtle motions are susceptible to information degradation due to their inherent subtlety and unavoidable external interference from noise, we carefully design Sparse High/Low-pass Filters to enhance the integrity of details and motion structures, and a Sparse Frequency Mixer to promote seamless recoupling. Besides, we innovatively design a contrastive regularization for this task to strengthen the model's ability to discriminate irrelevant features, reducing undesired motion magnification. Extensive experiments on both Real-world and Synthetic Datasets show that our FD4MM outperforms SOTA methods. Meanwhile, FD4MM reduces FLOPs by 1.63$\times$ and boosts inference speed by 1.68$\times$ than the latest method. Our code is available at https://github.com/Jiafei127/FD4MM.

Related papers

• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms. SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics. Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• SMURF: Continuous Dynamics for Motion-Deblurring Radiance Fields [14.681688453270523]
  We propose sequential motion understanding radiance fields (SMURF), a novel approach that employs neural ordinary differential equation (Neural-ODE) to model continuous camera motion. Our model, rigorously evaluated against benchmark datasets, demonstrates state-of-the-art performance both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2024-03-12T11:32:57Z)
• Event-Based Motion Magnification [28.057537257958963]
  We propose a dual-camera system consisting of an event camera and a conventional RGB camera for video motion magnification. This innovative combination enables a broad and cost-effective amplification of high-frequency motions. We demonstrate the effectiveness and accuracy of our dual-camera system and network, offering a cost-effective and flexible solution for motion detection and magnification.
  arXiv  Detail & Related papers  (2024-02-19T08:59:58Z)
• Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking [52.393359791978035]
  Motion2VecSets is a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. We parameterize 4D dynamics with latent sets instead of using global latent codes. For more temporally-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames.
  arXiv  Detail & Related papers  (2024-01-12T15:05:08Z)
• EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer [30.470336098766765]
  Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective.
  arXiv  Detail & Related papers  (2023-12-07T09:10:16Z)
• Inception Transformer [151.939077819196]
  Inception Transformer, or iFormer, learns comprehensive features with both high- and low-frequency information in visual data. We benchmark the iFormer on a series of vision tasks, and showcase that it achieves impressive performance on image classification, COCO detection and ADE20K segmentation.
  arXiv  Detail & Related papers  (2022-05-25T17:59:54Z)
• Wavelet-Based Network For High Dynamic Range Imaging [64.66969585951207]
  Existing methods, such as optical flow based and end-to-end deep learning based solutions, are error-prone either in detail restoration or ghosting artifacts removal. In this work, we propose a novel frequency-guided end-to-end deep neural network (FNet) to conduct HDR fusion in the frequency domain, and Wavelet Transform (DWT) is used to decompose inputs into different frequency bands. The low-frequency signals are used to avoid specific ghosting artifacts, while the high-frequency signals are used for preserving details.
  arXiv  Detail & Related papers  (2021-08-03T12:26:33Z)
• TSI: Temporal Saliency Integration for Video Action Recognition [32.18535820790586]
  We propose a Temporal Saliency Integration (TSI) block, which mainly contains a Salient Motion Excitation (SME) module and a Cross-scale Temporal Integration (CTI) module. SME aims to highlight the motion-sensitive area through local-global motion modeling. CTI is designed to perform multi-scale temporal modeling through a group of separate 1D convolutions respectively.
  arXiv  Detail & Related papers  (2021-06-02T11:43:49Z)
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance [60.75488333935592]
  Most state-of-the-art methods heavily rely on dense optical flow as motion representation. In this paper, we shed light on fast action recognition by lifting the reliance on optical flow. We design a novel motion cue called Persistence of Appearance (PA) In contrast to optical flow, our PA focuses more on distilling the motion information at boundaries.
  arXiv  Detail & Related papers  (2020-08-08T07:09:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.