Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition

• URL: http://arxiv.org/abs/2204.07636v2
• Date: Mon, 15 Jan 2024 11:11:23 GMT
• Title: Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition
• Authors: Philipp Flotho, Cosmas Heiss, Gabriele Steidl, Daniel J. Strauss
• Abstract summary: We propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is three-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition.
• Score: 2.1028463367241033
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this paper, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is three-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.

Related papers

• Textual Decomposition Then Sub-motion-space Scattering for Open-Vocabulary Motion Generation [74.94730615777212]
  Text-to-motion generation is a crucial task in computer vision, which generates the target 3D motion by the given text. The current annotated dataset's limited scale only allows them to achieve mapping from sub-text-space to sub-motion-space. This paper proposes to leverage the atomic motion as an intermediate representation, and leverage two orderly coupled steps, i.e., Textual Decomposition and Sub-motion-space Scattering.
  arXiv  Detail & Related papers  (2024-11-06T17:57:43Z)
• SpotFormer: Multi-Scale Spatio-Temporal Transformer for Facial Expression Spotting [11.978551396144532]
  In this paper, we propose an efficient framework for facial expression spotting. First, we propose a Sliding Window-based Multi-Resolution Optical flow (SW-MRO) feature, which calculates multi-resolution optical flow of the input sequence within compact sliding windows. Second, we propose SpotFormer, a multi-scale-temporal Transformer that simultaneously encodes facial-temporal relationships of the SW-MRO features for accurate frame-level probability estimation. Third, we introduce supervised contrastive learning into SpotFormer to enhance the discriminability between different types of expressions.
  arXiv  Detail & Related papers  (2024-07-30T13:02:08Z)
• Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring [71.60457491155451]
  Eliminating image blur produced by various kinds of motion has been a challenging problem. We propose a novel real-world deblurring filtering model called the Motion-adaptive Separable Collaborative Filter. Our method provides an effective solution for real-world motion blur removal and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-19T19:44:24Z)
• Gyroscope-Assisted Motion Deblurring Network [11.404195533660717]
  This paper presents a framework to synthetic and restore motion blur images using Inertial Measurement Unit (IMU) data. The framework includes a strategy for training triplet generation, and a Gyroscope-Aided Motion Deblurring (GAMD) network for blurred image restoration.
  arXiv  Detail & Related papers  (2024-02-10T01:30:24Z)
• Decouple Content and Motion for Conditional Image-to-Video Generation [6.634105805557556]
  conditional image-to-video (cI2V) generation is to create a believable new video by beginning with the condition, i.e., one image and text. Previous cI2V generation methods conventionally perform in RGB pixel space, with limitations in modeling motion consistency and visual continuity. We propose a novel approach by disentangling the target RGB pixels into two distinct components: spatial content and temporal motions.
  arXiv  Detail & Related papers  (2023-11-24T06:08:27Z)
• ExBluRF: Efficient Radiance Fields for Extreme Motion Blurred Images [58.24910105459957]
  We present ExBluRF, a novel view synthesis method for extreme motion blurred images. Our approach consists of two main components: 6-DOF camera trajectory-based motion blur formulation and voxel-based radiance fields. Compared with the existing works, our approach restores much sharper 3D scenes with the order of 10 times less training time and GPU memory consumption.
  arXiv  Detail & Related papers  (2023-09-16T11:17:25Z)
• 3D Motion Magnification: Visualizing Subtle Motions with Time Varying Radiance Fields [58.6780687018956]
  We present a 3D motion magnification method that can magnify subtle motions from scenes captured by a moving camera. We represent the scene with time-varying radiance fields and leverage the Eulerian principle for motion magnification. We evaluate the effectiveness of our method on both synthetic and real-world scenes captured under various camera setups.
  arXiv  Detail & Related papers  (2023-08-07T17:59:59Z)
• DFR: Depth from Rotation by Uncalibrated Image Rectification with Latitudinal Motion Assumption [6.369764116066747]
  We propose Depth-from-Rotation (DfR), a novel image rectification solution for uncalibrated rotating cameras. Specifically, we model the motion of a rotating camera as the camera rotates on a sphere with fixed latitude. We derive a 2-point analytical solver from directly computing the rectified transformations on the two images.
  arXiv  Detail & Related papers  (2023-07-11T09:11:22Z)
• MorphGANFormer: Transformer-based Face Morphing and De-Morphing [55.211984079735196]
  StyleGAN-based approaches to face morphing are among the leading techniques. We propose a transformer-based alternative to face morphing and demonstrate its superiority to StyleGAN-based methods.
  arXiv  Detail & Related papers  (2023-02-18T19:09:11Z)
• Grasping the Arrow of Time from the Singularity: Decoding Micromotion in Low-dimensional Latent Spaces from StyleGAN [105.99762358450633]
  We show that "micromotion" can be represented in low-rank spaces extracted from latent space of StyleGAN-v2 model for face generation. It can be represented as simple as an affine transformation over its latent feature. It demonstrates that the local feature geometry corresponding to one type of micromotion is aligned across different face subjects.
  arXiv  Detail & Related papers  (2022-04-27T04:38:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.