Self-Supervised Approach for Facial Movement Based Optical Flow

• URL: http://arxiv.org/abs/2105.01256v1
• Date: Tue, 4 May 2021 02:38:11 GMT
• Title: Self-Supervised Approach for Facial Movement Based Optical Flow
• Authors: Muhannad Alkaddour, Usman Tariq, Abhinav Dhall
• Abstract summary: We generate optical flow ground truth for face images using facial key-points. We train the FlowNetS architecture to test its performance on the generated dataset. The optical flow obtained using this work has promising applications in facial expression analysis.
• Score: 8.19666118455293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computing optical flow is a fundamental problem in computer vision. However, deep learning-based optical flow techniques do not perform well for non-rigid movements such as those found in faces, primarily due to lack of the training data representing the fine facial motion. We hypothesize that learning optical flow on face motion data will improve the quality of predicted flow on faces. The aim of this work is threefold: (1) exploring self-supervised techniques to generate optical flow ground truth for face images; (2) computing baseline results on the effects of using face data to train Convolutional Neural Networks (CNN) for predicting optical flow; and (3) using the learned optical flow in micro-expression recognition to demonstrate its effectiveness. We generate optical flow ground truth using facial key-points in the BP4D-Spontaneous dataset. The generated optical flow is used to train the FlowNetS architecture to test its performance on the generated dataset. The performance of FlowNetS trained on face images surpassed that of other optical flow CNN architectures, demonstrating its usefulness. Our optical flow features are further compared with other methods using the STSTNet micro-expression classifier, and the results indicate that the optical flow obtained using this work has promising applications in facial expression analysis.

Related papers

• FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model [15.525822826375371]
  This paper proposes FacialFlowNet (FFN), a novel large-scale facial optical flow dataset, and the Decomposed Facial Flow Model (DecFlow) FFN comprises 9,635 identities and 105,970 image pairs, offering unprecedented diversity for detailed facial and head motion analysis. DecFlow features a facial semantic-aware decoder, excelling in accurately decomposing facial flow into head and expression components.
  arXiv  Detail & Related papers  (2024-09-09T07:49:13Z)
• Vision-Informed Flow Image Super-Resolution with Quaternion Spatial Modeling and Dynamic Flow Convolution [49.45309818782329]
  Flow image super-resolution (FISR) aims at recovering high-resolution turbulent velocity fields from low-resolution flow images. Existing FISR methods mainly process the flow images in natural image patterns. We propose the first flow visual property-informed FISR algorithm.
  arXiv  Detail & Related papers  (2024-01-29T06:48:16Z)
• UFD-PRiME: Unsupervised Joint Learning of Optical Flow and Stereo Depth through Pixel-Level Rigid Motion Estimation [4.445751695675388]
  Both optical flow and stereo disparities are image matches and can therefore benefit from joint training. We design a first network that estimates flow and disparity jointly and is trained without supervision. A second network, trained with optical flow from the first as pseudo-labels, takes disparities from the first network, estimates 3D rigid motion at every pixel, and reconstructs optical flow again.
  arXiv  Detail & Related papers  (2023-10-07T07:08:25Z)
• Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow [25.23550076996421]
  We propose to leverage the geometric connection between optical flow estimation and stereo matching. We turn the monocular depth datasets into stereo ones via virtual disparity. We also introduce virtual camera motion into stereo data to produce additional flows along the vertical direction.
  arXiv  Detail & Related papers  (2023-10-03T06:56:07Z)
• Neuromorphic Optical Flow and Real-time Implementation with Event Cameras [47.11134388304464]
  We build on the latest developments in event-based vision and spiking neural networks. We propose a new network architecture that improves the state-of-the-art self-supervised optical flow accuracy. We demonstrate high speed optical flow prediction with almost two orders of magnitude reduced complexity.
  arXiv  Detail & Related papers  (2023-04-14T14:03:35Z)
• Deep 360$^\circ$ Optical Flow Estimation Based on Multi-Projection Fusion [10.603670927163002]
  This paper focuses on the 360$circ$ optical flow estimation using deep neural networks to support increasingly popular VR applications. We propose a novel multi-projection fusion framework that fuses the optical flow predicted by the models trained using different projection methods. We also build the first large-scale panoramic optical flow dataset to support the training of neural networks and the evaluation of panoramic optical flow estimation methods.
  arXiv  Detail & Related papers  (2022-07-27T16:48:32Z)
• Sensor-Guided Optical Flow [53.295332513139925]
  This paper proposes a framework to guide an optical flow network with external cues to achieve superior accuracy on known or unseen domains. We show how these can be obtained by combining depth measurements from active sensors with geometry and hand-crafted optical flow algorithms.
  arXiv  Detail & Related papers  (2021-09-30T17:59:57Z)
• Dense Optical Flow from Event Cameras [55.79329250951028]
  We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras. Our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC.
  arXiv  Detail & Related papers  (2021-08-24T07:39:08Z)
• PCA Event-Based Otical Flow for Visual Odometry [0.0]
  We present a Principal Component Analysis approach to the problem of event-based optical flow estimation. We show that the best variant of our proposed method, dedicated to the real-time context of visual odometry, is about two times faster compared to state-of-the-art implementations.
  arXiv  Detail & Related papers  (2021-05-08T18:30:44Z)
• Learning optical flow from still images [53.295332513139925]
  We introduce a framework to generate accurate ground-truth optical flow annotations quickly and in large amounts from any readily available single real picture. We virtually move the camera in the reconstructed environment with known motion vectors and rotation angles. When trained with our data, state-of-the-art optical flow networks achieve superior generalization to unseen real data.
  arXiv  Detail & Related papers  (2021-04-08T17:59:58Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.