Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation

• URL: http://arxiv.org/abs/2303.14926v1
• Date: Mon, 27 Mar 2023 05:53:01 GMT
• Title: Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation
• Authors: Clinton Ansun Mo, Kun Hu, Chengjiang Long, Zhiyong Wang
• Abstract summary: We propose a novel framework to formulate latent motion with precision-based constraints. Our proposed method demonstrates both superior datasets accuracy and high visual similarity to ground truth motions.
• Score: 20.314332409748637
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deriving sophisticated 3D motions from sparse keyframes is a particularly challenging problem, due to continuity and exceptionally skeletal precision. The action features are often derivable accurately from the full series of keyframes, and thus, leveraging the global context with transformers has been a promising data-driven embedding approach. However, existing methods are often with inputs of interpolated intermediate frame for continuity using basic interpolation methods with keyframes, which result in a trivial local minimum during training. In this paper, we propose a novel framework to formulate latent motion manifolds with keyframe-based constraints, from which the continuous nature of intermediate token representations is considered. Particularly, our proposed framework consists of two stages for identifying a latent motion subspace, i.e., a keyframe encoding stage and an intermediate token generation stage, and a subsequent motion synthesis stage to extrapolate and compose motion data from manifolds. Through our extensive experiments conducted on both the LaFAN1 and CMU Mocap datasets, our proposed method demonstrates both superior interpolation accuracy and high visual similarity to ground truth motions.

Related papers

• Thin-Plate Spline-based Interpolation for Animation Line Inbetweening [54.69811179222127]
  Chamfer Distance (CD) is commonly adopted for evaluating inbetweening performance. We propose a simple yet effective method for animation line inbetweening that adopts thin-plate spline-based transformation. Our method outperforms existing approaches by delivering high-quality results with enhanced fluidity.
  arXiv  Detail & Related papers  (2024-08-17T08:05:31Z)
• Flexible Motion In-betweening with Diffusion Models [16.295323675781184]
  We investigate the potential of diffusion models in generating diverse human motions guided by compares. Unlike previous inbetweening methods, we propose a simple unified model capable of generating precise and diverse motions. We evaluate the performance of CondMDI on the text-conditioned HumanML3D dataset.
  arXiv  Detail & Related papers  (2024-05-17T23:55:51Z)
• Motion-aware Latent Diffusion Models for Video Frame Interpolation [51.78737270917301]
  Motion estimation between neighboring frames plays a crucial role in avoiding motion ambiguity. We propose a novel diffusion framework, motion-aware latent diffusion models (MADiff) Our method achieves state-of-the-art performance significantly outperforming existing approaches.
  arXiv  Detail & Related papers  (2024-04-21T05:09:56Z)
• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• Modeling Continuous Motion for 3D Point Cloud Object Tracking [54.48716096286417]
  This paper presents a novel approach that views each tracklet as a continuous stream. At each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank. To enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed.
  arXiv  Detail & Related papers  (2023-03-14T02:58:27Z)
• Meta-Interpolation: Time-Arbitrary Frame Interpolation via Dual Meta-Learning [65.85319901760478]
  We consider processing different time-steps with adaptively generated convolutional kernels in a unified way with the help of meta-learning. We develop a dual meta-learned frame framework to synthesize intermediate frames with the guidance of context information and optical flow.
  arXiv  Detail & Related papers  (2022-07-27T17:36:23Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• Video Frame Interpolation via Generalized Deformable Convolution [18.357839820102683]
  Video frame aims at synthesizing intermediate frames from nearby source frames while maintaining spatial and temporal consistencies. Existing deeplearning-based video frame methods can be divided into two categories: flow-based methods and kernel-based methods. A novel mechanism named generalized deformable convolution is proposed, which can effectively learn motion in a data-driven manner and freely select sampling points in space-time.
  arXiv  Detail & Related papers  (2020-08-24T20:00:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.