Learning a Generative Motion Model from Image Sequences based on a Latent Motion Matrix

• URL: http://arxiv.org/abs/2011.01741v2
• Date: Sun, 31 Jan 2021 13:26:54 GMT
• Title: Learning a Generative Motion Model from Image Sequences based on a Latent Motion Matrix
• Authors: Julian Krebs, Herv\'e Delingette, Nicholas Ayache and Tommaso Mansi
• Abstract summary: We learn a probabilistic motion model from simulating temporal-temporal registration in a sequence of images. We show improved registration accuracy-temporally smoother consistencys compared to three state-of-the-art registration algorithms. We also demonstrate the model's applicability for motion analysis, simulation and super-resolution by an improved motion reconstruction from sequences with missing frames.
• Score: 8.774604259603302
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space - the motion matrix - which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation. More precisely, the motion matrix allows to transport the recovered motion from one subject to another simulating for example a pathological motion in a healthy subject without the need for inter-subject registration. The method is based on a conditional latent variable model that is trained using amortized variational inference. This unsupervised generative model follows a novel multivariate Gaussian process prior and is applied within a temporal convolutional network which leads to a diffeomorphic motion model. Temporal consistency and generalizability is further improved by applying a temporal dropout training scheme. Applied to cardiac cine-MRI sequences, we show improved registration accuracy and spatio-temporally smoother deformations compared to three state-of-the-art registration algorithms. Besides, we demonstrate the model's applicability for motion analysis, simulation and super-resolution by an improved motion reconstruction from sequences with missing frames compared to linear and cubic interpolation.

Related papers

• Sequential Posterior Sampling with Diffusion Models [15.028061496012924]
  We propose a novel approach that models the transition dynamics to improve the efficiency of sequential diffusion posterior sampling in conditional image synthesis. We demonstrate the effectiveness of our approach on a real-world dataset of high frame rate cardiac ultrasound images. Our method opens up new possibilities for real-time applications of diffusion models in imaging and other domains requiring real-time inference.
  arXiv  Detail & Related papers  (2024-09-09T07:55:59Z)
• Generalizable Implicit Motion Modeling for Video Frame Interpolation [51.966062283735596]
  Motion is critical in flow-based Video Frame Interpolation (VFI) We introduce General Implicit Motion Modeling (IMM), a novel and effective approach to motion modeling VFI. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion.
  arXiv  Detail & Related papers  (2024-07-11T17:13:15Z)
• Motion Flow Matching for Human Motion Synthesis and Editing [75.13665467944314]
  We propose emphMotion Flow Matching, a novel generative model for human motion generation featuring efficient sampling and effectiveness in motion editing applications. Our method reduces the sampling complexity from thousand steps in previous diffusion models to just ten steps, while achieving comparable performance in text-to-motion and action-to-motion generation benchmarks.
  arXiv  Detail & Related papers  (2023-12-14T12:57:35Z)
• Executing your Commands via Motion Diffusion in Latent Space [51.64652463205012]
  We propose a Motion Latent-based Diffusion model (MLD) to produce vivid motion sequences conforming to the given conditional inputs. Our MLD achieves significant improvements over the state-of-the-art methods among extensive human motion generation tasks.
  arXiv  Detail & Related papers  (2022-12-08T03:07:00Z)
• JNMR: Joint Non-linear Motion Regression for Video Frame Interpolation [47.123769305867775]
  Video frame (VFI) aims to generate frames by warping learnable motions from the bidirectional historical references. We reformulate VFI as a Joint Non-linear Motion Regression (JNMR) strategy to model the complicated motions of inter-frame. We show that the effectiveness and significant improvement of joint motion regression compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-06-09T02:47:29Z)
• Dynamic imaging using Motion-Compensated SmooThness Regularization on Manifolds (MoCo-SToRM) [19.70386996879205]
  We introduce an unsupervised motion-compensated reconstruction scheme for high-resolution free-breathing pulmonary MRI. We model the image frames in the time series as the deformed version of the 3D template image volume. We assume the deformation maps to be points on a smooth manifold in high-dimensional space.
  arXiv  Detail & Related papers  (2021-12-06T22:04:57Z)
• Dynamic imaging using motion-compensated smoothness regularization on manifolds (MoCo-SToRM) [23.093076134206513]
  We introduce an unsupervised deep manifold learning algorithm for motion-compensated dynamic MRI. The utility of the algorithm is demonstrated in the context of motion-compensated high-resolution lung MRI.
  arXiv  Detail & Related papers  (2021-11-21T19:52:01Z)
• Conditional Temporal Variational AutoEncoder for Action Video Prediction [66.63038712306606]
  ACT-VAE predicts pose sequences for an action clips from a single input image. When connected with a plug-and-play Pose-to-Image (P2I) network, ACT-VAE can synthesize image sequences.
  arXiv  Detail & Related papers  (2021-08-12T10:59:23Z)
• HuMoR: 3D Human Motion Model for Robust Pose Estimation [100.55369985297797]
  HuMoR is a 3D Human Motion Model for Robust Estimation of temporal pose and shape. We introduce a conditional variational autoencoder, which learns a distribution of the change in pose at each step of a motion sequence. We demonstrate that our model generalizes to diverse motions and body shapes after training on a large motion capture dataset.
  arXiv  Detail & Related papers  (2021-05-10T21:04:55Z)
• Graph-based Normalizing Flow for Human Motion Generation and Reconstruction [20.454140530081183]
  We propose a probabilistic generative model to synthesize and reconstruct long horizon motion sequences conditioned on past information and control signals. We evaluate the models on a mixture of motion capture datasets of human locomotion with foot-step and bone-length analysis.
  arXiv  Detail & Related papers  (2021-04-07T09:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.