Hamiltonian prior to Disentangle Content and Motion in Image Sequences

• URL: http://arxiv.org/abs/2112.01641v1
• Date: Thu, 2 Dec 2021 23:41:12 GMT
• Title: Hamiltonian prior to Disentangle Content and Motion in Image Sequences
• Authors: Asif Khan, Amos Storkey
• Abstract summary: We present a deep latent variable model for high dimensional sequential data. We split the motion space into subspaces, and introduce a unique Hamiltonian operator for each subspace. The explicit split of the motion space decomposes the Hamiltonian into symmetry groups and gives long-term separability.
• Score: 2.2133187119466116
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a deep latent variable model for high dimensional sequential data. Our model factorises the latent space into content and motion variables. To model the diverse dynamics, we split the motion space into subspaces, and introduce a unique Hamiltonian operator for each subspace. The Hamiltonian formulation provides reversible dynamics that learn to constrain the motion path to conserve invariant properties. The explicit split of the motion space decomposes the Hamiltonian into symmetry groups and gives long-term separability of the dynamics. This split also means representations can be learnt that are easy to interpret and control. We demonstrate the utility of our model for swapping the motion of two videos, generating sequences of various actions from a given image and unconditional sequence generation.

Related papers

• SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation [56.90807453045657]
  SynMotion is a motion-customized video generation model that jointly leverages semantic guidance and visual adaptation.<n>At the semantic level, we introduce the dual-em semantic comprehension mechanism which disentangles subject and motion representations.<n>At the visual level, we integrate efficient motion adapters into a pre-trained video generation model to enhance motion fidelity and temporal coherence.
  arXiv  Detail & Related papers  (2025-06-30T10:09:32Z)
• AnyTop: Character Animation Diffusion with Any Topology [54.07731933876742]
  We introduce AnyTop, a diffusion model that generates motions for diverse characters with distinct motion dynamics. Our work features a transformer-based denoising network, tailored for arbitrary skeleton learning. Our evaluation demonstrates that AnyTops well, even with as few as three training examples per topology, and can produce motions for unseen skeletons as well.
  arXiv  Detail & Related papers  (2025-02-24T17:00:36Z)
• Latent Space Energy-based Neural ODEs [73.01344439786524]
  This paper introduces a novel family of deep dynamical models designed to represent continuous-time sequence data. We train the model using maximum likelihood estimation with Markov chain Monte Carlo. Experiments on oscillating systems, videos and real-world state sequences (MuJoCo) illustrate that ODEs with the learnable energy-based prior outperform existing counterparts.
  arXiv  Detail & Related papers  (2024-09-05T18:14:22Z)
• MotionCrafter: One-Shot Motion Customization of Diffusion Models [66.44642854791807]
  We introduce MotionCrafter, a one-shot instance-guided motion customization method. MotionCrafter employs a parallel spatial-temporal architecture that injects the reference motion into the temporal component of the base model. During training, a frozen base model provides appearance normalization, effectively separating appearance from motion.
  arXiv  Detail & Related papers  (2023-12-08T16:31:04Z)
• Hamiltonian GAN [1.6589012298747952]
  We present a GAN-based video generation pipeline with a learned configuration space map and Hamiltonian neural network motion model. We train our model with a physics-inspired cyclic-inspired loss function which encourages a minimal representation of the configuration space and improves interpretability.
  arXiv  Detail & Related papers  (2023-08-22T06:03:00Z)
• We never go out of Style: Motion Disentanglement by Subspace Decomposition of Latent Space [38.54517335215281]
  We propose a novel method to decompose motion in videos by using a pretrained image GAN model. We discover disentangled motion subspaces in the latent space of widely used style-based GAN models. We evaluate the disentanglement properties of motion subspaces on face and car datasets.
  arXiv  Detail & Related papers  (2023-06-01T11:18:57Z)
• MoDi: Unconditional Motion Synthesis from Diverse Data [51.676055380546494]
  We present MoDi, an unconditional generative model that synthesizes diverse motions. Our model is trained in a completely unsupervised setting from a diverse, unstructured and unlabeled motion dataset. We show that despite the lack of any structure in the dataset, the latent space can be semantically clustered.
  arXiv  Detail & Related papers  (2022-06-16T09:06:25Z)
• NeMF: Neural Motion Fields for Kinematic Animation [6.570955948572252]
  We express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF) We use a neural network to learn this function for miscellaneous sets of motions. We train our model with diverse human motion dataset and quadruped dataset to prove its versatility.
  arXiv  Detail & Related papers  (2022-06-04T05:53:27Z)
• SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision [73.26414295633846]
  A recently proposed class of models attempts to learn latent dynamics from high-dimensional observations. Existing methods rely on image reconstruction quality, which does not always reflect the quality of the learnt latent dynamics. We develop a set of new measures, including a binary indicator of whether the underlying Hamiltonian dynamics have been faithfully captured.
  arXiv  Detail & Related papers  (2021-11-10T23:26:58Z)
• Hierarchical Style-based Networks for Motion Synthesis [150.226137503563]
  We propose a self-supervised method for generating long-range, diverse and plausible behaviors to achieve a specific goal location. Our proposed method learns to model the motion of human by decomposing a long-range generation task in a hierarchical manner. On large-scale skeleton dataset, we show that the proposed method is able to synthesise long-range, diverse and plausible motion.
  arXiv  Detail & Related papers  (2020-08-24T02:11:02Z)
• Graph Gamma Process Generalized Linear Dynamical Systems [60.467040479276704]
  We introduce graph gamma process (GGP) linear dynamical systems to model real multivariate time series. For temporal pattern discovery, the latent representation under the model is used to decompose the time series into a parsimonious set of multivariate sub-sequences. We use the generated random graph, whose number of nonzero-degree nodes is finite, to define both the sparsity pattern and dimension of the latent state transition matrix.
  arXiv  Detail & Related papers  (2020-07-25T04:16:34Z)
• Entanglement Dynamics of Random GUE Hamiltonians [0.0]
  We study dynamics of entanglement assuming that the overall time-evolution is governed by non-integrable Hamiltonians. We derive universal average time evolution of the reduced density matrix and the purity. We find general expressions for exponential $n$-point correlation functions in the gas of GUE eigenvalues.
  arXiv  Detail & Related papers  (2020-01-01T05:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.