Learning In-between Imagery Dynamics via Physical Latent Spaces

• URL: http://arxiv.org/abs/2310.09495v1
• Date: Sat, 14 Oct 2023 05:14:51 GMT
• Title: Learning In-between Imagery Dynamics via Physical Latent Spaces
• Authors: Jihun Han, Yoonsang Lee, Anne Gelb
• Abstract summary: We present a framework designed to learn the underlying dynamics between two images observed at consecutive time steps. By incorporating a latent variable that follows a physical model expressed in partial differential equations (PDEs), our approach ensures the interpretability of the learned model. We demonstrate the robustness and effectiveness of our learning framework through a series of numerical tests using geoscientific imagery data.
• Score: 0.7366405857677226
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a framework designed to learn the underlying dynamics between two images observed at consecutive time steps. The complex nature of image data and the lack of temporal information pose significant challenges in capturing the unique evolving patterns. Our proposed method focuses on estimating the intermediary stages of image evolution, allowing for interpretability through latent dynamics while preserving spatial correlations with the image. By incorporating a latent variable that follows a physical model expressed in partial differential equations (PDEs), our approach ensures the interpretability of the learned model and provides insight into corresponding image dynamics. We demonstrate the robustness and effectiveness of our learning framework through a series of numerical tests using geoscientific imagery data.

Related papers

• Deep ContourFlow: Advancing Active Contours with Deep Learning [3.9948520633731026]
  We present a framework for both unsupervised and one-shot approaches for image segmentation. It is capable of capturing complex object boundaries without the need for extensive labeled training data. This is particularly required in histology, a field facing a significant shortage of annotations.
  arXiv  Detail & Related papers  (2024-07-15T13:12:34Z)
• Single-shot Tomography of Discrete Dynamic Objects [1.1407697960152927]
  We present a novel method for the reconstruction of high-resolution temporal images in dynamic tomographic imaging. The implications of this research extend to improved visualization and analysis of dynamic processes in tomographic imaging.
  arXiv  Detail & Related papers  (2023-11-09T10:52:02Z)
• Latent Traversals in Generative Models as Potential Flows [113.4232528843775]
  We propose to model latent structures with a learned dynamic potential landscape. Inspired by physics, optimal transport, and neuroscience, these potential landscapes are learned as physically realistic partial differential equations. Our method achieves both more qualitatively and quantitatively disentangled trajectories than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-25T15:53:45Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Perception Over Time: Temporal Dynamics for Robust Image Understanding [5.584060970507506]
  Deep learning surpasses human-level performance in narrow and specific vision tasks. Human visual perception is orders of magnitude more robust to changes in the input stimulus. We introduce a novel method of incorporating temporal dynamics into static image understanding.
  arXiv  Detail & Related papers  (2022-03-11T21:11:59Z)
• NeurInt : Learning to Interpolate through Neural ODEs [18.104328632453676]
  We propose a novel generative model that learns a distribution of trajectories between two images. We demonstrate our approach's effectiveness in generating images improved quality as well as its ability to learn a diverse distribution over smooth trajectories for any pair of real source and target images.
  arXiv  Detail & Related papers  (2021-11-07T16:31:18Z)
• Moment evolution equations and moment matching for stochastic image EPDiff [68.97335984455059]
  Models of image deformation allow study of time-continuous effects transforming images by deforming the image domain. Applications include medical image analysis with both population trends and random subject specific variation. We use moment approximations of the corresponding Ito diffusion to construct estimators for statistical inference in the parameters full model.
  arXiv  Detail & Related papers  (2021-10-07T11:08:11Z)
• Learning Temporal Dynamics from Cycles in Narrated Video [85.89096034281694]
  We propose a self-supervised solution to the problem of learning to model how the world changes as time elapses. Our model learns modality-agnostic functions to predict forward and backward in time, which must undo each other when composed. We apply the learned dynamics model without further training to various tasks, such as predicting future action and temporally ordering sets of images.
  arXiv  Detail & Related papers  (2021-01-07T02:41:32Z)
• Heteroscedastic Uncertainty for Robust Generative Latent Dynamics [7.107159120605662]
  We present a method to jointly learn a latent state representation and the associated dynamics. As our main contribution, we describe how our representation is able to capture a notion of heteroscedastic or input-specific uncertainty. We present results from prediction and control experiments on two image-based tasks.
  arXiv  Detail & Related papers  (2020-08-18T21:04:33Z)
• Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond [62.730497582218284]
  We develop a new deep learning based framework to optimize a diffeomorphic model via multi-scale propagation. We conduct two groups of image registration experiments on 3D volume datasets including image-to-atlas registration on brain MRI data and image-to-image registration on liver CT data.
  arXiv  Detail & Related papers  (2020-04-30T03:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.