Image Morphing with Perceptual Constraints and STN Alignment

• URL: http://arxiv.org/abs/2004.14071v1
• Date: Wed, 29 Apr 2020 10:49:10 GMT
• Title: Image Morphing with Perceptual Constraints and STN Alignment
• Authors: Noa Fish, Richard Zhang, Lilach Perry, Daniel Cohen-Or, Eli Shechtman, Connelly Barnes
• Abstract summary: We propose a conditional GAN morphing framework operating on a pair of input images. A special training protocol produces sequences of frames, combined with a perceptual similarity loss, promote smooth transformation over time. We provide comparisons to classic as well as latent space morphing techniques, and demonstrate that, given a set of images for self-supervision, our network learns to generate visually pleasing morphing effects.
• Score: 70.38273150435928
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In image morphing, a sequence of plausible frames are synthesized and composited together to form a smooth transformation between given instances. Intermediates must remain faithful to the input, stand on their own as members of the set, and maintain a well-paced visual transition from one to the next. In this paper, we propose a conditional GAN morphing framework operating on a pair of input images. The network is trained to synthesize frames corresponding to temporal samples along the transformation, and learns a proper shape prior that enhances the plausibility of intermediate frames. While individual frame plausibility is boosted by the adversarial setup, a special training protocol producing sequences of frames, combined with a perceptual similarity loss, promote smooth transformation over time. Explicit stating of correspondences is replaced with a grid-based freeform deformation spatial transformer that predicts the geometric warp between the inputs, instituting the smooth geometric effect by bringing the shapes into an initial alignment. We provide comparisons to classic as well as latent space morphing techniques, and demonstrate that, given a set of images for self-supervision, our network learns to generate visually pleasing morphing effects featuring believable in-betweens, with robustness to changes in shape and texture, requiring no correspondence annotation.

Related papers

• Framer: Interactive Frame Interpolation [73.06734414930227]
  Framer targets producing smoothly transitioning frames between two images as per user creativity. Our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and the trajectory automatically.
  arXiv  Detail & Related papers  (2024-10-24T17:59:51Z)
• SRIF: Semantic Shape Registration Empowered by Diffusion-based Image Morphing and Flow Estimation [2.336821026049481]
  We propose SRIF, a novel Semantic shape Registration framework based on diffusion-based Image morphing and flow estimation. SRIF achieves high-quality dense correspondences on challenging shape pairs, but also delivers smooth, semantically meaningful in between.
  arXiv  Detail & Related papers  (2024-09-18T03:47:24Z)
• Technical Report: Masked Skeleton Sequence Modeling for Learning Larval Zebrafish Behavior Latent Embeddings [5.922172844641853]
  We introduce a novel self-supervised learning method for extracting latent embeddings from behaviors of larval zebrafish. For the skeletal sequences of swimming zebrafish, we propose a pioneering Transformer-CNN architecture, the Sequence Spatial-Temporal Transformer (SSTFormer)
  arXiv  Detail & Related papers  (2024-03-23T02:58:10Z)
• DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation [56.514462874501675]
  We propose a dynamic sparse attention based Transformer model to achieve fine-level matching with favorable efficiency. The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on. Experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details.
  arXiv  Detail & Related papers  (2022-07-13T11:12:03Z)
• Frame Averaging for Equivariant Shape Space Learning [85.42901997467754]
  A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries. We present a framework for incorporating equivariance in encoders and decoders by introducing two contributions.
  arXiv  Detail & Related papers  (2021-12-03T06:41:19Z)
• Neural Crossbreed: Neural Based Image Metamorphosis [11.357156231073862]
  We propose a feed-forward neural network that can learn a semantic change of input images in a latent space to create the morphing effect. Because the network learns a semantic change, a sequence of meaningful intermediate images can be generated without requiring the user to specify explicit correspondences.
  arXiv  Detail & Related papers  (2020-09-02T08:56:47Z)
• Learning to Caricature via Semantic Shape Transform [95.25116681761142]
  We propose an algorithm based on a semantic shape transform to produce shape exaggerations. We show that the proposed framework is able to render visually pleasing shape exaggerations while maintaining their facial structures.
  arXiv  Detail & Related papers  (2020-08-12T03:41:49Z)
• Fast Symmetric Diffeomorphic Image Registration with Convolutional Neural Networks [11.4219428942199]
  We present a novel, efficient unsupervised symmetric image registration method. We evaluate our method on 3D image registration with a large scale brain image dataset.
  arXiv  Detail & Related papers  (2020-03-20T22:07:24Z)
• Fine-grained Image-to-Image Transformation towards Visual Recognition [102.51124181873101]
  We aim at transforming an image with a fine-grained category to synthesize new images that preserve the identity of the input image. We adopt a model based on generative adversarial networks to disentangle the identity related and unrelated factors of an image. Experiments on the CompCars and Multi-PIE datasets demonstrate that our model preserves the identity of the generated images much better than the state-of-the-art image-to-image transformation models.
  arXiv  Detail & Related papers  (2020-01-12T05:26:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.