Painting Many Pasts: Synthesizing Time Lapse Videos of Paintings

• URL: http://arxiv.org/abs/2001.01026v2
• Date: Sat, 25 Apr 2020 22:20:46 GMT
• Title: Painting Many Pasts: Synthesizing Time Lapse Videos of Paintings
• Authors: Amy Zhao, Guha Balakrishnan, Kathleen M. Lewis, Fr\'edo Durand, John V. Guttag, Adrian V. Dalca
• Abstract summary: We introduce a new video synthesis task: synthesizing time lapse videos depicting how a given painting might have been created. We present a probabilistic model that, given a single image of a completed painting, recurrently synthesizes steps of the painting process. We demonstrate that this model can be used to sample many time steps, enabling long-term video synthesis.
• Score: 23.99927916916298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a new video synthesis task: synthesizing time lapse videos depicting how a given painting might have been created. Artists paint using unique combinations of brushes, strokes, and colors. There are often many possible ways to create a given painting. Our goal is to learn to capture this rich range of possibilities. Creating distributions of long-term videos is a challenge for learning-based video synthesis methods. We present a probabilistic model that, given a single image of a completed painting, recurrently synthesizes steps of the painting process. We implement this model as a convolutional neural network, and introduce a novel training scheme to enable learning from a limited dataset of painting time lapses. We demonstrate that this model can be used to sample many time steps, enabling long-term stochastic video synthesis. We evaluate our method on digital and watercolor paintings collected from video websites, and show that human raters find our synthetic videos to be similar to time lapse videos produced by real artists. Our code is available at https://xamyzhao.github.io/timecraft.

Related papers

• Inverse Painting: Reconstructing The Painting Process [24.57538165449989]
  We formulate this as an autoregressive image generation problem, in which an initially blank "canvas" is iteratively updated. The model learns from real artists by training on many painting videos.
  arXiv  Detail & Related papers  (2024-09-30T17:56:52Z)
• Lumiere: A Space-Time Diffusion Model for Video Generation [75.54967294846686]
  We introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once. This is in contrast to existing video models which synthesize distants followed by temporal super-resolution. By deploying both spatial and (importantly) temporal down- and up-sampling, our model learns to directly generate a full-frame-rate, low-resolution video.
  arXiv  Detail & Related papers  (2024-01-23T18:05:25Z)
• WAIT: Feature Warping for Animation to Illustration video Translation using GANs [12.681919619814419]
  We introduce a new problem for video stylizing where an unordered set of images are used. Most of the video-to-video translation methods are built on an image-to-image translation model. We propose a new generator network with feature warping layers which overcomes the limitations of the previous methods.
  arXiv  Detail & Related papers  (2023-10-07T19:45:24Z)
• Hierarchical Masked 3D Diffusion Model for Video Outpainting [20.738731220322176]
  We introduce a masked 3D diffusion model for video outpainting. This allows us to use multiple guide frames to connect the results of multiple video clip inferences. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem.
  arXiv  Detail & Related papers  (2023-09-05T10:52:21Z)
• Towards Smooth Video Composition [59.134911550142455]
  Video generation requires consistent and persistent frames with dynamic content over time. This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs) We show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, brings a smooth frame transition without compromising the per-frame quality.
  arXiv  Detail & Related papers  (2022-12-14T18:54:13Z)
• Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing [23.143394242978125]
  paint2pix learns to predict "what a user wants to draw" from rudimentary brushstroke inputs. paint2pix can be used for progressive image synthesis from scratch. Our approach also forms a surprisingly convenient approach for real image editing.
  arXiv  Detail & Related papers  (2022-08-17T06:08:11Z)
• StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN [70.31913835035206]
  We present a novel approach to the video synthesis problem that helps to greatly improve visual quality. We make use of a pre-trained StyleGAN network, the latent space of which allows control over the appearance of the objects it was trained for. Our temporal architecture is then trained not on sequences of RGB frames, but on sequences of StyleGAN latent codes.
  arXiv  Detail & Related papers  (2021-07-15T09:58:15Z)
• Strumming to the Beat: Audio-Conditioned Contrastive Video Textures [112.6140796961121]
  We introduce a non-parametric approach for infinite video texture synthesis using a representation learned via contrastive learning. We take inspiration from Video Textures, which showed that plausible new videos could be generated from a single one by stitching its frames together in a novel yet consistent order. Our model outperforms baselines on human perceptual scores, can handle a diverse range of input videos, and can combine semantic and audio-visual cues in order to synthesize videos that synchronize well with an audio signal.
  arXiv  Detail & Related papers  (2021-04-06T17:24:57Z)
• Learning Joint Spatial-Temporal Transformations for Video Inpainting [58.939131620135235]
  We propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. We simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss.
  arXiv  Detail & Related papers  (2020-07-20T16:35:48Z)
• Compositional Video Synthesis with Action Graphs [112.94651460161992]
  Videos of actions are complex signals containing rich compositional structure in space and time. We propose to represent the actions in a graph structure called Action Graph and present the new Action Graph To Video'' synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation.
  arXiv  Detail & Related papers  (2020-06-27T09:39:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.