ToonCrafter: Generative Cartoon Interpolation

• URL: http://arxiv.org/abs/2405.17933v1
• Date: Tue, 28 May 2024 07:58:33 GMT
• Title: ToonCrafter: Generative Cartoon Interpolation
• Authors: Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, Tien-Tsin Wong,
• Abstract summary: We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion.
• Score: 63.52353451649143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce ToonCrafter, a novel approach that transcends traditional correspondence-based cartoon video interpolation, paving the way for generative interpolation. Traditional methods, that implicitly assume linear motion and the absence of complicated phenomena like dis-occlusion, often struggle with the exaggerated non-linear and large motions with occlusion commonly found in cartoons, resulting in implausible or even failed interpolation results. To overcome these limitations, we explore the potential of adapting live-action video priors to better suit cartoon interpolation within a generative framework. ToonCrafter effectively addresses the challenges faced when applying live-action video motion priors to generative cartoon interpolation. First, we design a toon rectification learning strategy that seamlessly adapts live-action video priors to the cartoon domain, resolving the domain gap and content leakage issues. Next, we introduce a dual-reference-based 3D decoder to compensate for lost details due to the highly compressed latent prior spaces, ensuring the preservation of fine details in interpolation results. Finally, we design a flexible sketch encoder that empowers users with interactive control over the interpolation results. Experimental results demonstrate that our proposed method not only produces visually convincing and more natural dynamics, but also effectively handles dis-occlusion. The comparative evaluation demonstrates the notable superiority of our approach over existing competitors.

Related papers

• IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation [58.297199313494]
  Implicit methods capture motion semantics directly from driving video, but suffer from identity leakage and entanglement between motion and appearance.<n>We propose a novel implicit motion representation that compresses per-frame motion into compact 1D motion tokens.<n>Our methodology employs a three-stage training strategy to enhance the training efficiency and ensure high fidelity.
  arXiv  Detail & Related papers  (2026-02-07T11:17:20Z)
• ToonComposer: Streamlining Cartoon Production with Generative Post-Keyframing [60.81602269917522]
  ToonComposer is a generative model that unifies inbetweening and colorization into a single post-keyframing stage.<n>Requiring as few as a single sketch and a colored reference frame, ToonComposer excels with sparse inputs.<n>Our evaluation demonstrates that ToonComposer outperforms existing methods in visual quality, motion consistency, and production efficiency.
  arXiv  Detail & Related papers  (2025-08-14T17:50:11Z)
• FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors [64.54220123913154]
  We introduce FramePainter as an efficient instantiation of image-to-video generation problem. It only uses a lightweight sparse control encoder to inject editing signals. It domainantly outperforms previous state-of-the-art methods with far less training data.
  arXiv  Detail & Related papers  (2025-01-14T16:09:16Z)
• Framer: Interactive Frame Interpolation [73.06734414930227]
  Framer targets producing smoothly transitioning frames between two images as per user creativity. Our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and the trajectory automatically.
  arXiv  Detail & Related papers  (2024-10-24T17:59:51Z)
• Thin-Plate Spline-based Interpolation for Animation Line Inbetweening [54.69811179222127]
  Chamfer Distance (CD) is commonly adopted for evaluating inbetweening performance. We propose a simple yet effective method for animation line inbetweening that adopts thin-plate spline-based transformation. Our method outperforms existing approaches by delivering high-quality results with enhanced fluidity.
  arXiv  Detail & Related papers  (2024-08-17T08:05:31Z)
• Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics [67.97235923372035]
  We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories, Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions.
  arXiv  Detail & Related papers  (2024-08-08T17:59:38Z)
• Disentangling Foreground and Background Motion for Enhanced Realism in Human Video Generation [15.569467643817447]
  We introduce a technique that concurrently learns both foreground and background dynamics by segregating their movements using distinct motion representations. We train on real-world videos enhanced with this innovative motion depiction approach. To further extend video generation to longer sequences without accumulating errors, we adopt a clip-by-clip generation strategy.
  arXiv  Detail & Related papers  (2024-05-26T00:53:26Z)
• AniClipart: Clipart Animation with Text-to-Video Priors [28.76809141136148]
  We introduce AniClipart, a system that transforms static images into high-quality motion sequences guided by text-to-video priors. Experimental results show that the proposed AniClipart consistently outperforms existing image-to-video generation models.
  arXiv  Detail & Related papers  (2024-04-18T17:24:28Z)
• An Animation-based Augmentation Approach for Action Recognition from Discontinuous Video [11.293897932762809]
  Action recognition, an essential component of computer vision, plays a pivotal role in multiple applications. CNNs suffer performance declines when trained with discontinuous video frames, which is a frequent scenario in real-world settings. To overcome this issue, we introduce the 4A pipeline, which employs a series of sophisticated techniques.
  arXiv  Detail & Related papers  (2024-04-10T04:59:51Z)
• Perception-Oriented Video Frame Interpolation via Asymmetric Blending [20.0024308216849]
  Previous methods for Video Frame Interpolation (VFI) have encountered challenges, notably the manifestation of blur and ghosting effects. We propose PerVFI (Perception-oriented Video Frame Interpolation) to mitigate these challenges. Experimental results validate the superiority of PerVFI, demonstrating significant improvements in perceptual quality compared to existing methods.
  arXiv  Detail & Related papers  (2024-04-10T02:40:17Z)
• DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors [63.43133768897087]
  We propose a method to convert open-domain images into animated videos. The key idea is to utilize the motion prior to text-to-video diffusion models by incorporating the image into the generative process as guidance. Our proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image.
  arXiv  Detail & Related papers  (2023-10-18T14:42:16Z)
• Deep Animation Video Interpolation in the Wild [115.24454577119432]
  In this work, we formally define and study the animation video code problem for the first time. We propose an effective framework, AnimeInterp, with two dedicated modules in a coarse-to-fine manner. Notably, AnimeInterp shows favorable perceptual quality and robustness for animation scenarios in the wild.
  arXiv  Detail & Related papers  (2021-04-06T13:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.