Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening

• URL: http://arxiv.org/abs/2308.13273v2
• Date: Wed, 14 Aug 2024 08:48:40 GMT
• Title: Bridging the Gap: Sketch-Aware Interpolation Network for High-Quality Animation Sketch Inbetweening
• Authors: Jiaming Shen, Kun Hu, Wei Bao, Chang Wen Chen, Zhiyong Wang,
• Abstract summary: We propose a novel deep learning method - Sketch-Aware Interpolation Network (SAIN) This approach incorporates multi-level guidance that formulates region-level correspondence, stroke-level correspondence and pixel-level dynamics. A multi-stream U-Transformer is then devised to characterize sketch inbetweening patterns using these multi-level guides through the integration of self / cross-attention mechanisms.
• Score: 58.09847349781176
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hand-drawn 2D animation workflow is typically initiated with the creation of sketch keyframes. Subsequent manual inbetweens are crafted for smoothness, which is a labor-intensive process and the prospect of automatic animation sketch interpolation has become highly appealing. Yet, common frame interpolation methods are generally hindered by two key issues: 1) limited texture and colour details in sketches, and 2) exaggerated alterations between two sketch keyframes. To overcome these issues, we propose a novel deep learning method - Sketch-Aware Interpolation Network (SAIN). This approach incorporates multi-level guidance that formulates region-level correspondence, stroke-level correspondence and pixel-level dynamics. A multi-stream U-Transformer is then devised to characterize sketch inbetweening patterns using these multi-level guides through the integration of self / cross-attention mechanisms. Additionally, to facilitate future research on animation sketch inbetweening, we constructed a large-scale dataset - STD-12K, comprising 30 sketch animation series in diverse artistic styles. Comprehensive experiments on this dataset convincingly show that our proposed SAIN surpasses the state-of-the-art interpolation methods.

Related papers

• Thin-Plate Spline-based Interpolation for Animation Line Inbetweening [54.69811179222127]
  Chamfer Distance (CD) is commonly adopted for evaluating inbetweening performance. We propose a simple yet effective method for animation line inbetweening that adopts thin-plate spline-based transformation. Our method outperforms existing approaches by delivering high-quality results with enhanced fluidity.
  arXiv  Detail & Related papers  (2024-08-17T08:05:31Z)
• Sketch Video Synthesis [52.134906766625164]
  We propose a novel framework for sketching videos represented by the frame-wise B'ezier curve. Our method unlocks applications in sketch-based video editing and video doodling, enabled through video composition.
  arXiv  Detail & Related papers  (2023-11-26T14:14:04Z)
• Deep Geometrized Cartoon Line Inbetweening [98.35956631655357]
  Inbetweening involves generating intermediate frames between two black-and-white line drawings. Existing frame methods that rely on matching and warping whole images are unsuitable for line inbetweening. We propose AnimeInbet, which geometrizes geometric line drawings into endpoints and reframes the inbetweening task as a graph fusion problem. Our method can effectively capture the sparsity and unique structure of line drawings while preserving the details during inbetweening.
  arXiv  Detail & Related papers  (2023-09-28T17:50:05Z)
• Improving the Perceptual Quality of 2D Animation Interpolation [37.04208600867858]
  Traditional 2D animation is labor-intensive, often requiring animators to draw twelve illustrations per second of movement. Lower framerates result in larger displacements and occlusions, discrete perceptual elements (e.g. lines and solid-color regions) pose difficulties for texture-oriented convolutional networks. Previous work tried addressing these issues, but used unscalable methods and focused on pixel-perfect performance. We build a scalable system more appropriately centered on perceptual quality for this artistic domain.
  arXiv  Detail & Related papers  (2021-11-24T20:51:29Z)
• Sketch Me A Video [32.38205496481408]
  We introduce a new video synthesis task by employing two rough bad-drwan sketches only as input to create a realistic portrait video. A two-stage Sketch-to-Video model is proposed, which consists of two key novelties.
  arXiv  Detail & Related papers  (2021-10-10T05:40:11Z)
• Deep Sketch-guided Cartoon Video Inbetweening [24.00033622396297]
  We propose a framework to produce cartoon videos by fetching the color information from two inputs while following the animated motion guided by a user sketch. By explicitly considering the correspondence between frames and the sketch, we can achieve higher quality results than other image synthesis methods.
  arXiv  Detail & Related papers  (2020-08-10T14:22:04Z)
• Multi-Density Sketch-to-Image Translation Network [65.4028451067947]
  We propose the first multi-level density sketch-to-image translation framework, which allows the input sketch to cover a wide range from rough object outlines to micro structures. Our method has been successfully verified on various datasets for different applications including face editing, multi-modal sketch-to-photo translation, and anime colorization.
  arXiv  Detail & Related papers  (2020-06-18T16:21:04Z)
• SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence [68.63311821718416]
  We study the problem of multi-view sketch correspondence, where we take as input multiple freehand sketches with different views of the same object. This problem is challenging since the visual features of corresponding points at different views can be very different. We take a deep learning approach and learn a novel local sketch descriptor from data.
  arXiv  Detail & Related papers  (2020-01-16T11:31:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.