AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models

• URL: http://arxiv.org/abs/2503.08417v1
• Date: Tue, 11 Mar 2025 13:28:59 GMT
• Title: AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models
• Authors: Kwan Yun, Seokhyeon Hong, Chaelin Kim, Junyong Noh,
• Abstract summary: We introduce AnyMoLe, a novel method to generate motion in-between frames for arbitrary characters without external data.<n>Our approach employs a two-stage frame generation process to enhance contextual understanding.
• Score: 5.224806515926022
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite recent advancements in learning-based motion in-betweening, a key limitation has been overlooked: the requirement for character-specific datasets. In this work, we introduce AnyMoLe, a novel method that addresses this limitation by leveraging video diffusion models to generate motion in-between frames for arbitrary characters without external data. Our approach employs a two-stage frame generation process to enhance contextual understanding. Furthermore, to bridge the domain gap between real-world and rendered character animations, we introduce ICAdapt, a fine-tuning technique for video diffusion models. Additionally, we propose a ``motion-video mimicking'' optimization technique, enabling seamless motion generation for characters with arbitrary joint structures using 2D and 3D-aware features. AnyMoLe significantly reduces data dependency while generating smooth and realistic transitions, making it applicable to a wide range of motion in-betweening tasks.

Related papers

• Video Motion Graphs [17.57582826585202]
  We present Motion Graphs, a system designed to generate realistic human motion videos. The system synthesizes new videos by first retrieving video clips with gestures matching the conditions and then generating frames to seamlessly connect clip boundaries.
  arXiv  Detail & Related papers  (2025-03-26T04:20:14Z)
• MotionMatcher: Motion Customization of Text-to-Video Diffusion Models via Motion Feature Matching [27.28898943916193]
  Text-to-video (T2V) diffusion models have promising capabilities in synthesizing realistic videos from input text prompts.<n>In this work, we tackle the motion customization problem, where a reference video is provided as motion guidance.<n>We propose MotionMatcher, a motion customization framework that fine-tunes the pre-trained T2V diffusion model at the feature level.
  arXiv  Detail & Related papers  (2025-02-18T19:12:51Z)
• MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models [59.10171699717122]
  MoTrans is a customized motion transfer method enabling video generation of similar motion in new context.<n> multimodal representations from recaptioned prompt and video frames promote the modeling of appearance.<n>Our method effectively learns specific motion pattern from singular or multiple reference videos.
  arXiv  Detail & Related papers  (2024-12-02T10:07:59Z)
• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms. SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics. Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• Animate Your Motion: Turning Still Images into Dynamic Videos [58.63109848837741]
  We introduce Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. SMCD incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions. Our design significantly enhances video quality, motion precision, and semantic coherence.
  arXiv  Detail & Related papers  (2024-03-15T10:36:24Z)
• LatentMan: Generating Consistent Animated Characters using Image Diffusion Models [44.18315132571804]
  We propose a zero-shot approach for generating consistent videos of animated characters based on Text-to-Image (T2I) diffusion models. Our proposed approach outperforms existing zero-shot T2V approaches in generating videos of animated characters in terms of pixel-wise consistency and user preference.
  arXiv  Detail & Related papers  (2023-12-12T10:07:37Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• MoVideo: Motion-Aware Video Generation with Diffusion Models [97.03352319694795]
  We propose a novel motion-aware generation (MoVideo) framework that takes motion into consideration from two aspects: video depth and optical flow. MoVideo achieves state-of-the-art results in both text-to-video and image-to-video generation, showing promising prompt consistency, frame consistency and visual quality.
  arXiv  Detail & Related papers  (2023-11-19T13:36:03Z)
• Animation from Blur: Multi-modal Blur Decomposition with Motion Guidance [83.25826307000717]
  We study the challenging problem of recovering detailed motion from a single motion-red image. Existing solutions to this problem estimate a single image sequence without considering the motion ambiguity for each region. In this paper, we explicitly account for such motion ambiguity, allowing us to generate multiple plausible solutions all in sharp detail.
  arXiv  Detail & Related papers  (2022-07-20T18:05:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.