Sound Sparks Motion: Audio and Text Tuning for Video Editing

• URL: http://arxiv.org/abs/2605.15307v1
• Date: Thu, 14 May 2026 18:20:50 GMT
• Title: Sound Sparks Motion: Audio and Text Tuning for Video Editing
• Abstract summary: We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model.<n>Sound Sparks Motion tunes its internal multimodal conditioning signals at test time.<n>Our results highlight multimodal conditioning tuning as a promising direction for motion-aware video editing.
• Score: 53.136757756110626
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Motion-centric video editing remains difficult for large generative video models, which often respond well to appearance changes but struggle to produce specific, localized actions or state transitions in an existing clip. We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model by tuning its internal multimodal conditioning signals at test time. Rather than modifying model weights, our method tunes only two lightweight variables: an audio latent derived from the source video and a residual perturbation in the text-conditioning. We find that this combination can encourage motion edits that the underlying model often struggles to realize under prompt-only control. Since there is no direct way to evaluate temporal alignment between text and motion, we guide the tuning process using a vision-language model that provides feedback indicating whether the intended motion appears in the generated video. This simple supervision yields an effective semantic objective for motion editing, while regularization and perceptual-temporal constraints help preserve content and visual quality. Beyond per-video tuning, we show that the learned latent controls are transferable across videos, suggesting that they capture reusable motion-edit directions rather than overfitting to a single example. Our results highlight multimodal conditioning tuning, particularly through the audio pathway, as a promising direction for motion-aware video editing, and suggest that test-time tuning can serve as a lightweight probing mechanism that helps reveal latent motion controls embedded in the model's multimodal conditioning. Code and data are available via our project page: https://amirhossein-razlighi.github.io/Sound_Sparks_Motion/

Related papers

• MotionV2V: Editing Motion in a Video [53.791975554391534]
  We propose modifying video motion by editing sparse trajectories extracted from the input.<n>We term the deviation between input and output trajectories a "motion edit"<n>Our approach allows for edits that start at any timestamp and propagate naturally.
  arXiv  Detail & Related papers  (2025-11-25T18:57:25Z)
• Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising [23.044483059783143]
  Diffusion-based video generation can create realistic videos, yet existing image- and text-based conditioning fails to offer precise motion control.<n>We introduce Time-to-Move (TTM), a training-free, plug-and-play framework for motion- and appearance-controlled video generation.
  arXiv  Detail & Related papers  (2025-11-09T22:47:50Z)
• AnimateAnything: Consistent and Controllable Animation for Video Generation [24.576022028967195]
  We present a unified controllable video generation approach AnimateAnything. It facilitates precise and consistent video manipulation across various conditions. Experiments demonstrate that our method outperforms the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-11-16T16:36:49Z)
• MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion [94.66090422753126]
  MotionFollower is a lightweight score-guided diffusion model for video motion editing. It delivers superior motion editing performance and exclusively supports large camera movements and actions. Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory.
  arXiv  Detail & Related papers  (2024-05-30T17:57:30Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• MotionEditor: Editing Video Motion via Content-Aware Diffusion [96.825431998349]
  MotionEditor is a diffusion model for video motion editing. It incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence.
  arXiv  Detail & Related papers  (2023-11-30T18:59:33Z)
• Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders [29.658535633701035]
  We propose a novel conditional variational autoencoder (VAE) that explicitly models one-to-many audio-to-motion mapping. We show that our method generates more realistic and diverse motions than state-of-the-art methods, quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-15T11:15:51Z)
• Strumming to the Beat: Audio-Conditioned Contrastive Video Textures [112.6140796961121]
  We introduce a non-parametric approach for infinite video texture synthesis using a representation learned via contrastive learning. We take inspiration from Video Textures, which showed that plausible new videos could be generated from a single one by stitching its frames together in a novel yet consistent order. Our model outperforms baselines on human perceptual scores, can handle a diverse range of input videos, and can combine semantic and audio-visual cues in order to synthesize videos that synchronize well with an audio signal.
  arXiv  Detail & Related papers  (2021-04-06T17:24:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.