TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models

• URL: http://arxiv.org/abs/2609.01277v1
• Date: Tue, 01 Sep 2026 14:12:57 GMT
• Title: TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Models
• Abstract summary: emphinference-time speech scheduling is a novel task that places coupled speech and visual articulation within user-specified begin--end intervals.<n>We propose textbfTimeSteer, a training-free framework that localizes each utterance's source span.<n>TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.
• Score: 49.44322041671119
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although pretrained joint audio-visual diffusion models offer rich control over \emph{what} to generate, they provide no explicit control over \emph{when} an utterance should occur. To address this, we study \emph{inference-time speech scheduling}, a novel task that places coupled speech and visual articulation within user-specified begin--end intervals without finetuning the backbone model. We uncover two intrinsic properties of the denoising process that enable this task. First, a timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. Second, the predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content. Building on these discoveries, we propose \textbf{TimeSteer}, a training-free framework that localizes each utterance's source span through \textbf{Source Span Localization} and transfers the associated audio-visual latent content from the source interval to the specified target interval through \textbf{Region-Aware Latent Remapping}. We further introduce \textbf{SpeechShift}, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.

Related papers

• Vorch-Omni: Multi-Task Orchestration of Sight and Sound [51.39284238118125]
  Vorch- Omni is a unified framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-language formulation.<n>It flexibly treats video and audio signals as either conditioning inputs or generation targets.<n>It supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing.
  arXiv  Detail & Related papers  (2026-08-06T09:38:30Z)
• Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation [58.79612575170589]
  Existing methods struggle to match overlapping acoustic events with visual instances and handle audio-visual dynamics.<n>This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms.<n>The code will be open-sourced once the paper is accepted.
  arXiv  Detail & Related papers  (2026-08-04T07:35:57Z)
• Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation [113.24577778437295]
  Baton is a framework that introduces explicit semantic planning into joint video-audio generation.<n>Our key insight is that complementing coarse text guidance with semantically rich, modality-aware planned tokens, can simultaneously restore fine-grained semantic detail.<n> experiments on benchmarks show the effectiveness of Baton both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2026-05-24T17:55:11Z)
• Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs [48.50855715191533]
  Large audio language models are increasingly used for complex audio understanding tasks.<n>They struggle with temporal tasks that require precise temporal grounding, such as word alignment and speaker diarization.<n>We propose frame-level internal tool use, a method that trains audio LMs to use their own internal audio representations to perform temporal grounding directly.
  arXiv  Detail & Related papers  (2026-02-10T19:19:52Z)
• Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis [57.5830191022097]
  A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation.<n>We adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs.<n>Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines.
  arXiv  Detail & Related papers  (2025-11-07T17:07:56Z)
• CLASP: Cross-modal Salient Anchor-based Semantic Propagation for Weakly-supervised Dense Audio-Visual Event Localization [15.861700882671418]
  This paper explores DAVEL under a new and more challenging weakly-supervised setting (W-DAVEL task)<n>We exploit textitcross-modal salient anchors, which are defined as reliable timestamps that are well predicted under weak supervision.<n>We establish benchmarks for W-DAVEL on both the UnAV-100 and ActivityNet1.3 datasets.
  arXiv  Detail & Related papers  (2025-08-06T15:49:53Z)
• VCSE: Time-Domain Visual-Contextual Speaker Extraction Network [54.67547526785552]
  We propose a two-stage time-domain visual-contextual speaker extraction network named VCSE. In the first stage, we pre-extract a target speech with visual cues and estimate the underlying phonetic sequence. In the second stage, we refine the pre-extracted target speech with the self-enrolled contextual cues.
  arXiv  Detail & Related papers  (2022-10-09T12:29:38Z)
• Temporal and cross-modal attention for audio-visual zero-shot learning [38.02396786726476]
  generalised zero-shot learning for video classification requires understanding the relations between the audio and visual information. We propose a multi-modal and Temporal Cross-attention Framework (modelName) for audio-visual generalised zero-shot learning. We show that our proposed framework that ingests temporal features yields state-of-the-art performance on the ucf, vgg, and activity benchmarks for (generalised) zero-shot learning.
  arXiv  Detail & Related papers  (2022-07-20T15:19:30Z)
• End-to-End Adversarial Text-to-Speech [33.01223309795122]
  We learn to synthesise speech from normalised text or phonemes in an end-to-end manner. Our proposed generator is feed-forward and thus efficient for both training and inference. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses.
  arXiv  Detail & Related papers  (2020-06-05T17:41:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.