Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization

• URL: http://arxiv.org/abs/2508.08550v1
• Date: Tue, 12 Aug 2025 01:38:31 GMT
• Title: Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization
• Abstract summary: Video dubbing aims to translate original speech in visual media programs from the source language to the target language.<n>Due to varying information densities across languages, target speech often mismatches the source speech duration.<n>We propose the Segment Supervised Preference Optimization (SSPO) method, which employs a segment-wise sampling strategy.
• Score: 4.705596004268067
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Video dubbing aims to translate original speech in visual media programs from the source language to the target language, relying on neural machine translation and text-to-speech technologies. Due to varying information densities across languages, target speech often mismatches the source speech duration, causing audio-video synchronization issues that significantly impact viewer experience. In this study, we approach duration alignment in LLM-based video dubbing machine translation as a preference optimization problem. We propose the Segment Supervised Preference Optimization (SSPO) method, which employs a segment-wise sampling strategy and fine-grained loss to mitigate duration mismatches between source and target lines. Experimental results demonstrate that SSPO achieves superior performance in duration alignment tasks.

Related papers

• Video-guided Machine Translation with Global Video Context [4.074619388936337]
  We propose a globally video-guided multimodal translation framework to capture global narrative context in long videos.<n>An attention mechanism is employed to focus on highly relevant visual content, while preserving the remaining video features.<n>Experiments on a large-scale documentary translation dataset demonstrate that our method significantly outperforms baseline models.
  arXiv  Detail & Related papers  (2026-04-08T07:57:05Z)
• CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Models [66.56549019393042]
  Video-language models (VLMs) achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order.<n>We propose a scalable framework for counterfactual video generation that synthesizes videos differing only in actions or temporal structure while preserving scene context.
  arXiv  Detail & Related papers  (2026-01-08T10:03:07Z)
• GAID: Frame-Level Gated Audio-Visual Integration with Directional Perturbation for Text-Video Retrieval [12.483734449829235]
  GAID is a framework that integrates audio and visual features under textual guidance.<n>DASP injects structure-aware perturbations into text embeddings, enhancing robustness and discrimination without incurring multi-pass inference.<n>Experiments on MSR-VTT, DiDeMo, LSMDC, and VATEX show consistent state-of-the-art results with notable efficiency gains.
  arXiv  Detail & Related papers  (2025-08-03T10:44:24Z)
• AVC-DPO: Aligned Video Captioning via Direct Preference Optimization [50.08618093204503]
  Video multimodal large language models (video MLLMs) have achieved substantial progress in video captioning tasks.<n>We propose Aligned Video Captioning via Direct Preference Optimization (AVC-DPO), a post-training framework designed to enhance captioning capabilities in video MLLMs through preference alignment.<n>We have achieved exceptional performance in the LOVE@PRCV'25 Workshop Track 1A: Video Detailed Captioning Challenge, achieving first place on the Video Detailed Captioning benchmark.
  arXiv  Detail & Related papers  (2025-07-02T08:51:45Z)
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model [76.06585781346601]
  Speech language models (Speech LMs) enable end-to-end speech-text modelling within a single model.<n>The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality.
  arXiv  Detail & Related papers  (2025-06-04T23:53:49Z)
• Video-Language Alignment via Spatio-Temporal Graph Transformer [26.109883502401885]
  Video-language alignment is a crucial task that benefits various downstream applications, e.g., video-text retrieval and question answering. We propose a novel Spatio-Temporal Graph Transformer module to uniformly learn spatial and temporal contexts for video-language alignment pre-training.
  arXiv  Detail & Related papers  (2024-07-16T12:52:32Z)
• Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video [22.60291297308379]
  We investigate the feasibility in transforming the video summarization task into a Natural Language Processing (NLP) task. Our method achieves state-of-the-art performance on the SumMe dataset in rank correlation coefficients.
  arXiv  Detail & Related papers  (2024-05-14T18:07:04Z)
• Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing [71.02335065794384]
  We propose a model that directly optimize both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.
  arXiv  Detail & Related papers  (2023-02-25T04:23:25Z)
• VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing [73.56970726406274]
  Video dubbing aims to translate the original speech in a film or television program into the speech in a target language. To ensure the translated speech to be well aligned with the corresponding video, the length/duration of the translated speech should be as close as possible to that of the original speech. We propose a machine translation system tailored for the task of video dubbing, which directly considers the speech duration of each token in translation.
  arXiv  Detail & Related papers  (2022-11-30T12:09:40Z)
• Modeling Motion with Multi-Modal Features for Text-Based Video Segmentation [56.41614987789537]
  Text-based video segmentation aims to segment the target object in a video based on a describing sentence. We propose a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation.
  arXiv  Detail & Related papers  (2022-04-06T02:42:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.