Relative Time Intervals Representation for Word-level Timestamping with Masked Training

• URL: http://arxiv.org/abs/2608.24041v1
• Date: Tue, 25 Aug 2026 03:59:42 GMT
• Title: Relative Time Intervals Representation for Word-level Timestamping with Masked Training
• Abstract summary: Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored.<n>We replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities.<n>Our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.
• Score: 21.238122108925975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored. Our work addresses this gap by enabling SpeechLLMs to jointly model speech content and temporal structure, effectively transforming them from ``content understanding machines" into ``temporal-aware content understanding machines". Specifically, we replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities. To efficiently infuse timestamp prediction ability into pre-trained large language models, we introduce a hybrid fine-tuning strategy: full-parameter fine-tuning of the timestamp-augmented embedding layer and language model head, combined with LoRA fine-tuning of the decoder layers. Moreover, we design a masked timestamp training objective, preventing the model from over-relying on ground-truth timestamps, and thereby enhancing robustness against noisy real-world annotations. Extensive experiments demonstrate that our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.

Related papers

• In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions [51.236148875603185]
  We extend an existing speech-aware language model to predict timestamps directly alongside transcripts.<n>We introduce a set of novel lightweight training strategies that improve alignment robustness while preserving recognition quality.
  arXiv  Detail & Related papers  (2026-04-14T20:56:24Z)
• Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs [48.50855715191533]
  Large audio language models are increasingly used for complex audio understanding tasks.<n>They struggle with temporal tasks that require precise temporal grounding, such as word alignment and speaker diarization.<n>We propose frame-level internal tool use, a method that trains audio LMs to use their own internal audio representations to perform temporal grounding directly.
  arXiv  Detail & Related papers  (2026-02-10T19:19:52Z)
• Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMs [54.502280390499756]
  We propose TimeWarp to create a targeted synthetic temporal dataset to fine-tune the model's responses to encourage it to focus on the given input video.<n>We demonstrate that when our method is applied to existing models, it significantly improves performance on temporal understanding benchmarks.
  arXiv  Detail & Related papers  (2025-10-04T21:48:40Z)
• Word Level Timestamp Generation for Automatic Speech Recognition and Translation [28.176210372699618]
  We introduce a data-driven approach for enabling word-level timestamp prediction in the Canary model.<n>Our method demonstrates precision and recall rates between 80% and 90%, with timestamp prediction errors ranging from 20 to 120 ms across four languages.
  arXiv  Detail & Related papers  (2025-05-21T15:24:29Z)
• VideoExpert: Augmented LLM for Temporal-Sensitive Video Understanding [48.745013691038295]
  VideoExpert is a general-purpose MLLM suitable for several temporal-sensitive video tasks.<n>The Temporal Expert is responsible for modeling time sequences and performing temporal grounding.<n>The Spatial Expert focuses on content detail analysis and instruction following.<n>By offloading temporal grounding from content generation, VideoExpert prevents text pattern biases in timestamp predictions.
  arXiv  Detail & Related papers  (2025-04-10T07:33:39Z)
• CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions [0.5120567378386615]
  We fine-tune the model to produce more verbatim speech transcriptions. We employ several techniques to increase robustness against multiple speakers and background noise.
  arXiv  Detail & Related papers  (2024-08-29T14:52:42Z)
• Handling Numeric Expressions in Automatic Speech Recognition [56.972851337263755]
  We compare cascaded and end-to-end approaches to recognize and format numeric expressions.<n>Results show that adapted end-to-end models offer competitive performance with the advantage of lower latency and inference cost.
  arXiv  Detail & Related papers  (2024-07-18T09:46:19Z)
• Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters [71.02335065794384]
  We introduce target factors in a transformer model to predict durations jointly with target language phoneme sequences. We show that our model improves translation quality and isochrony compared to previous work.
  arXiv  Detail & Related papers  (2023-05-22T16:36:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.