In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

• URL: http://arxiv.org/abs/2604.22817v1
• Date: Tue, 14 Apr 2026 20:56:24 GMT
• Title: In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions
• Abstract summary: We extend an existing speech-aware language model to predict timestamps directly alongside transcripts.<n>We introduce a set of novel lightweight training strategies that improve alignment robustness while preserving recognition quality.
• Score: 51.236148875603185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in speech-aware language models have coupled strong acoustic encoders with large language models, enabling systems that move beyond transcription to produce richer outputs. Among these, word-level timestamp prediction is critical for applications such as captioning, media search, and multimodal synchronization, yet it is often handled by external alignment tools. In this work, we extend an existing speech-aware language model to predict timestamps directly alongside transcripts. We introduce a set of novel lightweight training strategies that improve alignment robustness while preserving recognition quality. Experiments across multiple datasets show that these strategies not only enhance timestamp accuracy, but also yield gains in overall ASR performance. Together, they demonstrate an efficient and unified approach to speech recognition with precise timestamp prediction.

Related papers

• Relative Time Intervals Representation for Word-level Timestamping with Masked Training [21.238122108925975]
  Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored.<n>We replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities.<n>Our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.
  arXiv  Detail & Related papers  (2026-08-25T03:59:42Z)
• Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models [51.75139461584678]
  Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order.<n>We propose an instruction-level framework that uses audio-aware large language models as fine-grained judges to verify target event presence and temporal relations.<n>Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
  arXiv  Detail & Related papers  (2026-07-15T03:13:06Z)
• PRiSM: Benchmarking Phone Realization in Speech Models [70.82595415252682]
  Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis.<n>We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception.
  arXiv  Detail & Related papers  (2026-01-20T15:00:36Z)
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model [76.06585781346601]
  Speech language models (Speech LMs) enable end-to-end speech-text modelling within a single model.<n>The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality.
  arXiv  Detail & Related papers  (2025-06-04T23:53:49Z)
• CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions [0.5120567378386615]
  We fine-tune the model to produce more verbatim speech transcriptions. We employ several techniques to increase robustness against multiple speakers and background noise.
  arXiv  Detail & Related papers  (2024-08-29T14:52:42Z)
• AudioSetMix: Enhancing Audio-Language Datasets with LLM-Assisted Augmentations [1.2101820447447276]
  Multi-modal learning in the audio-language domain has seen significant advancements in recent years. However, audio-language learning faces challenges due to limited and lower-quality data compared to image-language tasks. Our method systematically generates audio-caption pairs by augmenting audio clips with natural language labels and corresponding audio signal processing operations. This scalable method produces AudioSetMix, a high-quality training dataset for text-and-audio related models.
  arXiv  Detail & Related papers  (2024-05-17T21:08:58Z)
• Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation [51.399695200838586]
  We propose a streaming Transformer-Transducer (T-T) model able to jointly produce many-to-one and one-to-many transcription and translation using a single decoder. Experiments on it,es,de->en prove the effectiveness of our approach, enabling the generation of one-to-many joint outputs with a single decoder for the first time.
  arXiv  Detail & Related papers  (2023-10-23T11:00:27Z)
• Adapting Multi-Lingual ASR Models for Handling Multiple Talkers [63.151811561972515]
  State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages. We propose an approach to adapt USMs for multi-talker ASR. We first develop an enhanced version of serialized output training to jointly perform multi-talker ASR and utterance timestamp prediction.
  arXiv  Detail & Related papers  (2023-05-30T05:05:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.