TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

• URL: http://arxiv.org/abs/2608.29999v1
• Date: Sun, 30 Aug 2026 19:49:21 GMT
• Title: TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
• Abstract summary: Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify.<n>We present TEMPO, the first unified model to handle audio, speech, and music timestamping tasks.
• Score: 58.80225070159013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large audio-language models (LALMs) describe audio at the clip level but cannot assign timestamps to the events, speakers, or sounds they identify. Despite being essential for downstream tasks like speech recognition and dense audio captioning, timestamping remains a key limitation of most LALMs. We present TEMPO (Temporally-grounded Multi-task Post-training), the first unified model to handle audio, speech, and music timestamping tasks. Our core contribution is a supervised fine-tuning (SFT) stage built on three innovations: atomic timestamp tokens, a time-aware projector that injects sinusoidal wall-clock encodings into audio frame embeddings, and a distance-aware Gaussian loss. Our training is based on a synthetic-to-real curriculum. We further introduce, to our knowledge, the first application of reinforcement learning to unified audio timestamping, using GRPO with verifiable temporal rewards that directly optimize the evaluation objectives. Rather than serving as the primary source of performance gains, GRPO acts as a refinement stage on top of the SFT checkpoint, providing modest additional improvements. To support this work, we build a training dataset containing 119K samples and an evaluation benchmark containing 10K samples, drawn from established corpora across five tasks. On this benchmark, TEMPO outperforms Audio Flamingo Next and Qwen3-Omni, two state-of-the-art LALMs explicitly trained on timestamped data. Experiments confirm that SFT delivers most of these gains, with GRPO providing consistent but moderate refinements.

Related papers

• Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models [51.75139461584678]
  Recent text-to-audio models generate high-quality audio, but often fail to follow instructions involving multiple sound events and temporal order.<n>We propose an instruction-level framework that uses audio-aware large language models as fine-grained judges to verify target event presence and temporal relations.<n>Experiments show that our method improves event completeness, temporal ordering, and joint instruction-following accuracy across existing benchmarks and S3Bench, while maintaining audio quality.
  arXiv  Detail & Related papers  (2026-07-15T03:13:06Z)
• GigaChat Audio: Time-aware Large Audio Language Model [65.51589021112385]
  We present a time-aware audio LLM that answers questions with explicit timestamps over up to 120 minutes of input.<n>Our approach interleaves periodic time markers with continuous audio tokens using large-scale synthetic supervision from a cascaded pipeline.
  arXiv  Detail & Related papers  (2026-07-11T16:34:25Z)
• AudioMosaic: Contrastive Masked Audio Representation Learning [53.52371029884106]
  We introduce textbfAudioMosaic, a contrastive learning-based audio encoder for general audio understanding.<n>AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches.<n>Experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T00:56:51Z)
• Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models [56.91801348360746]
  Large audio-language models (LALMs) generalize across speech, sound, and music.<n> unified decoders can exhibit a emphtemporal smoothing bias<n>We propose emphTemporal Contrastive Decoding (TCD), a training-free decoding method for unified LALMs.
  arXiv  Detail & Related papers  (2026-04-16T02:30:41Z)
• Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt [26.402383197553778]
  We propose Audio-Side Time Prompt and leverage Reinforcement Learning to develop the TimePro-RL framework for fine-grained temporal perception.<n>Specifically, we encodes as embedding timestamps and interleave them within the audio feature sequence as temporal coordinates to prompt the model.<n>Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning.
  arXiv  Detail & Related papers  (2026-04-15T10:50:29Z)
• Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs [48.50855715191533]
  Large audio language models are increasingly used for complex audio understanding tasks.<n>They struggle with temporal tasks that require precise temporal grounding, such as word alignment and speaker diarization.<n>We propose frame-level internal tool use, a method that trains audio LMs to use their own internal audio representations to perform temporal grounding directly.
  arXiv  Detail & Related papers  (2026-02-10T19:19:52Z)
• Language-based Audio Moment Retrieval [14.227865973426843]
  We propose and design a new task called audio moment retrieval (AMR)<n>Unlike conventional language-based audio retrieval tasks, AMR aims to predict relevant moments in untrimmed long audio based on a text query.<n>We build a dedicated dataset, Clotho-Moment, consisting of large-scale simulated audio recordings with moment annotations.<n>We then propose a DETR-based model, named Audio Moment DETR (AM-DETR), as a fundamental framework for AMR tasks.
  arXiv  Detail & Related papers  (2024-09-24T02:24:48Z)
• Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models [0.9285295512807729]
  The Audio Question Answering (AQA) task includes audio event classification, audio captioning, and open-ended reasoning.<n>LALMs excel in general audio understanding, but are limited in temporal reasoning.<n>This paper addresses these challenges and limitations in audio temporal reasoning.
  arXiv  Detail & Related papers  (2024-09-10T05:26:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.