TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Abstract Overview
The paper addresses a central limitation of large audio-language models: they typically describe an entire clip but do not assign precise timestamps to events, speakers, or musical elements. It introduces TEMPO, a unified post-training framework for timestamped generation across five tasks spanning speech, general audio, and music: multi-speaker ASR, speaker diarization, audio temporal grounding, dense audio captioning, and timestamped music captioning. The method centers on supervised fine-tuning with atomic timestamp tokens, a time-aware multimodal projector that injects wall-clock information into audio features, and a distance-aware Gaussian timestamp loss, trained with a synthetic-to-real curriculum. On top of this, the authors add GRPO-based reinforcement learning with verifiable temporal rewards as a refinement stage, and evaluate the model on a benchmark built from established corpora covering about 10K test examples.
Novelty
The work presents what it describes as the first unified large audio-language model for timestamping across speech, sound, and music within a single decoder and output format. It also introduces the first application of reinforcement learning with verifiable temporal rewards to unified multi-task audio timestamping, extending beyond prior systems that were limited to fewer tasks or supervised-only training.
Results
TEMPO shows its clearest gains on speech tasks: compared with Qwen3-Omni, the multi-task RL model reduces multi-speaker ASR WER from 69.7% to 43.5% and improves diarization mIoU from 44.4% to 71.1%, while raising speaker-labeled F1 from 31.5% to 56.3%. On dense audio captioning, it exceeds TimeAudio and Qwen3-Omni in event localization metrics, reaching 59.3% event F1 and 68.5% mIoU after multi-task RL. Ablations indicate that most of the performance gains come from the supervised fine-tuning design and synthetic-to-real curriculum, with reinforcement learning providing smaller, refinement-level improvements across most tasks.
Key Points
- TEMPO unifies five timestamped audio tasks across speech, sound, and music in one large audio-language model with a shared decoder and task-tagged prompting.
- Its main technical recipe combines atomic timestamp tokens, wall-clock-aware audio projection, and a distance-aware timestamp loss, trained in a two-stage synthetic-to-real curriculum.
- Empirically, the strongest gains are on multi-speaker ASR and diarization, while reinforcement learning acts mainly as a refinement step rather than the primary source of improvement.