Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

• URL: http://arxiv.org/abs/2608.02831v1
• Date: Mon, 03 Aug 2026 19:46:13 GMT
• Title: Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning
• Abstract summary: We introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving rewards.<n>AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group.<n>Our analysis shows that the gains scale with the capability of the generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and growth.
• Score: 57.5472764796581
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.

Related papers

• AffectOmni: RL-Verifiable People-Centric Grounded Affective Reasoning for Social and Art-Related Scenes [59.134503227559634]
  Multimodal large language models (MLLMs) achieve strong performance on VQA and scene understanding.<n>Models may predict correct answers while neglecting people-centric cues such as micro expressions and body language.<n>We propose Affect Omni, a framework for verifiable affective reasoning.
  arXiv  Detail & Related papers  (2026-08-24T11:25:04Z)
• X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment [38.53954341285787]
  X$3$-OPD is a cross-modal on-policy distillation framework for large audio-language models.<n>It transfers reasoning capabilities from a powerful text teacher to an audio-language student.<n>X$3$-OPD substantially improves audio-grounded reasoning and chain-of-thought quality.
  arXiv  Detail & Related papers  (2026-07-23T17:35:20Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following [17.52339026461815]
  We introduce a novel dynamic rubric-based evaluation paradigm that adaptively decomposes complex audio captions into a variable number of independent, verifiable binary rubric items.<n>We benchmark this capability with the AnyAudio-Judge Bench, a comprehensive, bilingual benchmark comprising 7,920 meticulously curated samples.<n>Experiments demonstrate that AnyAudio-Judge not only significantly enhances zero-shot alignment detection compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-02T04:00:32Z)
• Audio-DeepThinker: Progressive Reasoning-Aware Reinforcement Learning for High-Quality Chain-of-Thought Emergence in Audio Language Models [33.669071786618495]
  Existing methods for enhancing audio reasoning rely on supervised chain-of-thought fine-tuning or on reinforcement learning.<n>We propose Audio-DeepThinker, a framework built on two core ideas.<n>Stage 1 trains on foundational audio QA with the hybrid reward to foster basic reasoning patterns, while Stage 2 shifts to acoustically challenging boundary cases.<n>Audio-DeepThinker achieves state-of-the-art results on MMAR (74.0%), MMAU-test-mini (78.5%), and MMSU (77.26%)
  arXiv  Detail & Related papers  (2026-04-20T12:43:00Z)
• Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs [53.41256816230278]
  Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion.<n>While excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception.<n>We propose Unified Audio (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components.
  arXiv  Detail & Related papers  (2026-04-14T09:30:12Z)
• Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution [79.98699884805636]
  Reasoning Execution by Multiple Listeners (REMUL) is a multi-party reinforcement learning approach.<n>REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful.<n>Speakers are rewarded for producing reasoning that is clear to listeners.
  arXiv  Detail & Related papers  (2026-02-18T02:55:55Z)
• Step-Audio-R1 Technical Report [70.37077572409319]
  We introduce Step-Audio-R1, the first audio reasoning model that successfully unlocks reasoning capabilities in the audio domain.<n>Our model exhibits strong audio reasoning capabilities, surpassing Gemini 2.5 Pro and achieving performance comparable to the state-of-the-art Gemini 3 Pro.
  arXiv  Detail & Related papers  (2025-11-19T20:12:50Z)
• AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoning [3.949628618389608]
  AURA is a benchmark for evaluating the cross-modal reasoning capabilities of Audio-Visual Large Language Models (AV-LLMs) and Omni-modal Language Models (OLMs)<n>AURA includes questions across six challenging cognitive domains, such as causality, timbre and pitch, tempo and AV synchronization, unanswerability, implicit distractions, and skill profiling.<n>We propose a novel metric, AuraScore, which addresses the lack of robust tools for evaluating reasoning fidelity.
  arXiv  Detail & Related papers  (2025-08-10T20:06:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.