MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques

• URL: http://arxiv.org/abs/2608.11755v1
• Date: Wed, 12 Aug 2026 07:49:17 GMT
• Title: MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
• Abstract summary: We introduce MUSECRITIC, a semi-scalar reward model that generates a natural-language critique covering five aesthetic dimensions.<n>Using MUSECRITIC with GRPO improves Muse-0.6B on all nine aesthetic metrics from SongEval and Audiobox Aesthetics.
• Score: 58.249736525099614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-form song generation models continue to improve in duration, structural integrity, and acoustic complexity, making reliable aesthetic rewards increasingly important for aligning these models with human preferences. However, reward models for complete songs remain limited, and existing evaluators typically predict scores in a single forward pass without providing readable explanations. We introduce MUSECRITIC, a semi-scalar reward model that generates a natural-language critique covering five aesthetic dimensions and uses it as an intermediate representation to predict continuous reward scores. MUSECRITIC follows a two-stage training pipeline: a teacher model first provides high-quality critiques for supervised fine-tuning, after which the fine-tuned model generates its own critiques for reward learning, mitigating distribution shift between training and inference. On an in-domain test set of 200 SongEval songs, MUSECRITIC reduces macro-averaged mean squared error from 0.2875 to 0.2316 and improves macro-averaged LCC, SRCC, and Kendall's tau to 0.9068, 0.8838, and 0.7178, respectively. On the out-of-domain Music Arena benchmark with 733 preference pairs, it achieves the highest accuracy of 71.35%. Moreover, using MUSECRITIC with GRPO improves Muse-0.6B on all nine aesthetic metrics from SongEval and Audiobox Aesthetics. These results demonstrate that critique-conditioned reward modeling reduces scoring error and provides an effective optimization signal for song generation. The project repository is available at https://github.com/WuqnEl/MuseCritic.

Related papers

• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time [51.256131853751754]
  We show that teaching reward models to produce explicit, multi-dimensional critiques transforms them from passive evaluators into active optimization tools.<n>We introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data.<n>The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models.
  arXiv  Detail & Related papers  (2026-04-13T15:38:09Z)
• SpecTM: Spectral Targeted Masking for Trustworthy Foundation Models [0.0]
  SpecTM is a physics-informed masking design that encourages the reconstruction of targeted bands from cross-spectral context.<n>It achieves R2 = 0.695 (current week) and R2 = 0.620 (8-day-ahead) predictions surpassing all baseline models by (+34% (0.51 Ridge) and +99% (SVR 0.31)<n>It outperforms strong baselines with 2.2x superior label efficiency under extreme scarcity.
  arXiv  Detail & Related papers  (2026-03-23T15:26:51Z)
• Scaling Self-Supervised Representation Learning for Symbolic Piano Performance [52.661197827466886]
  We study the capabilities of generative autoregressive transformer models trained on large amounts of symbolic solo-piano transcriptions.<n>We use a comparatively smaller, high-quality subset to finetune models to produce musical continuations, perform symbolic classification tasks, and produce general-purpose contrastive MIDI embeddings.
  arXiv  Detail & Related papers  (2025-06-30T14:00:14Z)
• Aligning Text-to-Music Evaluation with Human Preferences [63.08368388389259]
  We study the design space of reference-based divergence metrics for evaluating generative acoustic text-to-music (TTM) models.<n>We find that not only is the standard FAD setup inconsistent on both synthetic and human preference data, but that nearly all existing metrics fail to effectively capture desiderata.<n>We propose a new metric, the MAUVE Audio Divergence (MAD), computed on representations from a self-supervised audio embedding model.
  arXiv  Detail & Related papers  (2025-03-20T19:31:04Z)
• Self-Generated Critiques Boost Reward Modeling for Language Models [57.60881438647227]
  Critic-RM is a framework that improves reward models using self-generated critiques without extra supervision.<n> Experiments show that Critic-RM improves reward modeling accuracy by 3.7%-7.3% compared to standard reward models and LLM judges.
  arXiv  Detail & Related papers  (2024-11-25T18:28:26Z)
• CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling [27.86204841898399]
  Reward modeling in large language models is susceptible to reward hacking.<n>We propose Context-Aware Reward Modeling (CARMO) to mitigate this problem.<n>We establish a new state-of-the-art performance in zero-shot settings for generative models, achieving a 2.1% improvement on Reward Bench.
  arXiv  Detail & Related papers  (2024-10-28T21:18:49Z)
• Critique-out-Loud Reward Models [20.631830494414096]
  We introduce Critique-out-Loud (CLoud) reward models. CLoud reward models operate by first generating a natural language critique of the assistant's response. We demonstrate the success of CLoud reward models for both Llama-3-8B and 70B base models.
  arXiv  Detail & Related papers  (2024-08-21T17:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.