PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Models

• URL: http://arxiv.org/abs/2601.03531v1
• Date: Wed, 07 Jan 2026 02:44:38 GMT
• Title: PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Models
• Abstract summary: Large Audio-Language Models (LALMs) have demonstrated strong performance in audio understanding and generation.<n>We formalize the task of Personalized LALMs (PALM) for recognizing personal concepts and reasoning within personal context.<n>Our experiments on representative open-source LALMs, show that existing training-free prompting and supervised fine-tuning strategies, while yield improvements, remains limited in modeling personalized knowledge.
• Score: 21.26163892337167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Audio-Language Models (LALMs) have demonstrated strong performance in audio understanding and generation. Yet, our extensive benchmarking reveals that their behavior is largely generic (e.g., summarizing spoken content) and fails to adequately support personalized question answering (e.g., summarizing what my best friend says). In contrast, human conditions their interpretation and decision-making on each individual's personal context. To bridge this gap, we formalize the task of Personalized LALMs (PALM) for recognizing personal concepts and reasoning within personal context. Moreover, we create the first benchmark (PALM-Bench) to foster the methodological advances in PALM and enable structured evaluation on several tasks across multi-speaker scenarios. Our extensive experiments on representative open-source LALMs, show that existing training-free prompting and supervised fine-tuning strategies, while yield improvements, remains limited in modeling personalized knowledge and transferring them across tasks robustly. Data and code will be released.

Related papers

• JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions [66.71387365429062]
  JASTIN is an instruction-driven audio evaluation framework.<n>It formulates audio assessment as a self-instructed reasoning task.<n>It consistently outperforms general MLLMs across speech, sound, music, and out-of-domain evaluation tasks.
  arXiv  Detail & Related papers  (2026-05-06T05:18:42Z)
• Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models [67.09110757873142]
  We introduce Text2DistBench, a reading comprehension benchmark for evaluating LLMs' ability to infer distributional knowledge from natural language.<n>Built from real-world YouTube comments about movie and music entities, the benchmark provides models with entity metadata and associated comments.<n>To support reliable and long-term evaluation, the construction pipeline of Text2DistBench is fully automated and continuously updated to incorporate newly emerging entities.
  arXiv  Detail & Related papers  (2026-03-13T19:26:08Z)
• Enhancing Speech Large Language Models through Reinforced Behavior Alignment [5.647822820528311]
  This paper introduces a framework termed Reinforced Behavior Alignment (RBA) to bolster the language generation proficiency of SpeechLMs.<n>Instead of relying on supervised fine-tuning from human annotations, RBA employs a self-synthesis methodology to generate extensive, high-fidelity alignment data.<n> Experimental results demonstrate that this method effectively enhances the instruction-following capabilities of SpeechLMs.
  arXiv  Detail & Related papers  (2025-08-25T07:31:48Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations [112.81207927088117]
  PersonaConvBench is a benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs)<n>We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements.
  arXiv  Detail & Related papers  (2025-05-20T09:13:22Z)
• PersoBench: Benchmarking Personalized Response Generation in Large Language Models [6.8046587254152735]
  We present a new benchmark, PersoBench, to evaluate the personalization ability of large language models (LLMs) in persona-aware dialogue generation. Our analysis, conducted on three well-known persona-aware datasets, evaluates multiple dimensions of response quality, including fluency, diversity, coherence, and personalization.
  arXiv  Detail & Related papers  (2024-10-04T07:29:41Z)
• A Large-Scale Evaluation of Speech Foundation Models [110.95827399522204]
  We establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the foundation model paradigm for speech. We propose a unified multi-tasking framework to address speech processing tasks in SUPERB using a frozen foundation model followed by task-specialized, lightweight prediction heads.
  arXiv  Detail & Related papers  (2024-04-15T00:03:16Z)
• Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models [54.21695754082441]
  We propose a framework to teach Large Language Models (LLMs) to generate explainable stock predictions. A reflective agent learns how to explain past stock movements through self-reasoning, while the PPO trainer trains the model to generate the most likely explanations. Our framework can outperform both traditional deep-learning and LLM methods in prediction accuracy and Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-02-06T03:18:58Z)
• Can Large Language Models Understand Context? [17.196362853457412]
  This paper introduces a context understanding benchmark by adapting existing datasets to suit the evaluation of generative models. Experimental results indicate that pre-trained dense models struggle with understanding more nuanced contextual features when compared to state-of-the-art fine-tuned models. As LLM compression holds growing significance in both research and real-world applications, we assess the context understanding of quantized models under in-context-learning settings.
  arXiv  Detail & Related papers  (2024-02-01T18:55:29Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech [107.81472531864195]
  Text language models have shown remarkable zero-shot capability in generalizing to unseen tasks when provided with well-formulated instructions. We present Dynamic-SUPERB, a benchmark for building universal speech models capable of leveraging instruction tuning to perform multiple tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2023-09-18T06:43:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.