Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity

• URL: http://arxiv.org/abs/2503.00771v2
• Date: Sat, 12 Apr 2025 09:19:52 GMT
• Title: Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
• Authors: Yupu Hao, Pengfei Cao, Zhuoran Jin, Huanxuan Liao, Yubo Chen, Kang Liu, Jun Zhao,
• Abstract summary: We introduce a novel benchmark ETAPP for evaluating personalized tool invocation.<n>To improve the accuracy of our evaluation, we propose a key-point-based evaluation method.<n>The effectiveness of our preference-setting and key-point-based evaluation method is also validated.
• Score: 17.723293304671877
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. However, most of the current benchmarks primarily focus on either personalization of text generation or direct tool-utilizing, without considering both. In this work, we introduce a novel benchmark ETAPP for evaluating personalized tool invocation, establishing a sandbox environment, and a comprehensive dataset of 800 testing cases covering diverse user profiles. To improve the accuracy of our evaluation, we propose a key-point-based LLM evaluation method, mitigating biases in the LLM-as-a-judge system by manually annotating key points for each test case and providing them to LLM as the reference. Additionally, we evaluate the excellent LLMs and provide an in-depth analysis. Furthermore, we investigate the impact of different tool-invoking strategies on LLMs' personalization performance and the effects of fine-tuning in our task. The effectiveness of our preference-setting and key-point-based evaluation method is also validated. Our findings offer insights into improving personalized LLM agents. Our Code is available at https://github.com/hypasd-art/ETAPP.

Related papers

• Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalization [68.79814761867314]
  We propose Difference-aware Personalization Learning (DPL) to enhance Large Language Models (LLMs) personalization.<n>DPL strategically selects representative users for comparison and establishes a structured standard to extract task-relevant differences.<n>Experiments on real-world datasets demonstrate that DPL significantly enhances LLM personalization.
  arXiv  Detail & Related papers  (2025-03-04T09:53:26Z)
• Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference [63.03859517284341]
  An automatic evaluation framework aims to rank LLMs based on their alignment with human preferences.<n>An automatic LLM bencher consists of four components: the input set, the evaluation model, the evaluation type and the aggregation method.
  arXiv  Detail & Related papers  (2024-12-31T17:46:51Z)
• AI PERSONA: Towards Life-long Personalization of LLMs [28.21436822048565]
  We introduce the task of life-long personalization of large language models. We will release all codes and data for building and benchmarking life-long personalized LLM systems.
  arXiv  Detail & Related papers  (2024-12-17T17:17:03Z)
• MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time [50.41806216615488]
  Large Language Models (LLMs) acquire extensive knowledge and remarkable abilities from extensive text corpora. To make LLMs more usable, aligning them with human preferences is essential. We propose an effective method, textbf MetaAlign, which aims to help LLMs dynamically align with various explicit or implicit preferences specified at inference time.
  arXiv  Detail & Related papers  (2024-10-18T05:31:13Z)
• Few-shot Personalization of LLMs with Mis-aligned Responses [40.0349773257245]
  This paper proposes a new approach for a few-shot personalization of large language models (LLMs) Our key idea is to learn a set of personalized prompts for each user by progressively improving the prompts using LLMs. During an iterative process of prompt improvement, we incorporate the contexts of mis-aligned responses by LLMs.
  arXiv  Detail & Related papers  (2024-06-26T18:29:12Z)
• Exploring Safety-Utility Trade-Offs in Personalized Language Models [26.792174008353008]
  We show that large language models (LLMs) suffer from personalization bias, where their performance is impacted when they are personalized to a user's identity.<n>We quantify personalization bias by evaluating the performance of LLMs along two axes - safety and utility.<n>We discuss several strategies to mitigate personalization bias using preference tuning and prompt-based defenses.
  arXiv  Detail & Related papers  (2024-06-17T00:17:11Z)
• RepEval: Effective Text Evaluation with LLM Representation [55.26340302485898]
  RepEval is a metric that leverages the projection of Large Language Models (LLMs) representations for evaluation. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics.
  arXiv  Detail & Related papers  (2024-04-30T13:50:55Z)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models [52.98743860365194]
  We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN) At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents.
  arXiv  Detail & Related papers  (2024-01-02T18:53:13Z)
• T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step [69.64348626180623]
  Large language models (LLM) have achieved remarkable performance on various NLP tasks. How to evaluate and analyze the tool-utilization capability of LLMs is still under-explored. We introduce T-Eval to evaluate the tool utilization capability step by step.
  arXiv  Detail & Related papers  (2023-12-21T17:02:06Z)
• Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction [15.793007223588672]
  Large Language Models (LLMs) have demonstrated exceptional capabilities in generalizing to new tasks in a zero-shot or few-shot manner. We investigate various LLMs in different sizes, ranging from 250M to 540B parameters and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios.
  arXiv  Detail & Related papers  (2023-05-10T21:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.