When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs

• URL: http://arxiv.org/abs/2605.29420v1
• Date: Thu, 28 May 2026 06:14:07 GMT
• Title: When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs
• Abstract summary: Persona prompting is widely used to steer large language models, but its practical value remains unclear.<n>We compare four prompting conditions across 1,140 open-ended questions spanning 38 expert roles and six domains.<n>Role prompting performs best on advisory questions and in domains such as medicine and psychology.
• Score: 6.180594609315986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Persona prompting is widely used to steer large language models, yet its practical value remains unclear. Prior work often evaluates persona prompting using aggregate scores, making it difficult to determine whether expert-role prompting consistently improves response quality or instead changes responses along different quality dimensions. We study this question through a controlled comparison of four prompting conditions across 1,140 open-ended questions spanning 38 expert roles and six domains: no role prompt, a generic domain-expert prompt, embedding-based role retrieval, and a hybrid retrieval method combining embedding search with LLM-based role selection. Aggregate results show only small overall differences between conditions. However, metric-level analysis reveals a consistent tradeoff that aggregate averages obscure: role prompting systematically increases expertise depth while reducing clarity. These effects are highly conditional rather than universal. Role prompting performs best on advisory questions and in domains such as medicine and psychology, where structured expert framing and risk communication are intrinsically valuable. In contrast, baseline prompting performs better on conceptual and explanatory questions in finance, legal, science, and technology domains, where concise plain-language explanation is more important. We further show that hybrid retrieval significantly improves over embedding-only role selection, although better role retrieval does not eliminate the broader expertise-depth versus clarity tradeoff. Overall, our findings suggest that persona prompting primarily reshapes response characteristics rather than broadly improving capability, and that multi-metric evaluation is necessary for understanding its effects.

Related papers

• Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation [12.756079464205982]
  We assess how sensitive popular summarization metrics are to both informational and persona differences.<n>We additionally conduct an expert human evaluation, measuring summary preferences based on information satisfaction.
  arXiv  Detail & Related papers  (2026-08-14T16:41:23Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthromorphism, and Maxims [54.31707479192195]
  Large language models (LLMs) are being increasingly used to answer subjective, information-seeking questions.<n>Existing evaluations for subjective cultural queries largely focus on factual correctness, ignoring how the response is framed.<n>We introduce FRANZ, an automated FRAmework for respoNse characteriZation to conduct communicative audit of LLM responses along four dimensions.
  arXiv  Detail & Related papers  (2026-06-01T17:01:31Z)
• Adaptive Multi-Agent Response Refinement in Conversational Systems [33.2240994465021]
  Large Language Models (LLMs) have demonstrated remarkable success in conversational systems by generating human-like responses.<n>They can fall short, especially when required to account for personalization or specific knowledge.<n>We propose refining responses through a multi-agent framework, where each agent is assigned a specific role for each aspect.
  arXiv  Detail & Related papers  (2025-11-11T14:48:34Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detection [0.0]
  This paper introduces Reinforced Unanswerability Learning (RUL), a novel hybrid training paradigm for large language models (LLMs)<n>RUL integrates a discriminative unanswerability prediction head with the LLM's generative core, guided by a multi-stage learning strategy.<n>Experiments demonstrate RUL's superior performance, achieving significantly higher accuracy in unanswerability detection across sentence, paragraph, and ranking levels.
  arXiv  Detail & Related papers  (2025-06-01T17:59:27Z)
• Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization [49.95746521480879]
  We introduce PKUE (Precise Knowledge Utilization Enhancement), which fine-tunes the model on self-generated responses to precise and simple factual questions.<n>Extensive experiments demonstrate that PKUE significantly improves LLM overall performance.
  arXiv  Detail & Related papers  (2025-02-26T13:34:52Z)
• Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions [25.158868133182025]
  We present a method for evaluating the output of generative large language models (LLMs)<n>We use ranking models trained on annotated document collections as a substitute for explicit relevance.<n>In a user study, our method correlates with the preferences of a human expert.
  arXiv  Detail & Related papers  (2024-08-19T09:27:45Z)
• When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models [34.831938712535084]
  Commercial AI systems commonly define the role of the Large Language Models (LLM) in system prompts. It remains unclear how different personas affect a model's performance on objective tasks. We curate a list of 162 roles covering 6 types of interpersonal relationships and 8 domains of expertise.
  arXiv  Detail & Related papers  (2023-11-16T17:48:55Z)
• ExpertQA: Expert-Curated Questions and Attributed Answers [51.68314045809179]
  We conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality. We collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.
  arXiv  Detail & Related papers  (2023-09-14T16:54:34Z)
• Multi-Stage Conversational Passage Retrieval: An Approach to Fusing Term Importance Estimation and Neural Query Rewriting [56.268862325167575]
  We tackle conversational passage retrieval (ConvPR) with query reformulation integrated into a multi-stage ad-hoc IR system. We propose two conversational query reformulation (CQR) methods: (1) term importance estimation and (2) neural query rewriting. For the former, we expand conversational queries using important terms extracted from the conversational context with frequency-based signals. For the latter, we reformulate conversational queries into natural, standalone, human-understandable queries with a pretrained sequence-tosequence model.
  arXiv  Detail & Related papers  (2020-05-05T14:30:20Z)
• Review-guided Helpful Answer Identification in E-commerce [38.276241153439955]
  Product-specific community question answering platforms can greatly help address the concerns of potential customers. The user-provided answers on such platforms often vary a lot in their qualities. Helpfulness votes from the community can indicate the overall quality of the answer, but they are often missing.
  arXiv  Detail & Related papers  (2020-03-13T11:34:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.