Thinking Before Speaking: A Role-playing Model with Mindset

• URL: http://arxiv.org/abs/2409.13752v1
• Date: Sat, 14 Sep 2024 02:41:48 GMT
• Title: Thinking Before Speaking: A Role-playing Model with Mindset
• Authors: Baohua Zhang, Yongyi Huang, Wenyao Cui, Huaping Zhang,
• Abstract summary: Large Language Models (LLMs) are skilled at simulating human behaviors. These models tend to perform poorly when confronted with knowledge that the assumed role does not possess. We propose a Thinking Before Speaking (TBS) model in this paper.
• Score: 0.6428333375712125
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Role-playing is an easy task for Large Language Models (LLMs), as they are skilled at simulating human behaviors. Many current studies have enabled LLMs to generate responses in the tone of a specific role by fine-tuning the models or using specialized prompts. However, it is typically easy to recognize when a role is being played by LLMs. These models tend to perform poorly when confronted with knowledge that the assumed role does not possess, or a question that requires the specific experience or logic of the role to answer. To address this problem and make LLMs act more like real roles, we propose a Thinking Before Speaking (TBS) model in this paper. Unlike other studies, we first extend the data based on the character's real-life scenarios and the historical dialogue, supplementing each pair of dialogue with the character's mindset. Then we add few data points that include elements beyond the role's knowledge, and fine-tune the LLMs. This approach can help LLMs adopt the role's thought process and logic, avoiding responses that fall outside the role's knowledge base. We have also prepared a dataset and evaluation metrics to test these capabilities. Experimental results show that our TBS model can better emulate a role in terms of tone, knowledge, and mindset.

Related papers

• Benchmarking Bias in Large Language Models during Role-Playing [21.28427555283642]
  We introduce BiasLens, a fairness testing framework designed to expose biases in Large Language Models (LLMs) during role-playing. Our approach uses LLMs to generate 550 social roles across a comprehensive set of 11 demographic attributes, producing 33,000 role-specific questions. Using the generated questions as the benchmark, we conduct extensive evaluations of six advanced LLMs released by OpenAI, Mistral AI, Meta, Alibaba, and DeepSeek. Our benchmark reveals 72,716 biased responses across the studied LLMs, with individual models yielding between 7,754 and 16,963 biased responses.
  arXiv  Detail & Related papers  (2024-11-01T13:47:00Z)
• Language Models Show Stable Value Orientations Across Diverse Role-Plays [4.906478894661688]
  We show that large language models (LLMs) exhibit consistent value orientations despite adopting diverse personas. We introduce the role-play-at-scale methodology, which involves prompting LLMs with randomized, diverse personas. This approach reveals consistent patterns in LLM responses across diverse role-play scenarios, indicating deeply encoded inherent tendencies.
  arXiv  Detail & Related papers  (2024-08-16T23:24:10Z)
• Capturing Minds, Not Just Words: Enhancing Role-Playing Language Models with Personality-Indicative Data [58.92110996840019]
  We propose to enhance role-playing language models (RPLMs) via personality-indicative data. Specifically, we leverage questions from psychological scales and distill advanced RPAs to generate dialogues that grasp the minds of characters. Experimental results validate that RPLMs trained with our dataset exhibit advanced role-playing capabilities for both general and personality-related evaluations.
  arXiv  Detail & Related papers  (2024-06-27T06:24:00Z)
• Prompt Framework for Role-playing: Generation and Evaluation [3.2845546753303867]
  Large language models (LLM) have demonstrated remarkable abilities in generating natural language, understanding user instruction, and mimicking human language use. We introduce a framework that uses prompts to leverage the state-of-the-art (SOTA) LLMs to construct role-playing dialogue datasets and evaluate the role-playing performance.
  arXiv  Detail & Related papers  (2024-06-02T06:09:56Z)
• Character is Destiny: Can Large Language Models Simulate Persona-Driven Decisions in Role-Playing? [59.0123596591807]
  We benchmark the ability of Large Language Models in persona-driven decision-making. We investigate whether LLMs can predict characters' decisions provided with the preceding stories in high-quality novels. The results demonstrate that state-of-the-art LLMs exhibit promising capabilities in this task, yet there is substantial room for improvement.
  arXiv  Detail & Related papers  (2024-04-18T12:40:59Z)
• On the Decision-Making Abilities in Role-Playing using Large Language Models [6.550638804145713]
  Large language models (LLMs) are increasingly utilized for role-playing tasks. This paper focuses on evaluating the decision-making abilities of LLMs post role-playing.
  arXiv  Detail & Related papers  (2024-02-29T02:22:23Z)
• Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment [62.898963074989766]
  We introduce Ditto, a self-alignment method for role-play. This method creates a role-play training set comprising 4,000 characters, surpassing the scale of currently available datasets by tenfold. We present the first comprehensive cross-supervision alignment experiment in the role-play domain.
  arXiv  Detail & Related papers  (2024-01-23T03:56:22Z)
• Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves [57.974103113675795]
  We present a method named Rephrase and Respond' (RaR) which allows Large Language Models to rephrase and expand questions posed by humans. RaR serves as a simple yet effective prompting method for improving performance. We show that RaR is complementary to the popular Chain-of-Thought (CoT) methods, both theoretically and empirically.
  arXiv  Detail & Related papers  (2023-11-07T18:43:34Z)
• Character-LLM: A Trainable Agent for Role-Playing [67.35139167985008]
  Large language models (LLMs) can be used to serve as agents to simulate human behaviors. We introduce Character-LLM that teach LLMs to act as specific people such as Beethoven, Queen Cleopatra, Julius Caesar, etc.
  arXiv  Detail & Related papers  (2023-10-16T07:58:56Z)
• In-Context Impersonation Reveals Large Language Models' Strengths and Biases [56.61129643802483]
  We ask LLMs to assume different personas before solving vision and language tasks. We find that LLMs pretending to be children of different ages recover human-like developmental stages. In a language-based reasoning task, we find that LLMs impersonating domain experts perform better than LLMs impersonating non-domain experts.
  arXiv  Detail & Related papers  (2023-05-24T09:13:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.