CHARM: Character Hallucination for Multicultural Role Play Benchmark

• URL: http://arxiv.org/abs/2609.01352v1
• Date: Tue, 01 Sep 2026 14:57:14 GMT
• Title: CHARM: Character Hallucination for Multicultural Role Play Benchmark
• Abstract summary: We introduce CHARM, a benchmark of 40 real and fictional characters.<n>It probes two boundary types, Temporal (historical vs. modern) and Cross-Universe (entities outside a character's narrative or historical universe), using abstention-enabled multiple-choice questions.<n>We show that hallucination is driven predominantly by compliance failures.
• Score: 10.7522550144773
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Role-playing large language models (LLMs) are expected to adopt a character's style while also respecting that character's knowledge boundaries. Prior evaluations detect character hallucination but rarely distinguish whether errors arise from failure to recognize a boundary or from failure to comply despite recognition. We introduce CHARM, a multicultural benchmark of 40 real and fictional characters drawn from five cultural-linguistic regions, and validated by native reviewers. It probes two boundary types, Temporal (historical vs. modern) and Cross-Universe (entities outside a character's narrative or historical universe), using abstention-enabled multiple-choice questions. We propose a two-stage evaluation that separates Boundary-Awareness (explicit recognition that a query is out of scope) from Boundary-Compliance (abstention when answering concrete questions). Evaluations across six LLMs show that hallucination is driven predominantly by compliance failures. Models frequently acknowledge that a query lies outside the character's knowledge yet still provide factual, out-of-character answers. By re-posing the same questions to the target character, we confirm that a large fraction of these cases are verified parametric overrides; the model stores the relevant fact but fails to suppress it. We also observe systematic cultural variation in these failures, consistent with imbalances in how characters from different regions are represented in model knowledge.

Related papers

• PersonaShot: Benchmarking Person-Centric Narrative Continuity in Multi-Shot Video Generation [118.49150310327475]
  PersonaShot is the first person-centric benchmark for narrative continuity in multi-shot video generation.<n>It contains approximately 1,000 multi-shot segments and 16 metrics spanning physical continuity, affective dynamics, and cinematic grammar.
  arXiv  Detail & Related papers  (2026-08-17T15:34:54Z)
• Staying In Character: Perspective-Bounded Memory For Book-Based Role-Playing Agents [5.808135444478086]
  We propose REVERIEMEM, a three-layer memory architecture for book-based character agents.<n>The episodic layer stores first-person scene memories; the semantic layer stores visibility-tagged facts; and the personality layer stores situation-dependent speech and behaviour patterns.<n>For evaluation, we construct KBF-QA, a 4,386-question benchmark over eight novels for testing knowledge boundaries.
  arXiv  Detail & Related papers  (2026-06-24T09:37:49Z)
• Think Before you Write: QA-Guided Reasoning for Character Descriptions in Books [52.08691427871138]
  We propose a training framework that decouples reasoning from generation.<n>Our approach consists of a reasoning model that produces a structured QA reasoning trace and a generation model that conditions on this trace to produce the final character description.
  arXiv  Detail & Related papers  (2026-04-13T13:19:56Z)
• The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluation [17.386684382460242]
  Large language models (LLMs) are increasingly used to evaluate system outputs in tasks such as reasoning, question answering, and creative writing.<n>We test this ideal via controlled cue perturbations-synthetic metadata labels injected into evaluation prompts for six judge models.<n>We study six cue families: source, temporal, age, gender, ethnicity, and educational status.
  arXiv  Detail & Related papers  (2026-02-08T14:45:23Z)
• Fame Fades, Nature Remains: Disentangling the Character Identity of Role-Playing Agents [13.029517493304505]
  We propose a multidimensional construct that disentangles a character into two distinct layers: textbf(1) Parametric Identity, referring to character-specific knowledge encoded from the LLM's pre-training, and textbf(2) Attributive Identity, capturing fine-grained behavioral properties such as personality traits and moral values.<n>Our findings pinpoint negative social natures as the primary bottleneck in RPA fidelity, guiding future character construction and evaluation.
  arXiv  Detail & Related papers  (2026-01-08T08:33:40Z)
• Breaking the Assistant Mold: Modeling Behavioral Variation in LLM Based Procedural Character Generation [62.54606886226136]
  Procedural content generation has enabled vast virtual worlds through levels, maps, and quests, but large-scale character generation remains underexplored.<n>We identify two alignment-induced biases in existing methods.<n>We introduce PersonaWeaver, a framework that disentangles world-building from behavioral-building.
  arXiv  Detail & Related papers  (2026-01-06T20:18:01Z)
• Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts [2.2816872489992135]
  We introduce Beyond One World, a benchmark for character-grounded roleplay spanning 30 iconic heroes and 90 canon-specific versions.<n>We score responses for canonical accuracy and reasoning fidelity.<n>We propose Think-Act Matching, a metric that quantifies alignment between reasons and actions.
  arXiv  Detail & Related papers  (2025-10-16T06:39:27Z)
• Revisiting LLM Value Probing Strategies: Are They Robust and Expressive? [81.49470136653665]
  We evaluate the robustness and expressiveness of value representations across three widely used probing strategies.<n>We show that the demographic context has little effect on the free-text generation, and the models' values only weakly correlate with their preference for value-based actions.
  arXiv  Detail & Related papers  (2025-07-17T18:56:41Z)
• CharacterBench: Benchmarking Character Customization of Large Language Models [80.29164862682063]
  We propose CharacterBench, the largest bilingual generative benchmark, with 22,859 human-annotated samples covering 3,956 characters.<n>We define 11 dimensions of 6 aspects, classified as sparse and dense dimensions based on whether character features evaluated by specific dimensions manifest in each response.<n>We also develop CharacterJudge model for cost-effective and stable evaluations.
  arXiv  Detail & Related papers  (2024-12-16T15:55:34Z)
• Mitigating Hallucination in Fictional Character Role-Play [19.705708068900076]
  We focus on the evaluation and mitigation of hallucination in fictional character role-play. We introduce a dataset with over 2,000 characters and 72,000 interviews, including 18,000 adversarial questions. We propose RoleFact, a role-playing method that mitigates hallucination by modulating the influence of parametric knowledge.
  arXiv  Detail & Related papers  (2024-06-25T03:56:33Z)
• TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models [55.51648393234699]
  We introduce TimeChara, a new benchmark designed to evaluate point-in-time character hallucination in role-playing LLMs. We propose Narrative-Experts, a method that decomposes the reasoning steps and utilizes narrative experts to reduce point-in-time character hallucinations effectively.
  arXiv  Detail & Related papers  (2024-05-28T10:19:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.