PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency

• URL: http://arxiv.org/abs/2603.25620v1
• Date: Thu, 26 Mar 2026 16:34:34 GMT
• Title: PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency
• Abstract summary: Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants.<n>We propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning.
• Score: 13.864376043056387
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants across diverse domains. Yet there is no systematic method for verifying whether a persona agent's responses remain free of contradictions and factual inaccuracies throughout an interaction. A principle from interrogation methodology offers a lens: no matter how elaborate a fabricated identity, systematic interrogation will expose its contradictions. We apply this principle to propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning. PICon evaluates consistency along three core dimensions: internal consistency (freedom from self-contradiction), external consistency (alignment with real-world facts), and retest consistency (stability under repetition). Evaluating seven groups of persona agents alongside 63 real human participants, we find that even systems previously reported as highly consistent fail to meet the human baseline across all three dimensions, revealing contradictions and evasive responses under chained questioning. This work provides both a conceptual foundation and a practical methodology for evaluating persona agents before trusting them as substitutes for human participants. We provide the source code and an interactive demo at: https://kaist-edlab.github.io/picon/

Related papers

• AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility [104.46861849039357]
  Agent systems are advancing quickly across domains, but their evaluation remains fragmented.<n>The root problem is the lack of an open, agent-agnostic assessment interface.<n>We advocate Agentified Agent Assessment (AAA), where evaluation is performed by judge agents and all participants interact through standardized protocols.
  arXiv  Detail & Related papers  (2026-06-11T17:23:54Z)
• The Governance of Human-LLM Interaction: Safety Gating, Civility Steering, and Affective Default Lock-In [0.9558392439655014]
  Large language models (LLMs) increasingly mediate high-stakes interactions in finance, medicine, and mental-health support.<n>We frame interaction style as a governance object: provider-side alignment blocks harmful content, but stabilizes communicative defaults.<n>We show that prompt steerability and regression-to-default are observable indicators of provider control over communicative form.
  arXiv  Detail & Related papers  (2026-06-06T13:36:37Z)
• The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberation [68.98390038721963]
  We show that multi-agent discussion erases up to 72% of issue-critical facts.<n>This loss is consequential: retained evidence can reconstruct the issue misleadingly, final stances remain anchored in base-model priors, and a single malicious agent can inject misinformation into the shrinking shared context.<n>We call for evaluations that measure which facts, uncertainties, and legitimate disagreements survive interaction.
  arXiv  Detail & Related papers  (2026-06-02T02:15:40Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Dual Optimal: Make Your LLM Peer-like with Dignity [35.58723842086038]
  Current aligned language models exhibit a dual failure mode we term the Evasive Servant.<n>We propose the Dignified Peer framework, which counters servility with anti-sycophancy and trustworthiness.
  arXiv  Detail & Related papers  (2026-04-01T14:48:44Z)
• Beyond Preset Identities: How Agents Form Stances and Boundaries in Generative Societies [28.436766185842767]
  This paper proposes a novel mixed-methods framework combining computational virtual ethnography with quantitative socio-cognitive profiling.<n>It formalizes three new metrics: Innate Value Bias (IVB), Persuasion Sensitivity, and Trust-Action Decoupling (TAD)<n>Findings expose the fragility of static prompt engineering, providing a methodological and quantitative foundation for dynamic alignment in human-agent hybrid societies.
  arXiv  Detail & Related papers  (2026-03-24T16:38:46Z)
• Simulating and Understanding Deceptive Behaviors in Long-Horizon Interactions [18.182800471968132]
  We introduce the first simulation framework for probing and evaluating deception in large language models.<n>We conduct experiments across 11 frontier models, spanning both closed and open-source systems.<n>We find that deception is model-dependent, increases with event pressure, and consistently erodes supervisor trust.
  arXiv  Detail & Related papers  (2025-10-05T02:18:23Z)
• Can an Individual Manipulate the Collective Decisions of Multi-Agents? [53.01767232004823]
  M-Spoiler is a framework that simulates agent interactions within a multi-agent system to generate adversarial samples.<n>M-Spoiler introduces a stubborn agent that actively aids in optimizing adversarial samples.<n>Our findings confirm the risks posed by the knowledge of an individual agent in multi-agent systems.
  arXiv  Detail & Related papers  (2025-09-20T01:54:20Z)
• The Traitors: Deception and Trust in Multi-Agent Language Model Simulations [0.0]
  We introduce The Traitors, a multi-agent simulation framework inspired by social deduction games.<n>We develop a suite of evaluation metrics capturing deception success, trust dynamics, and collective inference quality.<n>Our initial experiments across DeepSeek-V3, GPT-4o-mini, and GPT-4o (10 runs per model) reveal a notable asymmetry.
  arXiv  Detail & Related papers  (2025-05-19T10:01:35Z)
• Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models [75.85319609088354]
  Sentient Agent as a Judge (SAGE) is an evaluation framework for large language models.<n>SAGE instantiates a Sentient Agent that simulates human-like emotional changes and inner thoughts during interaction.<n>SAGE provides a principled, scalable and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents.
  arXiv  Detail & Related papers  (2025-05-01T19:06:10Z)
• Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance [73.19687314438133]
  We study how reliance is affected by contextual features of an interaction. We find that contextual characteristics significantly affect human reliance behavior. Our results show that calibration and language quality alone are insufficient in evaluating the risks of human-LM interactions.
  arXiv  Detail & Related papers  (2024-07-10T18:00:05Z)
• Online Decision Mediation [72.80902932543474]
  Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior. In clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances.
  arXiv  Detail & Related papers  (2023-10-28T05:59:43Z)
• ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs [61.07130026622437]
  Large Language Models (LLMs) still struggle with natural language reasoning tasks. Motivated by the society of minds, we propose ReConcile. A multi-model multi-agent framework designed as a round table conference among diverse LLM agents.
  arXiv  Detail & Related papers  (2023-09-22T17:12:45Z)
• Interrogating the Black Box: Transparency through Information-Seeking Dialogues [9.281671380673306]
  We propose to construct an investigator agent to query a learning agent to investigate its adherence to an ethical policy. This formal dialogue framework is the main contribution of this paper. We argue that the introduced formal dialogue framework opens many avenues both in the area of compliance checking and in the analysis of properties of opaque systems.
  arXiv  Detail & Related papers  (2021-02-09T09:14:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.