PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators

• URL: http://arxiv.org/abs/2604.25840v1
• Date: Tue, 28 Apr 2026 16:46:25 GMT
• Title: PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators
• Abstract summary: We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically grounded diagnostics of depression patient simulator behavior.<n>Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses.<n>Results from a human study demonstrate that our benchmark is strongly aligned with expert judgments.
• Score: 14.323763649788907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulating depressed patients is particularly challenging, as safety constraints and high patient variability complicate simulations and underscore the need for simulators that capture diverse and realistic patient behaviors. However, existing evaluations heavily rely on LLM-judges with poorly specified prompts and do not assess behavioral diversity. We introduce PSI-Bench, an automatic evaluation framework that provides interpretable, clinically grounded diagnostics of depression patient simulator behavior across turn-, dialogue-, and population-level dimensions. Using PSI-Bench, we benchmark seven LLMs across two simulator frameworks and find that simulators produce overly long, lexically diverse responses, show reduced variability, resolve emotions too quickly, and follow a uniform negative-to-positive trajectory. We also show that the simulation framework has a larger impact on fidelity than the model scale. Results from a human study demonstrate that our benchmark is strongly aligned with expert judgments. Our work reveals key limitations of current depression patient simulators and provides an interpretable, extensible benchmark to guide future simulator design and evaluation.

Related papers

• SSR: Can Simulated Patients Learn to Stigmatize Themselves? Modeling Self-Stigma through Internal Monologue [22.38071918112141]
  We introduce a novel simulation framework grounded in the psychological 3A1H model of self-stigmatization.<n>Our core innovation is the creation of a textbfStigmatized Self-Reflection dataset, where we augment mental health dialogues with internal monologues that reflect stigma-aware reasoning.
  arXiv  Detail & Related papers  (2026-06-06T16:50:57Z)
• Patients With Personality: Realistic Patient Simulation through Controlled Diversity and Selective Disclosure [24.569940011188994]
  PatientsWithPersonality (PWP) is a patient simulation framework that generates realistic yet diverse virtual patient responses.<n>Grounded in HEXACO, a six-dimensional personality space used to quantify and parameterize human behavioral traits, our approach enables fine-grained control over conversational style, cooperativeness, and information disclosure.
  arXiv  Detail & Related papers  (2026-05-13T15:26:15Z)
• Beyond Prompt: Fine-grained Simulation of Cognitively Impaired Standardized Patients via Stochastic Steering [58.85421622061983]
  We propose StsPatient for the fine-grained simulation of cognitively impaired patients.<n>We capture domain-specific features by extracting steering vectors from contrastive pairs of instructions and responses.<n>StsPatient significantly outperforms baselines in both clinical authenticity and severity controllability.
  arXiv  Detail & Related papers  (2026-04-14T02:37:46Z)
• Multi-Stage Patient Role-Playing Framework for Realistic Clinical Interactions [2.1897719729390173]
  We propose the first Chinese patient simulation dataset (Ch-PatientSim)<n>Patients are simulated based on a five-dimensional persona structure.<n>To address issues of the persona class imbalance, a portion of the dataset is augmented using few-shot generation, followed by manual verification.
  arXiv  Detail & Related papers  (2026-01-16T02:34:22Z)
• Adversarial Training for Failure-Sensitive User Simulation in Mental Health Dialogue Optimization [1.964225730132177]
  A key property of effective simulators is their ability to expose failure modes of the systems they evaluate.<n>We present an adversarial training framework that improves user simulator realism through a competitive dynamic between a generator (user simulator) and a discriminator.
  arXiv  Detail & Related papers  (2025-12-23T21:21:08Z)
• MindEval: Benchmarking Language Models on Multi-turn Mental Health Support [10.524387723320432]
  MindEval is a framework for automatically evaluating language models in realistic, multi-turn mental health therapy conversations.<n>We quantitatively validate the realism of our simulated patients against human-generated text and by demonstrating strong correlations between automatic and human expert judgments.<n>We evaluate 12 state-of-the-art LLMs and show that all models struggle, scoring below 4 out of 6 on average, with particular weaknesses in problematic AI-specific patterns of communication.
  arXiv  Detail & Related papers  (2025-11-23T15:19:29Z)
• SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors [58.87134689752605]
  We introduce SimBench, the first large-scale, standardized benchmark for a robust, reproducible science of LLM simulation.<n>We show that even the best LLMs today have limited simulation ability (score: 40.80/100), performance scales log-linearly with model size.<n>We demonstrate that simulation ability correlates most strongly with deep, knowledge-intensive reasoning.
  arXiv  Detail & Related papers  (2025-10-20T13:14:38Z)
• Organ-Agents: Virtual Human Physiology Simulator via LLMs [66.40796430669158]
  Organ-Agents is a multi-agent framework that simulates human physiology via LLM-driven agents.<n>We curated data from 7,134 sepsis patients and 7,895 controls, generating high-resolution trajectories across 9 systems and 125 variables.<n>Organ-Agents achieved high simulation accuracy on 4,509 held-out patients, with per-system MSEs 0.16 and robustness across SOFA-based severity strata.
  arXiv  Detail & Related papers  (2025-08-20T01:58:45Z)
• AI Agents for Conversational Patient Triage: Preliminary Simulation-Based Evaluation with Real-World EHR Data [3.4206930658402115]
  We present a Patient Simulator that leverages real world patient encounters.<n>The simulator provides a realistic approach to patient presentation and multi-turn conversation with a symptom-checking agent.
  arXiv  Detail & Related papers  (2025-06-04T14:56:08Z)
• MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis [58.67342568632529]
  MoodAngels is the first specialized multi-agent framework for mood disorder diagnosis.<n>MoodSyn is an open-source dataset of 1,173 synthetic psychiatric cases.
  arXiv  Detail & Related papers  (2025-06-04T09:18:25Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.