MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

• URL: http://arxiv.org/abs/2607.08257v1
• Date: Thu, 09 Jul 2026 09:03:42 GMT
• Title: MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters
• Abstract summary: MentalHospital is a virtual evaluation environment for psychiatric clinical encounters.<n>MentalHospital instantiates the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning workflow.<n>Five domain-specific evaluators cover communication empathy, interviewing professionalism, clinical-note quality, diagnostic rigor, and treatment appropriateness.
• Score: 21.18455609800945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet existing benchmarks rarely simulate complete psychiatric clinical encounters. We introduce $\textbf{MentalHospital}$, a virtual evaluation environment for LLM-based psychiatric clinical encounters. MentalHospital instantiates the Subjective Interviewing, Objective Examination, Diagnostic Assessment, and Treatment Planning (S.O.A.P.) workflow, using skill-augmented standardized patients constructed from 1,193 de-identified psychiatric electronic health record (EHR) cases spanning all major ICD-11 categories and 76 disorders. Each encounter is assessed through a dual-track protocol that combines objective comparison against EHR-derived references with subjective assessment of clinical process quality. To scale specialist judgment, we develop $\textbf{MentalEval}$, five domain-specific evaluators covering communication empathy, interviewing professionalism, clinical-note quality, diagnostic rigor, and treatment appropriateness, trained with rubric-grounded SFT and expert-guided DPO. Survey responses from 22 clinicians support MentalHospital's clinical fidelity (3.88/5), while MentalEval achieves strong expert alignment with an average QWK of 0.944. Benchmarking shows that even the strongest LLM trails clinicians by 37.28 percentage points in objective psychiatric competence, with mental status assessment as a key bottleneck.

Related papers

• ResidencyRL: Reinforcement Learning in Simulated Clinical Environments [53.76751756107077]
  We present ResidencyRL, a reinforcement learning (RL) method for training clinical artificial intelligence (AI) agents through simulated multi-turn clinical encounters.<n>On heldout evaluations, the ResidencyRL agent improves diagnostic accuracy by 7.0% under adversarial conditions.<n>Blinded expert clinicians validated these gains, preferring the trained agent in 87.6% of side-by-side comparisons.
  arXiv  Detail & Related papers  (2026-08-07T17:04:41Z)
• The MADRS Pipeline: Supporting Depression Assessment in Clinical Trials [2.4031761937654763]
  Depression is a major mental disorder for which diagnosis relies on clinical assessments.<n>Existing solutions primarily focus on detecting the disorder from different text sources.<n>There is still limited support for clinical trials, where assessments are conducted through structured interviews.<n>Our pipeline converts audio interviews into transcripts, maps them into the ten MADRS symptom items, estimates their severity, and identifies problematic clinical ratings associated with them.
  arXiv  Detail & Related papers  (2026-07-30T13:27:56Z)
• LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis [14.82377002030236]
  Mental disorders are highly prevalent worldwide.<n>The shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment.<n>We present LingxiDiagBench, a large-scale multi-agent benchmark.
  arXiv  Detail & Related papers  (2026-02-10T03:46:05Z)
• MentalSeek-Dx: Towards Progressive Hypothetico-Deductive Reasoning for Real-world Psychiatric Diagnosis [27.839664095206857]
  MentalSeek-Dx Bench is the first benchmark dedicated to disorder-level psychiatric diagnosis within real-world clinical settings.<n>It comprises 712 de-identified electronic health records annotated by board-certified psychiatrists under ICD-11 guidelines.<n>MentalSeek-Dx achieves state-of-the-art (SOTA) performance with only 14B parameters, establishing a clinically grounded framework for reliable psychiatric diagnosis.
  arXiv  Detail & Related papers  (2026-02-03T10:03:35Z)
• Responsible Evaluation of AI for Mental Health [72.85175110624736]
  Current approaches to evaluating AI tools in mental health care are fragmented and poorly aligned with clinical practice, social context, and first-hand user experience.<n>This paper argues for a rethinking of responsible evaluation by introducing an interdisciplinary framework that integrates clinical soundness, social context, and equity.
  arXiv  Detail & Related papers  (2026-01-20T12:55:10Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Psychiatry-Bench: A Multi-Task Benchmark for LLMs in Psychiatry [1.2879523047871226]
  PsychiatryBench is a rigorously curated benchmark grounded exclusively in expert-validated psychiatric textbooks and casebooks.<n> PsychiatryBench comprises eleven distinct question-answering tasks ranging from diagnostic reasoning and treatment planning to longitudinal follow-up, management planning, clinical approach, sequential case analysis, and multiple-choice/extended matching formats totaling over 5,300 expert-annotated items.
  arXiv  Detail & Related papers  (2025-09-07T20:57:24Z)
• Unveiling the Landscape of Clinical Depression Assessment: From Behavioral Signatures to Psychiatric Reasoning [43.26860213892083]
  Depression is a widespread mental disorder that affects millions worldwide.<n>Most studies rely on limited or non-clinically validated data, and often prioritize complex model design over real-world effectiveness.<n>We introduce C-MIND, a clinical neuropsychiatric multimodal diagnosis dataset collected over two years from real hospital visits.<n>Each participant completes three structured psychiatric tasks and receives a final diagnosis from expert clinicians, with informative audio, video, transcript, and functional near-infrared spectroscopy (fNIRS) signals recorded.
  arXiv  Detail & Related papers  (2025-08-06T15:13:24Z)
• MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis [58.67342568632529]
  MoodAngels is the first specialized multi-agent framework for mood disorder diagnosis.<n>MoodSyn is an open-source dataset of 1,173 synthetic psychiatric cases.
  arXiv  Detail & Related papers  (2025-06-04T09:18:25Z)
• Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling [50.83055329849865]
  PsyLLM is a large language model designed to integrate diagnostic and therapeutic reasoning for mental health counseling.<n>It processes real-world mental health posts from Reddit and generates multi-turn dialogue structures.<n>Our experiments demonstrate that PsyLLM significantly outperforms state-of-the-art baseline models.
  arXiv  Detail & Related papers  (2025-05-21T16:24:49Z)
• MAGI: Multi-Agent Guided Interview for Psychiatric Assessment [50.6150986786028]
  We present MAGI, the first framework that transforms the gold-standard Mini International Neuropsychiatric Interview (MINI) into automatic computational navigation.<n>We show that MAGI advances LLM- assisted mental health assessment by combining clinical rigor, conversational adaptability, and explainable reasoning.
  arXiv  Detail & Related papers  (2025-04-25T11:08:27Z)
• LLM Questionnaire Completion for Automatic Psychiatric Assessment [49.1574468325115]
  We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C)
  arXiv  Detail & Related papers  (2024-06-09T09:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.