Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis

• URL: http://arxiv.org/abs/2604.02359v1
• Date: Fri, 20 Mar 2026 04:31:03 GMT
• Title: Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
• Abstract summary: General-purpose Large Language Models (LLMs) are becoming widely adopted by people for mental health support.<n>Yet emerging evidence suggests there are significant risks associated with high-frequency use, particularly for individuals suffering from psychosis.<n>This research focuses on psychosis as a critical condition for LLM safety evaluation.
• Score: 1.53934570513443
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: General-purpose Large Language Models (LLMs) are becoming widely adopted by people for mental health support. Yet emerging evidence suggests there are significant risks associated with high-frequency use, particularly for individuals suffering from psychosis, as LLMs may reinforce delusions and hallucinations. Existing evaluations of LLMs in mental health contexts are limited by a lack of clinical validation and scalability of assessment. To address these issues, this research focuses on psychosis as a critical condition for LLM safety evaluation by (1) developing and validating seven clinician-informed safety criteria, (2) constructing a human-consensus dataset, and (3) testing automated assessment using an LLM as an evaluator (LLM-as-a-Judge) or taking the majority vote of several LLM judges (LLM-as-a-Jury). Results indicate that LLM-as-a-Judge aligns closely with the human consensus (Cohen's $κ_{\text{human} \times \text{gemini}} = 0.75$, $κ_{\text{human} \times \text{qwen}} = 0.68$, $κ_{\text{human} \times \text{kimi}} = 0.56$) and that the best judge slightly outperforms LLM-as-a-Jury (Cohen's $κ_{\text{human} \times \text{jury}} = 0.74$). Overall, these findings have promising implications for clinically grounded, scalable methods in LLM safety evaluations for mental health contexts.

Related papers

• Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses [13.970589950093988]
  State-of-the-art LLM-as-a-judge methods often fail in high-risk healthcare contexts.<n>We show that leading LLM judges achieve only 52% accuracy on mental health counseling data.<n>We propose a framework that integrates human expertise with LLMs to extract interpretable, domain-informed features.
  arXiv  Detail & Related papers  (2026-03-17T21:13:19Z)
• On Evaluating LLM Alignment by Evaluating LLMs as Judges [68.15541137648721]
  evaluating large language models' (LLMs) alignment requires them to be helpful, honest, safe, and to precisely follow human instructions.<n>We examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences.<n>We propose a benchmark that assesses alignment without directly evaluating model outputs.
  arXiv  Detail & Related papers  (2025-11-25T18:33:24Z)
• When Can We Trust LLMs in Mental Health? Large-Scale Benchmarks for Reliable LLM Evaluation [14.24379104658635]
  MentalBench-100k consolidates 10,000 one-turn conversations from three real scenarios datasets, each paired with nine LLM-generated responses.<n>MentalBench-70kreframes evaluation by comparing four high-performing LLM judges with human experts across 70,000 ratings on seven attributes.<n>Our analysis reveals systematic inflation by LLM judges, strong reliability for cognitive attributes such as guidance and informativeness, reduced precision for empathy, and some unreliability in safety and relevance.
  arXiv  Detail & Related papers  (2025-10-21T19:21:21Z)
• The Psychogenic Machine: Simulating AI Psychosis, Delusion Reinforcement and Harm Enablement in Large Language Models [3.775403046470809]
  "AI psychosis" is emerging where user-LLM interactions may exacerbate or induce psychosis or adverse psychological symptoms.<n>Psychosis-bench is a novel benchmark designed to evaluate the psychogenicity of LLMs.<n>We evaluated eight prominent LLMs for Delusion Confirmation, Harm Enablement, and Safety Intervention across explicit and implicit conversational contexts.
  arXiv  Detail & Related papers  (2025-09-13T20:10:28Z)
• LLM-as-a-Fuzzy-Judge: Fine-Tuning Large Language Models as a Clinical Evaluation Judge with Fuzzy Logic [3.1090878361419416]
  This paper proposes LLM-as-a-Fuzzy-Judge to align automated evaluation of medical students' clinical skills with subjective physicians' preferences.<n>The approach is fine-tuned to evaluate medical students' utterances within student-AI patient conversation scripts based on human annotations from four fuzzy sets.<n>The results show that the LLM-as-a-Fuzzy-Judge achieves over 80% accuracy, with major criteria items over 90%.
  arXiv  Detail & Related papers  (2025-06-12T18:31:49Z)
• Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation [7.867257950096845]
  Plain language summaries (PLSs) are essential for facilitating effective communication between clinicians and patients.<n>Large language models (LLMs) have recently shown promise in automating PLS generation, but their effectiveness in supporting health information comprehension remains unclear.<n>We conducted a large-scale crowdsourced evaluation of LLM-generated PLSs using Amazon Mechanical Turk with 150 participants.<n>Our findings indicate that while LLMs can generate PLSs that appear indistinguishable from human-written ones in subjective evaluations, human-written PLSs lead to significantly better comprehension.
  arXiv  Detail & Related papers  (2025-05-15T15:31:17Z)
• Med-CoDE: Medical Critique based Disagreement Evaluation Framework [72.42301910238861]
  The reliability and accuracy of large language models (LLMs) in medical contexts remain critical concerns.<n>Current evaluation methods often lack robustness and fail to provide a comprehensive assessment of LLM performance.<n>We propose Med-CoDE, a specifically designed evaluation framework for medical LLMs to address these challenges.
  arXiv  Detail & Related papers  (2025-04-21T16:51:11Z)
• LLM Internal States Reveal Hallucination Risk Faced With a Query [62.29558761326031]
  Humans have a self-awareness process that allows us to recognize what we don't know when faced with queries. This paper investigates whether Large Language Models can estimate their own hallucination risk before response generation. By a probing estimator, we leverage LLM self-assessment, achieving an average hallucination estimation accuracy of 84.32% at run time.
  arXiv  Detail & Related papers  (2024-07-03T17:08:52Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Can AI Relate: Testing Large Language Model Response for Mental Health Support [23.97212082563385]
  Large language models (LLMs) are already being piloted for clinical use in hospital systems like NYU Langone, Dana-Farber and the NHS. We develop an evaluation framework for determining whether LLM response is a viable and ethical path forward for the automation of mental health treatment.
  arXiv  Detail & Related papers  (2024-05-20T13:42:27Z)
• Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment [8.948475969696075]
  Large Language Models (LLMs) are powerful zero-shot assessors used in real-world situations such as assessing written exams and benchmarking systems. We show that short universal adversarial phrases can be deceived to judge LLMs to predict inflated scores. It is found that judge-LLMs are significantly more susceptible to these adversarial attacks when used for absolute scoring.
  arXiv  Detail & Related papers  (2024-02-21T18:55:20Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.