Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses

• URL: http://arxiv.org/abs/2604.00014v1
• Date: Tue, 10 Mar 2026 17:40:24 GMT
• Title: Disentangling Prompt Element Level Risk Factors for Hallucinations and Omissions in Mental Health LLM Responses
• Abstract summary: Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking.<n>We introduce UTCO, a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing.<n>Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts.
• Score: 11.099638001699276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mental health concerns are often expressed outside clinical settings, including in high-distress help seeking, where safety-critical guidance may be needed. Consumer health informatics systems increasingly incorporate large language models (LLMs) for mental health question answering, yet many evaluations underrepresent narrative, high-distress inquiries. We introduce UTCO (User, Topic, Context, Tone), a prompt construction framework that represents an inquiry as four controllable elements for systematic stress testing. Using 2,075 UTCO-generated prompts, we evaluated Llama 3.3 and annotated hallucinations (fabricated or incorrect clinical content) and omissions (missing clinically necessary or safety-critical guidance). Hallucinations occurred in 6.5% of responses and omissions in 13.2%, with omissions concentrated in crisis and suicidal ideation prompts. Across regression, element-specific matching, and similarity-matched comparisons, failures were most consistently associated with context and tone, while user-background indicators showed no systematic differences after balancing. These findings support evaluating omissions as a primary safety outcome and moving beyond static benchmark question sets.

Related papers

• Towards Understanding and Measuring COGNITIVE ATROPHY in LLM Behaviour [10.419215828520414]
  Existing benchmarks measure knowledge, safety, or static response quality.<n>We formalize this missing dimension as COGNITIVE ATROPHY, a process-level behavioural measure in AI-mediated mental-health support.
  arXiv  Detail & Related papers  (2026-06-16T16:26:12Z)
• Beyond Accuracy: Risk-Sensitive Evaluation of Hallucinated Medical Advice [0.1609950046042424]
  We propose a risk-sensitive evaluation framework that quantifies hallucinations through the presence of risk-bearing language.<n>We apply this framework to three instruction-tuned language models using controlled patient-facing prompts designed as safety stress tests.
  arXiv  Detail & Related papers  (2026-02-07T02:25:44Z)
• Assessing the Quality of Mental Health Support in LLM Responses through Multi-Attribute Human Evaluation [14.243791046586347]
  The escalating global mental health crisis, marked by persistent treatment gaps, availability, and a shortage of qualified therapists, positions Large Language Models (LLMs) as a promising avenue for scalable support.<n>This paper introduces a human-grounded evaluation methodology designed to assess LLM generated responses in therapeutic dialogue.
  arXiv  Detail & Related papers  (2026-01-26T16:04:19Z)
• Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models [48.95516224614331]
  We introduce MedGaze-Bench, the first benchmark leveraging clinician gaze as a Cognitive Cursor to assess intent understanding across surgery, emergency simulation, and diagnostic interpretation.<n>Our benchmark addresses three fundamental challenges: visual homogeneity of anatomical structures, strict temporal-causal dependencies in clinical, and implicit adherence to safety protocols.
  arXiv  Detail & Related papers  (2026-01-11T02:20:40Z)
• Between Help and Harm: An Evaluation of Mental Health Crisis Handling by LLMs [6.0460961868478975]
  We introduce a unified taxonomy of six clinically-informed mental health crisis categories.<n>We benchmark three state-of-the-art LLMs for their ability to classify crisis types and generate safe, appropriate responses.<n>We identify systemic weaknesses in handling indirect or ambiguous risk signals, a reliance on formulaic and inauthentic default replies, and frequent misalignment with user context.
  arXiv  Detail & Related papers  (2025-09-29T14:42:23Z)
• Evaluating the Clinical Safety of LLMs in Response to High-Risk Mental Health Disclosures [29.742441212366312]
  This study evaluates the responses of six popular large language models (LLMs) to user prompts simulating crisis-level mental health disclosures.<n>Claude outperformed all others in global assessment, while Grok 3, ChatGPT, and LLAMA underperformed across multiple domains.
  arXiv  Detail & Related papers  (2025-09-01T16:01:08Z)
• SHALE: A Scalable Benchmark for Fine-grained Hallucination Evaluation in LVLMs [52.03164192840023]
  Large Vision-Language Models (LVLMs) still suffer from hallucinations, i.e., generating content inconsistent with input or established world knowledge.<n>We propose an automated data construction pipeline that produces scalable, controllable, and diverse evaluation data.<n>We construct SHALE, a benchmark designed to assess both faithfulness and factuality hallucinations.
  arXiv  Detail & Related papers  (2025-08-13T07:58:01Z)
• CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering [1.0262304700896199]
  We present CounselBench, a large-scale benchmark developed with 100 mental health professionals to evaluate and stress-test large language models (LLMs)<n>The first component, CounselBench-EVAL, contains 2,000 expert evaluations of answers from GPT-4, LLaMA 3, Gemini, and human therapists on patient questions from the public forum CounselChat.<n>Expert evaluations show that while LLMs achieve high scores on several dimensions, they also exhibit recurring issues, including unconstructive feedback, overgeneralization, and limited personalization or relevance.
  arXiv  Detail & Related papers  (2025-06-10T08:53:06Z)
• MoodAngels: A Retrieval-augmented Multi-agent Framework for Psychiatry Diagnosis [58.67342568632529]
  MoodAngels is the first specialized multi-agent framework for mood disorder diagnosis.<n>MoodSyn is an open-source dataset of 1,173 synthetic psychiatric cases.
  arXiv  Detail & Related papers  (2025-06-04T09:18:25Z)
• LLM Questionnaire Completion for Automatic Psychiatric Assessment [49.1574468325115]
  We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C)
  arXiv  Detail & Related papers  (2024-06-09T09:03:11Z)
• Towards Mitigating Hallucination in Large Language Models via Self-Reflection [63.2543947174318]
  Large language models (LLMs) have shown promise for generative and knowledge-intensive tasks including question-answering (QA) tasks. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets.
  arXiv  Detail & Related papers  (2023-10-10T03:05:44Z)
• Recall, Robustness, and Lexicographic Evaluation [49.13362412522523]
  The application of recall without a formal evaluative motivation has led to criticism of recall as a vague or inappropriate measure.<n>Our analysis is composed of three tenets: recall, robustness, and lexicographic evaluation.
  arXiv  Detail & Related papers  (2023-02-22T13:39:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.