MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models

• URL: http://arxiv.org/abs/2502.15418v1
• Date: Fri, 21 Feb 2025 12:37:58 GMT
• Title: MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models
• Abstract summary: Mental health remains a challenging problem all over the world, with issues like depression, anxiety becoming increasingly common.<n>There is a lack of standard benchmarking datasets for question answering (QA) in mental health.<n>Our work presents a novel multiple choice dataset, MHQA (Mental Health Question Answering), for benchmarking Language models (LMs)
• Score: 14.977595131058132
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mental health remains a challenging problem all over the world, with issues like depression, anxiety becoming increasingly common. Large Language Models (LLMs) have seen a vast application in healthcare, specifically in answering medical questions. However, there is a lack of standard benchmarking datasets for question answering (QA) in mental health. Our work presents a novel multiple choice dataset, MHQA (Mental Health Question Answering), for benchmarking Language models (LMs). Previous mental health datasets have focused primarily on text classification into specific labels or disorders. MHQA, on the other hand, presents question-answering for mental health focused on four key domains: anxiety, depression, trauma, and obsessive/compulsive issues, with diverse question types, namely, factoid, diagnostic, prognostic, and preventive. We use PubMed abstracts as the primary source for QA. We develop a rigorous pipeline for LLM-based identification of information from abstracts based on various selection criteria and converting it into QA pairs. Further, valid QA pairs are extracted based on post-hoc validation criteria. Overall, our MHQA dataset consists of 2,475 expert-verified gold standard instances called MHQA-gold and ~56.1k pairs pseudo labeled using external medical references. We report F1 scores on different LLMs along with few-shot and supervised fine-tuning experiments, further discussing the insights for the scores.

Related papers

• CARMA: Comprehensive Automatically-annotated Reddit Mental Health Dataset for Arabic [1.3320917259299652]
  We present CARMA, the first automatically annotated large-scale dataset of Arabic Reddit posts.<n>The dataset encompasses six mental health conditions, such as Anxiety, Autism, and Depression, and a control group.<n>We conduct qualitative and quantitative analyses of lexical and semantic differences between users, providing insights into the linguistic markers of specific mental health conditions.
  arXiv  Detail & Related papers  (2025-11-05T01:17:43Z)
• MedQARo: A Large-Scale Benchmark for Medical Question Answering in Romanian [50.767415194856135]
  We introduce MedQARo, the first large-scale medical QA benchmark in Romanian.<n>We construct a high-quality and large-scale dataset comprising 102,646 QA pairs related to cancer patients.
  arXiv  Detail & Related papers  (2025-08-22T13:48:37Z)
• CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering [1.0262304700896199]
  We present CounselBench, a large-scale benchmark developed with 100 mental health professionals to evaluate and stress-test large language models (LLMs)<n>The first component, CounselBench-EVAL, contains 2,000 expert evaluations of answers from GPT-4, LLaMA 3, Gemini, and human therapists on patient questions from the public forum CounselChat.<n>Expert evaluations show that while LLMs achieve high scores on several dimensions, they also exhibit recurring issues, including unconstructive feedback, overgeneralization, and limited personalization or relevance.
  arXiv  Detail & Related papers  (2025-06-10T08:53:06Z)
• Enhancing Depression Detection via Question-wise Modality Fusion [47.45016610508853]
  Depression is a highly prevalent and disabling condition that incurs substantial personal and societal costs. We propose a novel Question-wise Modality Fusion framework trained with a novel Imbalanced Ordinal Log-Loss function.
  arXiv  Detail & Related papers  (2025-03-26T12:34:34Z)
• Moving Beyond Medical Exam Questions: A Clinician-Annotated Dataset of Real-World Tasks and Ambiguity in Mental Healthcare [0.0545520830707066]
  We present an expert-created and annotated dataset spanning five critical domains of decision-making in mental healthcare. This dataset is designed to capture the nuanced clinical reasoning and daily ambiguities mental health practitioners encounter.
  arXiv  Detail & Related papers  (2025-02-22T03:10:16Z)
• MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding [20.83722922095852]
  MedXpertQA includes 4,460 questions spanning 17 specialties and 11 body systems. MM introduces expert-level exam questions with diverse images and rich clinical information.
  arXiv  Detail & Related papers  (2025-01-30T14:07:56Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders [59.515827458631975]
  Mental health disorders are one of the most serious diseases in the world.<n>Privacy concerns limit the accessibility of personalized treatment data.<n>MentalArena is a self-play framework to train language models.
  arXiv  Detail & Related papers  (2024-10-09T13:06:40Z)
• LLM Questionnaire Completion for Automatic Psychiatric Assessment [49.1574468325115]
  We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C)
  arXiv  Detail & Related papers  (2024-06-09T09:03:11Z)
• MentalQA: An Annotated Arabic Corpus for Questions and Answers of Mental Healthcare [0.1638581561083717]
  MentalQA is a novel Arabic dataset featuring conversational-style question-and-answer (QA) interactions. Data was collected from a question-answering medical platform. MentalQA offers a valuable foundation for developing Arabic text mining tools capable of supporting mental health professionals and individuals seeking information.
  arXiv  Detail & Related papers  (2024-05-21T09:16:38Z)
• Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions [19.436999992810797]
  We construct two new datasets: JAMA Clinical Challenge and Medbullets.<n> JAMA Clinical Challenge consists of questions based on challenging clinical cases, while Medbullets comprises simulated clinical questions.<n>We evaluate seven LLMs on the two datasets using various prompts.
  arXiv  Detail & Related papers  (2024-02-28T05:44:41Z)
• XAIQA: Explainer-Based Data Augmentation for Extractive Question Answering [1.1867812760085572]
  We introduce a novel approach, XAIQA, for generating synthetic QA pairs at scale from data naturally available in electronic health records. Our method uses the idea of a classification model explainer to generate questions and answers about medical concepts corresponding to medical codes.
  arXiv  Detail & Related papers  (2023-12-06T15:59:06Z)
• Med-Flamingo: a Multimodal Medical Few-shot Learner [58.85676013818811]
  We propose Med-Flamingo, a multimodal few-shot learner adapted to the medical domain. Based on OpenFlamingo-9B, we continue pre-training on paired and interleaved medical image-text data from publications and textbooks. We conduct the first human evaluation for generative medical VQA where physicians review the problems and blinded generations in an interactive app.
  arXiv  Detail & Related papers  (2023-07-27T20:36:02Z)
• Generating multiple-choice questions for medical question answering with distractors and cue-masking [17.837685583005566]
  Medical multiple-choice question answering (MCQA) is particularly difficult. Standard language modeling pretraining alone is not sufficient to achieve the best results.
  arXiv  Detail & Related papers  (2023-03-13T12:45:01Z)
• Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering [89.76059961309453]
  HeadQA dataset contains multiple-choice questions authorized for the public healthcare specialization exam. These questions are the most challenging for current QA systems. We present a Multi-step reasoning with Knowledge extraction framework (MurKe) We are striving to make full use of off-the-shelf pre-trained models.
  arXiv  Detail & Related papers  (2020-08-06T02:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.