DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews

• URL: http://arxiv.org/abs/2404.14463v1
• Date: Mon, 22 Apr 2024 09:07:50 GMT
• Title: DAIC-WOZ: On the Validity of Using the Therapist's prompts in Automatic Depression Detection from Clinical Interviews
• Authors: Sergio Burdisso, Ernesto Reyes-Ramírez, Esaú Villatoro-Tello, Fernando Sánchez-Vega, Pastor López-Monroy, Petr Motlicek,
• Abstract summary: Recent studies have reported enhanced performance when incorporating interviewer's prompts into the model. We discover that models using interviewer's prompts learn to focus on a specific region of the interviews, where questions about past experiences with mental health issues are asked. We achieve a 0.90 F1 score by intentionally exploiting it, the highest result reported to date on this dataset using only textual information.
• Score: 39.08557916089242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic depression detection from conversational data has gained significant interest in recent years. The DAIC-WOZ dataset, interviews conducted by a human-controlled virtual agent, has been widely used for this task. Recent studies have reported enhanced performance when incorporating interviewer's prompts into the model. In this work, we hypothesize that this improvement might be mainly due to a bias present in these prompts, rather than the proposed architectures and methods. Through ablation experiments and qualitative analysis, we discover that models using interviewer's prompts learn to focus on a specific region of the interviews, where questions about past experiences with mental health issues are asked, and use them as discriminative shortcuts to detect depressed participants. In contrast, models using participant responses gather evidence from across the entire interview. Finally, to highlight the magnitude of this bias, we achieve a 0.90 F1 score by intentionally exploiting it, the highest result reported to date on this dataset using only textual information. Our findings underline the need for caution when incorporating interviewers' prompts into models, as they may inadvertently learn to exploit targeted prompts, rather than learning to characterize the language and behavior that are genuinely indicative of the patient's mental health condition.

Related papers

• Cognitive Insights Across Languages: Enhancing Multimodal Interview Analysis [0.6062751776009752]
  We propose a multimodal model capable of predicting Mild Cognitive Impairment and cognitive scores. The proposed model demonstrates the ability to transcribe and differentiate between languages used in the interviews. Our approach involves in-depth research to implement various features obtained from the proposed modalities.
  arXiv  Detail & Related papers  (2024-06-11T17:59:31Z)
• LLM Questionnaire Completion for Automatic Psychiatric Assessment [49.1574468325115]
  We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C)
  arXiv  Detail & Related papers  (2024-06-09T09:03:11Z)
• Modeling User Preferences via Brain-Computer Interfacing [54.3727087164445]
  We use Brain-Computer Interfacing technology to infer users' preferences, their attentional correlates towards visual content, and their associations with affective experience. We link these to relevant applications, such as information retrieval, personalized steering of generative models, and crowdsourcing population estimates of affective experiences.
  arXiv  Detail & Related papers  (2024-05-15T20:41:46Z)
• Depression Detection on Social Media with Large Language Models [23.075317886505193]
  Depression detection aims to determine whether an individual suffers from depression by analyzing their history of posts on social media. We propose a novel depression detection system called DORIS, combining medical knowledge and the recent advances in large language models.
  arXiv  Detail & Related papers  (2024-03-16T01:01:16Z)
• From Classification to Clinical Insights: Towards Analyzing and Reasoning About Mobile and Behavioral Health Data With Large Language Models [22.140953523632724]
  We take a novel approach that leverages large language models to synthesize clinically useful insights from multi-sensor data. We develop chain of thought prompting methods that use LLMs to generate reasoning about how trends in data relate to conditions like depression and anxiety. We find models like GPT-4 correctly reference numerical data 75% of the time, and clinician participants express strong interest in using this approach to interpret self-tracking data.
  arXiv  Detail & Related papers  (2023-11-21T23:53:27Z)
• Evaluating Subjective Cognitive Appraisals of Emotions from Large Language Models [47.890846082224066]
  This work fills the gap by presenting CovidET-Appraisals, the most comprehensive dataset to-date that assesses 24 appraisal dimensions. CovidET-Appraisals presents an ideal testbed to evaluate the ability of large language models to automatically assess and explain cognitive appraisals.
  arXiv  Detail & Related papers  (2023-10-22T19:12:17Z)
• Explainable Depression Symptom Detection in Social Media [2.433983268807517]
  We propose using transformer-based architectures to detect and explain the appearance of depressive symptom markers in the users' writings. Our natural language explanations enable clinicians to interpret the models' decisions based on validated symptoms.
  arXiv  Detail & Related papers  (2023-10-20T17:05:27Z)
• Towards Mitigating Hallucination in Large Language Models via Self-Reflection [63.2543947174318]
  Large language models (LLMs) have shown promise for generative and knowledge-intensive tasks including question-answering (QA) tasks. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets.
  arXiv  Detail & Related papers  (2023-10-10T03:05:44Z)
• Identifying depression-related topics in smartphone-collected free-response speech recordings using an automatic speech recognition system and a deep learning topic model [7.825530847570242]
  We identified 29 topics in 3919 smartphone-collected speech recordings from 265 participants. Six topics with a median PHQ-8 greater than or equal to 10 were regarded as risk topics for depression. The correlation between topic shifts and changes in depression severity over time was also investigated.
  arXiv  Detail & Related papers  (2023-08-22T20:30:59Z)
• Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals [53.484562601127195]
  We point out the inability to infer behavioral conclusions from probing results. We offer an alternative method that focuses on how the information is being used, rather than on what information is encoded.
  arXiv  Detail & Related papers  (2020-06-01T15:00:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.