Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset

• URL: http://arxiv.org/abs/2403.00438v1
• Date: Fri, 1 Mar 2024 10:47:02 GMT
• Title: Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset
• Authors: Kirill Milintsevich (1 and 2), Kairit Sirts (2), Ga\"el Dias (1) ((1) University of Caen Normandy, (2) University of Tartu)
• Abstract summary: This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. Our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.

Related papers

• Depression Detection on Social Media with Large Language Models [23.075317886505193]
  Depression detection aims to determine whether an individual suffers from depression by analyzing their history of posts on social media. We propose a novel depression detection system called DORIS, combining medical knowledge and the recent advances in large language models.
  arXiv  Detail & Related papers  (2024-03-16T01:01:16Z)
• DepreSym: A Depression Symptom Annotated Corpus and the Role of LLMs as Assessors of Psychological Markers [3.5511184956329727]
  We present the DepreSym dataset, consisting of 21580 sentences annotated according to their relevance to the Beck Depression Inventory-II symptoms. This dataset serves as a valuable resource for advancing the development of models that incorporate depressive markers such as clinical symptoms.
  arXiv  Detail & Related papers  (2023-08-21T14:44:31Z)
• What Symptoms and How Long? An Interpretable AI Approach for Depression Detection in Social Media [0.5156484100374058]
  Depression is the most prevalent and serious mental illness, which induces grave financial and societal ramifications. This study contributes to IS literature with a novel interpretable deep learning model for depression detection in social media.
  arXiv  Detail & Related papers  (2023-05-18T20:15:04Z)
• Depression detection in social media posts using affective and social norm features [84.12658971655253]
  We propose a deep architecture for depression detection from social media posts. We incorporate profanity and morality features of posts and words in our architecture using a late fusion scheme. The inclusion of the proposed features yields state-of-the-art results in both settings.
  arXiv  Detail & Related papers  (2023-03-24T21:26:27Z)
• Semantic Similarity Models for Depression Severity Estimation [53.72188878602294]
  This paper presents an efficient semantic pipeline to study depression severity in individuals based on their social media writings. We use test user sentences for producing semantic rankings over an index of representative training sentences corresponding to depressive symptoms and severity levels. We evaluate our methods on two Reddit-based benchmarks, achieving 30% improvement over state of the art in terms of measuring depression severity.
  arXiv  Detail & Related papers  (2022-11-14T18:47:26Z)
• DEPTWEET: A Typology for Social Media Texts to Detect Depression Severities [0.46796109436086664]
  We leverage the clinical articulation of depression to build a typology for social media texts for detecting the severity of depression. It emulates the standard clinical assessment procedure Diagnostic and Statistical Manual of Mental Disorders (DSM-5) and Patient Health Questionnaire (PHQ-9) We present a new dataset of 40191 tweets labeled by expert annotators. Each tweet is labeled as 'non-depressed' or 'depressed'
  arXiv  Detail & Related papers  (2022-10-10T08:23:57Z)
• Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future [63.99570204416711]
  We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets. We define a uniform evaluation setup including a new formalization of the annotation error detection task. We release our datasets and implementations in an easy-to-use and open source software package.
  arXiv  Detail & Related papers  (2022-06-05T22:31:45Z)
• Journaling Data for Daily PHQ-2 Depression Prediction and Forecasting [47.93070579578704]
  We explore the potential of using actively-collected data to predict and forecast daily PHQ-2 scores on a newly-collected longitudinal dataset. We obtain a best MAE of 1.417 for daily prediction of PHQ-2 scores, which specifically in the used dataset have a range of 0 to 12. This illustrates the additive value that can be obtained by incorporating actively-collected data in a depression monitoring application.
  arXiv  Detail & Related papers  (2022-05-06T17:47:05Z)
• Data set creation and empirical analysis for detecting signs of depression from social media postings [0.0]
  Depression is a common mental illness that has to be detected and treated at an early stage to avoid serious consequences. We developed a gold standard data set that detects the levels of depression as not depressed', moderately depressed' and severely depressed' from the social media postings.
  arXiv  Detail & Related papers  (2022-02-07T10:24:33Z)
• Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data [74.60507696087966]
  Mental health conditions remain underdiagnosed even in countries with common access to advanced medical care. One promising data source to help monitor human behavior is daily smartphone usage. We study behavioral markers of daily mood using a recent dataset of mobile behaviors from adolescent populations at high risk of suicidal behaviors.
  arXiv  Detail & Related papers  (2021-06-24T17:46:03Z)
• Deep Multi-task Learning for Depression Detection and Prediction in Longitudinal Data [50.02223091927777]
  Depression is among the most prevalent mental disorders, affecting millions of people of all ages globally. Machine learning techniques have shown effective in enabling automated detection and prediction of depression for early intervention and treatment. We introduce a novel deep multi-task recurrent neural network to tackle this challenge, in which depression classification is jointly optimized with two auxiliary tasks.
  arXiv  Detail & Related papers  (2020-12-05T05:14:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.