MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

• URL: http://arxiv.org/abs/2110.15621v1
• Date: Fri, 29 Oct 2021 08:36:47 GMT
• Title: MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare
• Authors: Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, Erik Cambria
• Abstract summary: Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models. This paper trains and releases two pretrained language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community.
• Score: 29.14340469459733
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.

Related papers

• MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders [59.515827458631975]
  Mental health disorders are one of the most serious diseases in the world. Privacy concerns limit the accessibility of personalized treatment data. MentalArena is a self-play framework to train language models.
  arXiv  Detail & Related papers  (2024-10-09T13:06:40Z)
• Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales [4.805861461250903]
  We show how standard psychological questionnaires can be reformulated into natural language inference prompts. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs.
  arXiv  Detail & Related papers  (2024-09-29T11:00:41Z)
• LLM Questionnaire Completion for Automatic Psychiatric Assessment [49.1574468325115]
  We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C)
  arXiv  Detail & Related papers  (2024-06-09T09:03:11Z)
• Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis [19.32304448831033]
  We have collected a huge dataset from Chinese social media platforms. We enriched it with publicly available datasets to create a database encompassing 3.36 million text entries. To enhance the model's applicability to psychological text analysis, we integrated psychological lexicons into the pre-training masking mechanism.
  arXiv  Detail & Related papers  (2024-02-14T13:08:25Z)
• An Assessment on Comprehending Mental Health through Large Language Models [2.7044181783627086]
  More than 20% of adults may encounter at least one mental disorder in their lifetime. This study presents an initial evaluation of large language models in addressing this gap. Our results on the DAIC-WOZ dataset show that transformer-based models, like BERT or XLNet, outperform the large language models.
  arXiv  Detail & Related papers  (2024-01-09T14:50:04Z)
• Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health [23.458852189587073]
  This paper conducts domain-specific continued pretraining to capture the long context for mental health. Specifically, we train and release MentalXLNet and MentalLongformer based on XLNet and Longformer. We evaluate the mental health classification performance and the long-range ability of these two domain-specific pretrained models.
  arXiv  Detail & Related papers  (2023-04-20T16:43:56Z)
• Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps [75.84770193489639]
  We examine the impact of test loss, training corpus and model architecture on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words. We suggest good practices for future studies aiming at explaining the human language system using neural language models.
  arXiv  Detail & Related papers  (2022-07-07T15:37:17Z)
• Mental Illness Classification on Social Media Texts using Deep Learning and Transfer Learning [55.653944436488786]
  According to the World health organization (WHO), approximately 450 million people are affected. Mental illnesses, such as depression, anxiety, bipolar disorder, ADHD, and PTSD. This study analyzes unstructured user data on Reddit platform and classifies five common mental illnesses: depression, anxiety, bipolar disorder, ADHD, and PTSD.
  arXiv  Detail & Related papers  (2022-07-03T11:33:52Z)
• Improving Cross-Lingual Reading Comprehension with Self-Training [62.73937175625953]
  Current state-of-the-art models even surpass human performance on several benchmarks. Previous works have revealed the abilities of pre-trained multilingual models for zero-shot cross-lingual reading comprehension. This paper further utilized unlabeled data to improve the performance.
  arXiv  Detail & Related papers  (2021-05-08T08:04:30Z)
• Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation [75.3147962600095]
  We propose an automated framework for body language based emotion recognition starting from regular RGB videos. In collaboration with psychologists, we extend the framework for psychiatric symptom prediction. Because a specific application domain of the proposed framework may only supply a limited amount of data, the framework is designed to work on a small training set.
  arXiv  Detail & Related papers  (2020-10-30T18:45:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.