Evaluating Transformer Models for Suicide Risk Detection on Social Media

• URL: http://arxiv.org/abs/2410.08375v1
• Date: Thu, 10 Oct 2024 21:15:25 GMT
• Title: Evaluating Transformer Models for Suicide Risk Detection on Social Media
• Authors: Jakub Pokrywka, Jeremi I. Kaczmarek, Edward J. Gorzelańczyk,
• Abstract summary: This paper presents a study on leveraging state-of-the-art natural language processing solutions for identifying suicide risk in social media posts. We propose that these models, combined with minimal tuning, may have the potential to be effective solutions for automated suicide risk detection on social media.
• Score: 0.5461938536945723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The detection of suicide risk in social media is a critical task with potential life-saving implications. This paper presents a study on leveraging state-of-the-art natural language processing solutions for identifying suicide risk in social media posts as a submission for the "IEEE BigData 2024 Cup: Detection of Suicide Risk on Social Media" conducted by the kubapok team. We experimented with the following configurations of transformer-based models: fine-tuned DeBERTa, GPT-4o with CoT and few-shot prompting, and fine-tuned GPT-4o. The task setup was to classify social media posts into four categories: indicator, ideation, behavior, and attempt. Our findings demonstrate that the fine-tuned GPT-4o model outperforms two other configurations, achieving high accuracy in identifying suicide risk. Notably, our model achieved second place in the competition. By demonstrating that straightforward, general-purpose models can achieve state-of-the-art results, we propose that these models, combined with minimal tuning, may have the potential to be effective solutions for automated suicide risk detection on social media.

Related papers

• Su-RoBERTa: A Semi-supervised Approach to Predicting Suicide Risk through Social Media using Base Language Models [24.260983864615557]
  This paper is a study done on suicidal risk assessments using Reddit data. We have demonstrated that using smaller language models, i.e., less than 500M parameters, can also be effective. We propose Su-RoBERTa, a fine-tuned RoBERTa on suicide risk prediction task.
  arXiv  Detail & Related papers  (2024-12-02T10:31:12Z)
• Leveraging Large Language Models for Suicide Detection on Social Media with Limited Labels [3.1399304968349186]
  This paper explores the use of Large Language Models (LLMs) to automatically detect suicidal content in text-based social media posts. We develop an ensemble approach involving prompting with Qwen2-72B-Instruct, and using fine-tuned models such as Llama3-8B, Llama3.1-8B, and Gemma2-9B. Experimental results show that the ensemble model significantly improves the detection accuracy, by 5% points compared with the individual models.
  arXiv  Detail & Related papers  (2024-10-06T14:45:01Z)
• SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis [22.709733830774788]
  This study presents a Chinese social media dataset designed for fine-grained suicide risk classification. Seven pre-trained models were evaluated in two tasks: high and low suicide risk, and fine-grained suicide risk classification on a level of 0 to 10. Deep learning models show good performance in distinguishing between high and low suicide risk, with the best model achieving an F1 score of 88.39%.
  arXiv  Detail & Related papers  (2024-04-19T06:58:51Z)
• Non-Invasive Suicide Risk Prediction Through Speech Analysis [74.8396086718266]
  We present a non-invasive, speech-based approach for automatic suicide risk assessment. We extract three sets of features, including wav2vec, interpretable speech and acoustic features, and deep learning-based spectral representations. Our most effective speech model achieves a balanced accuracy of $66.2,%$.
  arXiv  Detail & Related papers  (2024-04-18T12:33:57Z)
• Navigating the OverKill in Large Language Models [84.62340510027042]
  We investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to an over-attention of harmful words like 'kill' and prompts emphasizing safety will exacerbate overkill. We introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon.
  arXiv  Detail & Related papers  (2024-01-31T07:26:47Z)
• Model Stealing Attack against Graph Classification with Authenticity, Uncertainty and Diversity [80.16488817177182]
  GNNs are vulnerable to the model stealing attack, a nefarious endeavor geared towards duplicating the target model via query permissions. We introduce three model stealing attacks to adapt to different actual scenarios.
  arXiv  Detail & Related papers  (2023-12-18T05:42:31Z)
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models [92.6951708781736]
  This work proposes a comprehensive trustworthiness evaluation for large language models with a focus on GPT-4 and GPT-3.5. We find that GPT models can be easily misled to generate toxic and biased outputs and leak private information. Our work illustrates a comprehensive trustworthiness evaluation of GPT models and sheds light on the trustworthiness gaps.
  arXiv  Detail & Related papers  (2023-06-20T17:24:23Z)
• DiffSTG: Probabilistic Spatio-Temporal Graph Forecasting with Denoising Diffusion Models [53.67562579184457]
  This paper focuses on probabilistic STG forecasting, which is challenging due to the difficulty in modeling uncertainties and complex dependencies. We present the first attempt to generalize the popular denoising diffusion models to STGs, leading to a novel non-autoregressive framework called DiffSTG. Our approach combines the intrinsic-temporal learning capabilities STNNs with the uncertainty measurements of diffusion models.
  arXiv  Detail & Related papers  (2023-01-31T13:42:36Z)
• A Quantitative and Qualitative Analysis of Suicide Ideation Detection using Deep Learning [5.192118773220605]
  This paper replicated competitive social media-based suicidality detection/prediction models. We evaluated the feasibility of detecting suicidal ideation using multiple datasets and different state-of-the-art deep learning models.
  arXiv  Detail & Related papers  (2022-06-17T10:23:37Z)
• Am I No Good? Towards Detecting Perceived Burdensomeness and Thwarted Belongingness from Suicide Notes [51.378225388679425]
  We present an end-to-end multitask system to address a novel task of detection of Perceived Burdensomeness (PB) and Thwarted Belongingness (TB) from suicide notes. We also introduce a manually translated code-mixed suicide notes corpus, CoMCEASE-v2.0, based on the benchmark CEASE-v2.0 dataset. We exploit the temporal orientation and emotion information in the suicide notes to boost overall performance.
  arXiv  Detail & Related papers  (2022-05-20T06:31:08Z)
• An ensemble deep learning technique for detecting suicidal ideation from posts in social media platforms [0.0]
  This paper proposes a LSTM-Attention-CNN combined model to analyze social media submissions to detect suicidal intentions. The proposed model demonstrated an accuracy of 90.3 percent and an F1-score of 92.6 percent.
  arXiv  Detail & Related papers  (2021-12-17T15:34:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.