Harmful Suicide Content Detection

• URL: http://arxiv.org/abs/2407.13942v1
• Date: Mon, 3 Jun 2024 03:43:44 GMT
• Title: Harmful Suicide Content Detection
• Authors: Kyumin Park, Myung Jae Baik, YeongJun Hwang, Yen Shin, HoJae Lee, Ruda Lee, Sang Min Lee, Je Young Hannah Sun, Ah Rah Lee, Si Yeun Yoon, Dong-ho Lee, Jihyung Moon, JinYeong Bak, Kyunghyun Cho, Jong-Woo Paik, Sungjoon Park,
• Abstract summary: We introduce a harmful suicide content detection task for classifying online suicide content into five harmfulness levels. Our contributions include proposing a novel detection task, a multi-modal Korean benchmark with expert annotations, and suggesting strategies using LLMs to detect illegal and harmful content.
• Score: 35.0823928978658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Harmful suicide content on the Internet is a significant risk factor inducing suicidal thoughts and behaviors among vulnerable populations. Despite global efforts, existing resources are insufficient, specifically in high-risk regions like the Republic of Korea. Current research mainly focuses on understanding negative effects of such content or suicide risk in individuals, rather than on automatically detecting the harmfulness of content. To fill this gap, we introduce a harmful suicide content detection task for classifying online suicide content into five harmfulness levels. We develop a multi-modal benchmark and a task description document in collaboration with medical professionals, and leverage large language models (LLMs) to explore efficient methods for moderating such content. Our contributions include proposing a novel detection task, a multi-modal Korean benchmark with expert annotations, and suggesting strategies using LLMs to detect illegal and harmful content. Owing to the potential harm involved, we publicize our implementations and benchmark, incorporating an ethical verification process.

Related papers

• Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights [50.89022445197919]
  We propose a speech-specific risk taxonomy, covering 8 risk categories under hostility (malicious sarcasm and threats), malicious imitation (age, gender, ethnicity), and stereotypical biases (age, gender, ethnicity) Based on the taxonomy, we create a small-scale dataset for evaluating current LMMs capability in detecting these categories of risk.
  arXiv  Detail & Related papers  (2024-06-25T10:08:45Z)
• SOS-1K: A Fine-grained Suicide Risk Classification Dataset for Chinese Social Media Analysis [22.709733830774788]
  This study presents a Chinese social media dataset designed for fine-grained suicide risk classification. Seven pre-trained models were evaluated in two tasks: high and low suicide risk, and fine-grained suicide risk classification on a level of 0 to 10. Deep learning models show good performance in distinguishing between high and low suicide risk, with the best model achieving an F1 score of 88.39%.
  arXiv  Detail & Related papers  (2024-04-19T06:58:51Z)
• Integrating Supervised Extractive and Generative Language Models for Suicide Risk Evidence Summarization [2.0969143947138016]
  We propose a method that integrates supervised extractive and generative language models for providing supporting evidence of suicide risk. First, we construct a BERT-based model for estimating sentence-level suicide risk and negative sentiment. Next, we precisely identify high suicide risk sentences by emphasizing elevated probabilities of both suicide risk and negative sentiment. Finally, we integrate generative summaries using the MentaLLaMa framework and extractive summaries from identified high suicide risk sentences and a specialized dictionary of suicidal risk words.
  arXiv  Detail & Related papers  (2024-03-20T21:16:10Z)
• Navigating the OverKill in Large Language Models [84.62340510027042]
  We investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to an over-attention of harmful words like 'kill' and prompts emphasizing safety will exacerbate overkill. We introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon.
  arXiv  Detail & Related papers  (2024-01-31T07:26:47Z)
• Multilingual Jailbreak Challenges in Large Language Models [96.74878032417054]
  In this study, we reveal the presence of multilingual jailbreak challenges within large language models (LLMs) We consider two potential risky scenarios: unintentional and intentional. We propose a novel textscSelf-Defense framework that automatically generates multilingual training data for safety fine-tuning.
  arXiv  Detail & Related papers  (2023-10-10T09:44:06Z)
• On the Security Risks of Knowledge Graph Reasoning [71.64027889145261]
  We systematize the security threats to KGR according to the adversary's objectives, knowledge, and attack vectors. We present ROAR, a new class of attacks that instantiate a variety of such threats. We explore potential countermeasures against ROAR, including filtering of potentially poisoning knowledge and training with adversarially augmented queries.
  arXiv  Detail & Related papers  (2023-05-03T18:47:42Z)
• Am I No Good? Towards Detecting Perceived Burdensomeness and Thwarted Belongingness from Suicide Notes [51.378225388679425]
  We present an end-to-end multitask system to address a novel task of detection of Perceived Burdensomeness (PB) and Thwarted Belongingness (TB) from suicide notes. We also introduce a manually translated code-mixed suicide notes corpus, CoMCEASE-v2.0, based on the benchmark CEASE-v2.0 dataset. We exploit the temporal orientation and emotion information in the suicide notes to boost overall performance.
  arXiv  Detail & Related papers  (2022-05-20T06:31:08Z)
• An ensemble deep learning technique for detecting suicidal ideation from posts in social media platforms [0.0]
  This paper proposes a LSTM-Attention-CNN combined model to analyze social media submissions to detect suicidal intentions. The proposed model demonstrated an accuracy of 90.3 percent and an F1-score of 92.6 percent.
  arXiv  Detail & Related papers  (2021-12-17T15:34:03Z)
• Deep Learning for Suicide and Depression Identification with Unsupervised Label Correction [0.0]
  Early detection of suicidal ideation in depressed individuals can allow for adequate medical attention and support. Recent NLP research focuses on classifying, from a given piece of text, if an individual is suicidal or clinically healthy. We propose SDCNL, a suicide versus classification method through a deep learning approach.
  arXiv  Detail & Related papers  (2021-02-18T15:40:07Z)
• Survival Modeling of Suicide Risk with Rare and Uncertain Diagnoses [15.732431764583323]
  We use medical claims data to study the risk of subsequent suicide attempts for patients who were hospitalized due to suicide attempts and later discharged. Almost 20% of "suspected" suicide attempts are identified from diagnosis codes indicating external causes of injury and poisoning with undermined intent.
  arXiv  Detail & Related papers  (2020-09-05T20:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.