Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM

• URL: http://arxiv.org/abs/2508.05775v2
• Date: Wed, 13 Aug 2025 07:40:27 GMT
• Title: Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM
• Authors: Chi Zhang, Changjia Zhu, Junjie Xiong, Xiaoran Xu, Lingyao Li, Yao Liu, Zhuo Lu,
• Abstract summary: Large Language Models (LLMs) have revolutionized content creation across digital platforms.<n>LLMs enable beneficial applications such as content generation, question and answering (Q&A), programming, and code reasoning.<n>They also pose serious risks by inadvertently or intentionally producing toxic, offensive, or biased content.
• Score: 13.066526969147501
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have revolutionized content creation across digital platforms, offering unprecedented capabilities in natural language generation and understanding. These models enable beneficial applications such as content generation, question and answering (Q&A), programming, and code reasoning. Meanwhile, they also pose serious risks by inadvertently or intentionally producing toxic, offensive, or biased content. This dual role of LLMs, both as powerful tools for solving real-world problems and as potential sources of harmful language, presents a pressing sociotechnical challenge. In this survey, we systematically review recent studies spanning unintentional toxicity, adversarial jailbreaking attacks, and content moderation techniques. We propose a unified taxonomy of LLM-related harms and defenses, analyze emerging multimodal and LLM-assisted jailbreak strategies, and assess mitigation efforts, including reinforcement learning with human feedback (RLHF), prompt engineering, and safety alignment. Our synthesis highlights the evolving landscape of LLM safety, identifies limitations in current evaluation methodologies, and outlines future research directions to guide the development of robust and ethically aligned language technologies.

Related papers

• A Survey on Data Security in Large Language Models [12.23432845300652]
  Large Language Models (LLMs) are a foundation in advancing natural language processing, power applications such as text generation, machine translation, and conversational systems.<n>Despite their transformative potential, these models inherently rely on massive amounts of training data, often collected from diverse and uncurated sources, which exposes them to serious data security risks.<n>Harmful or malicious data can compromise model behavior, leading to issues such as toxic output, hallucinations, and vulnerabilities to threats such as prompt injection or data poisoning.<n>This survey offers a comprehensive overview of the main data security risks facing LLMs and reviews current defense strategies, including adversarial
  arXiv  Detail & Related papers  (2025-08-04T11:28:34Z)
• Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity? [51.96049148869987]
  Large Language Models (LLMs) offer new tools and challenges for the security of computer systems.<n>We investigate whether classical game-theoretic frameworks can effectively capture the behaviours of LLM-driven actors and bots.
  arXiv  Detail & Related papers  (2025-08-04T08:57:14Z)
• Security Concerns for Large Language Models: A Survey [3.175227858236288]
  Large Language Models (LLMs) have caused a revolution in natural language processing, but their capabilities also introduce new security vulnerabilities.<n>We provide a comprehensive overview of the emerging security concerns around LLMs, categorizing threats into prompt injection and jailbreaking, adversarial attacks such as input perturbations and data poisoning, and worrisome risks inherent in autonomous LLM agents.<n>We conclude by emphasizing the importance of advancing robust, multi-layered security strategies to ensure LLMs are safe and beneficial.
  arXiv  Detail & Related papers  (2025-05-24T22:22:43Z)
• Persuasion with Large Language Models: a Survey [49.86930318312291]
  Large Language Models (LLMs) have created new disruptive possibilities for persuasive communication. In areas such as politics, marketing, public health, e-commerce, and charitable giving, such LLM Systems have already achieved human-level or even super-human persuasiveness. Our survey suggests that the current and future potential of LLM-based persuasion poses profound ethical and societal risks.
  arXiv  Detail & Related papers  (2024-11-11T10:05:52Z)
• Jailbreaking and Mitigation of Vulnerabilities in Large Language Models [8.345554966569479]
  Large Language Models (LLMs) have transformed artificial intelligence by advancing natural language understanding and generation.<n>Despite these advancements, LLMs have shown considerable vulnerabilities, particularly to prompt injection and jailbreaking attacks.<n>This review analyzes the state of research on these vulnerabilities and presents available defense strategies.
  arXiv  Detail & Related papers  (2024-10-20T00:00:56Z)
• AI Safety in Generative AI Large Language Models: A Survey [14.737084887928408]
  Large Language Model (LLMs) that exhibit generative AI capabilities are facing accelerated adoption and innovation. Generative AI (GAI) inevitably raises concerns about the risks and safety associated with these models. This article provides an up-to-date survey of recent trends in AI safety research of GAI-LLMs from a computer scientist's perspective.
  arXiv  Detail & Related papers  (2024-07-06T09:00:18Z)
• Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey [63.4581186135101]
  Large language models (LLMs) have made remarkable advancements in natural language processing.<n>Privacy and security issues have been revealed throughout their life cycle.<n>This survey outlines and analyzes potential countermeasures.
  arXiv  Detail & Related papers  (2024-06-12T07:55:32Z)
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion [117.178835165855]
  This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs. Our studies reveal a new and universal safety vulnerability of these models against code input. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization.
  arXiv  Detail & Related papers  (2024-03-12T17:55:38Z)
• Large Language Models are Vulnerable to Bait-and-Switch Attacks for Generating Harmful Content [33.99403318079253]
  Even safe text coming from large language models can be turned into potentially dangerous content through Bait-and-Switch attacks. The alarming efficacy of this approach highlights a significant challenge in developing reliable safety guardrails for LLMs.
  arXiv  Detail & Related papers  (2024-02-21T16:46:36Z)
• Rethinking Machine Unlearning for Large Language Models [85.92660644100582]
  We explore machine unlearning in the domain of large language models (LLMs)<n>This initiative aims to eliminate undesirable data influence (e.g., sensitive or illegal information) and the associated model capabilities.
  arXiv  Detail & Related papers  (2024-02-13T20:51:58Z)
• Security and Privacy Challenges of Large Language Models: A Survey [2.6986500640871482]
  Large Language Models (LLMs) have demonstrated extraordinary capabilities and contributed to multiple fields, such as generating and summarizing text, language translation, and question-answering. These models are also vulnerable to security and privacy attacks, such as jailbreaking attacks, data poisoning attacks, and Personally Identifiable Information (PII) leakage attacks. This survey provides a thorough review of the security and privacy challenges of LLMs for both training data and users, along with the application-based risks in various domains, such as transportation, education, and healthcare.
  arXiv  Detail & Related papers  (2024-01-30T04:00:54Z)
• FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicity [20.510512358961517]
  The widespread of generative artificial intelligence has heightened concerns about the potential harms posed by AI-generated texts.<n>Previous researchers have invested much effort in assessing the harmlessness of generative language models.
  arXiv  Detail & Related papers  (2023-11-30T14:18:47Z)
• A Survey on Detection of LLMs-Generated Content [97.87912800179531]
  The ability to detect LLMs-generated content has become of paramount importance. We aim to provide a detailed overview of existing detection strategies and benchmarks. We also posit the necessity for a multi-faceted approach to defend against various attacks.
  arXiv  Detail & Related papers  (2023-10-24T09:10:26Z)
• On the Risk of Misinformation Pollution with Large Language Models [127.1107824751703]
  We investigate the potential misuse of modern Large Language Models (LLMs) for generating credible-sounding misinformation. Our study reveals that LLMs can act as effective misinformation generators, leading to a significant degradation in the performance of Open-Domain Question Answering (ODQA) systems.
  arXiv  Detail & Related papers  (2023-05-23T04:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.