Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks

• URL: http://arxiv.org/abs/2603.11914v1
• Date: Thu, 12 Mar 2026 13:29:38 GMT
• Title: Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks
• Abstract summary: Large Language Models (LLMs) are increasingly trained to align with human values, primarily focusing on task level, i.e., refusing to execute harmful tasks.<n>Will LLMs, like morally conscious human beings, refuse to proceed when encountering harmful content in user-supplied material?<n>We first construct a harmful knowledge dataset (i.e., non-compliant with OpenAI's usage policy) to serve as the user-supplied harmful content.<n>We then design nine harmless tasks (i.e., compliant with OpenAI's usage policy) to simulate the real-world benign tasks
• Score: 32.56625507077786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly trained to align with human values, primarily focusing on task level, i.e., refusing to execute directly harmful tasks. However, a subtle yet crucial content-level ethical question is often overlooked: when performing a seemingly benign task, will LLMs -- like morally conscious human beings -- refuse to proceed when encountering harmful content in user-provided material? In this study, we aim to understand this content-level ethical question and systematically evaluate its implications for mainstream LLMs. We first construct a harmful knowledge dataset (i.e., non-compliant with OpenAI's usage policy) to serve as the user-supplied harmful content, with 1,357 entries across ten harmful categories. We then design nine harmless tasks (i.e., compliant with OpenAI's usage policy) to simulate the real-world benign tasks, grouped into three categories according to the extent of user-supplied content required: extensive, moderate, and limited. Leveraging the harmful knowledge dataset and the set of harmless tasks, we evaluate how nine LLMs behave when exposed to user-supplied harmful content during the execution of benign tasks, and further examine how the dynamics between harmful knowledge categories and tasks affect different LLMs. Our results show that current LLMs, even the latest GPT-5.2 and Gemini-3-Pro, often fail to uphold human-aligned ethics by continuing to process harmful content in harmless tasks. Furthermore, external knowledge from the ``Violence/Graphic'' category and the ``Translation'' task is more likely to elicit harmful responses from LLMs. We also conduct extensive ablation studies to investigate potential factors affecting this novel misuse vulnerability. We hope that our study could inspire enhanced safety measures among stakeholders to mitigate this overlooked content-level ethical risk.

Related papers

• Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Models [1.4092255915948473]
  Large Language Models (LLMs) have the potential to create harmful content.<n>In this study, we propose Amnesia, a lightweight activation-space adversarial attack.<n>Our experiments show that the proposed attack can induce various antisocial behaviors in LLMs.
  arXiv  Detail & Related papers  (2026-03-10T09:41:03Z)
• Evaluating the Sensitivity of LLMs to Harmful Contents in Long Input [53.19281984086319]
  Large language models (LLMs) increasingly support applications that rely on extended context, from document processing to retrieval-augmented generation.<n>We evaluate LLMs' sensitivity to harmful content under extended context, varying type (explicit vs. implicit), position (beginning, middle, end), prevalence (0.01-0.50 of the prompt), and context length (600-6000 tokens).<n>We observe similar patterns: performance peaks at moderate harmful prevalence (0.25) but declines when content is very sparse or dominant; recall decreases with increasing context length; harmful sentences at the beginning are generally detected more reliably; and explicit content is more consistently recognized than implicit
  arXiv  Detail & Related papers  (2025-10-07T12:33:21Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs [1.7451266777840306]
  Large language models (LLMs) have become integral to various real-world applications, leveraging massive, web-sourced datasets like Common Crawl, C4, and FineWeb for pretraining.<n>Training LLMs on such unfiltered data risks perpetuating toxic behaviors, spreading misinformation, and amplifying societal biases.<n>This paper presents a large-scale analysis of inappropriate content across these datasets, offering a comprehensive taxonomy that categorizes harmful webpages into Topical and Toxic based on their intent.
  arXiv  Detail & Related papers  (2025-05-04T06:37:20Z)
• Do LLMs Consider Security? An Empirical Study on Responses to Programming Questions [10.69738882390809]
  ChatGPT can volunteer context-specific information to developers, promoting safe coding practices.<n>We evaluate the degree of security awareness exhibited by three prominent LLMs: Claude 3, GPT-4, and Llama 3.<n>Our findings show that all three models struggle to accurately detect and warn users about vulnerabilities, achieving a detection rate of only 12.6% to 40% across our datasets.
  arXiv  Detail & Related papers  (2025-02-20T02:20:06Z)
• Persuasion with Large Language Models: a Survey [49.86930318312291]
  Large Language Models (LLMs) have created new disruptive possibilities for persuasive communication. In areas such as politics, marketing, public health, e-commerce, and charitable giving, such LLM Systems have already achieved human-level or even super-human persuasiveness. Our survey suggests that the current and future potential of LLM-based persuasion poses profound ethical and societal risks.
  arXiv  Detail & Related papers  (2024-11-11T10:05:52Z)
• KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language Models [69.99274367773997]
  Large Language Models (LLMs) often struggle with dynamically changing knowledge and handling unknown static information.<n>Retrieval-Augmented Generation (RAG) is employed to tackle these challenges and has a significant impact on improving LLM performance.<n>We propose a Knowledge Boundary Model (KBM) to express the known/unknown of a given question, and to determine whether a RAG needs to be triggered.
  arXiv  Detail & Related papers  (2024-11-09T15:12:28Z)
• CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models [60.59638232596912]
  We introduce CLAMBER, a benchmark for evaluating large language models (LLMs) Building upon the taxonomy, we construct 12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries.
  arXiv  Detail & Related papers  (2024-05-20T14:34:01Z)
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming [64.86326523181553]
  ALERT is a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models.
  arXiv  Detail & Related papers  (2024-04-06T15:01:47Z)
• The Ethics of Interaction: Mitigating Security Threats in LLMs [1.407080246204282]
  The paper delves into the nuanced ethical repercussions of such security threats on society and individual privacy. We scrutinize five major threats--prompt injection, jailbreaking, Personal Identifiable Information (PII) exposure, sexually explicit content, and hate-based content--to assess their critical ethical consequences and the urgency they create for robust defensive strategies.
  arXiv  Detail & Related papers  (2024-01-22T17:11:37Z)
• Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity [19.94836502156002]
  Large language models (LLMs) may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. We empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks.
  arXiv  Detail & Related papers  (2023-01-30T13:20:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.