Automated Harmfulness Testing for Code Large Language Models

• URL: http://arxiv.org/abs/2503.16740v1
• Date: Thu, 20 Mar 2025 23:06:06 GMT
• Title: Automated Harmfulness Testing for Code Large Language Models
• Authors: Honghao Tan, Haibo Wang, Diany Pressato, Yisen Xu, Shin Hwei Tan,
• Abstract summary: Generative AI systems powered by Large Language Models (LLMs) usually use content moderation to prevent harmful content spread.<n> Exposure to harmful content in software artifacts can negatively impact the mental health of developers.<n>We propose a coverage-guided harmfulness testing framework that generates prompts using diverse transformations and harmful keywords injected into benign programs.
• Score: 5.847020442165636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI systems powered by Large Language Models (LLMs) usually use content moderation to prevent harmful content spread. To evaluate the robustness of content moderation, several metamorphic testing techniques have been proposed to test content moderation software. However, these techniques mainly focus on general users (e.g., text and image generation). Meanwhile, a recent study shows that developers consider using harmful keywords when naming software artifacts to be an unethical behavior. Exposure to harmful content in software artifacts can negatively impact the mental health of developers, making content moderation for Code Large Language Models (Code LLMs) essential. We conduct a preliminary study on program transformations that can be misused to introduce harmful content into auto-generated code, identifying 32 such transformations. To address this, we propose CHT, a coverage-guided harmfulness testing framework that generates prompts using diverse transformations and harmful keywords injected into benign programs. CHT evaluates output damage to assess potential risks in LLM-generated explanations and code. Our evaluation of four Code LLMs and GPT-4o-mini reveals that content moderation in LLM-based code generation is easily bypassed. To enhance moderation, we propose a two-phase approach that first detects harmful content before generating output, improving moderation effectiveness by 483.76\%.

Related papers

• BingoGuard: LLM Content Moderation Tools with Risk Levels [67.53167973090356]
  Malicious content generated by large language models (LLMs) can pose varying degrees of harm. In this paper, we introduce per-topic severity rubrics for 11 harmful topics and build BingoGuard, an LLM-based moderation system.
  arXiv  Detail & Related papers  (2025-03-09T10:43:09Z)
• Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis [3.892345568697058]
  Large Language Models (LLMs) are one of the most promising developments in the field of artificial intelligence.<n>Developers routinely ask LLMs to generate code snippets, increasing productivity but also introducing ownership, privacy, correctness, and security issues.<n>Previous work highlighted how code generated by commercial LLMs is often not safe, containing vulnerabilities, bugs, and code smells.
  arXiv  Detail & Related papers  (2024-12-19T13:34:14Z)
• What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models [0.5735035463793009]
  We investigate the vulnerability of large language models (LLMs) to imperceptible attacks, where hidden character manipulation in source code misleads LLMs' behaviour while remaining undetectable to human reviewers. These attacks include coding reordering, invisible coding characters, code deletions, and code homoglyphs. Our findings confirm the susceptibility of LLMs to imperceptible coding character attacks, while different LLMs present different negative correlations between perturbation magnitude and performance.
  arXiv  Detail & Related papers  (2024-12-11T04:52:41Z)
• RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code [30.244754704562162]
  There is no research evaluating the ability of LLMs to resist malicious code generation. We conduct an empirical study on 11 representative LLMs to assess their ability to resist malicious code generation. Our findings indicate that current LLMs have a limited ability to resist malicious code generation with an average refusal rate of 40.36% in text-to-code scenario and 11.52% in code-to-code scenario.
  arXiv  Detail & Related papers  (2024-09-23T16:03:26Z)
• ShieldGemma: Generative AI Content Moderation Based on Gemma [49.91147965876678]
  ShieldGemma is a suite of safety content moderation models built upon Gemma2. Models provide robust, state-of-the-art predictions of safety risks across key harm types.
  arXiv  Detail & Related papers  (2024-07-31T17:48:14Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests. First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics. Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion [117.178835165855]
  This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs. Our studies reveal a new and universal safety vulnerability of these models against code input. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization.
  arXiv  Detail & Related papers  (2024-03-12T17:55:38Z)
• LLM4TDD: Best Practices for Test Driven Development Using Large Language Models [0.76146285961466]
  This paper explores the concept of LLM4TDD, where we guide Large Language Models to generate code iteratively using a test-driven development methodology. We conduct an empirical evaluation using ChatGPT and coding problems from LeetCode to investigate the impact of different test, prompt and problem attributes on the efficacy of LLM4TDD.
  arXiv  Detail & Related papers  (2023-12-07T20:37:54Z)
• Contrastive Decoding Improves Reasoning in Large Language Models [55.16503283583076]
  We show that Contrastive Decoding achieves large out-of-the-box improvements over greedy decoding on a variety of reasoning tasks. We show that Contrastive Decoding leads LLaMA-65B to outperform LLaMA 2, GPT-3.5 and PaLM 2-L on the HellaSwag commonsense reasoning benchmark.
  arXiv  Detail & Related papers  (2023-09-17T00:29:32Z)
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing [139.77117915309023]
  CRITIC allows large language models to validate and amend their own outputs in a manner similar to human interaction with tools. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs.
  arXiv  Detail & Related papers  (2023-05-19T15:19:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.