Social Bias in LLM-Generated Code: Benchmark and Mitigation

• URL: http://arxiv.org/abs/2605.00382v3
• Date: Tue, 05 May 2026 08:05:07 GMT
• Title: Social Bias in LLM-Generated Code: Benchmark and Mitigation
• Abstract summary: Large Language Models (LLMs) are increasingly deployed to generate code for human-centered applications where demographic fairness is critical.<n>Existing evaluations focus almost exclusively on functional correctness, leaving social bias in LLM-generated code largely unexamined.<n>We propose the Fairness Monitor Agent (FMA), a modular component that plugs into any existing code generation pipeline without modifying it.
• Score: 10.39413030802123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly deployed to generate code for human-centered applications where demographic fairness is critical. However, existing evaluations focus almost exclusively on functional correctness, leaving social bias in LLM-generated code largely unexamined. Extending our prior work on Solar, we conduct a comprehensive empirical study using SocialBias-Bench, a benchmark of 343 real-world coding tasks spanning seven demographic dimensions. We evaluate four prominent LLMs and find severe bias across all models, with Code Bias Scores reaching up to 60.58%. We further show that standard prompt-level interventions, such as Chain-of-Thought reasoning and fairness persona assignment, inadvertently amplify bias rather than reduce it. We then investigate whether structured multi-agent software process frameworks can improve fairness, finding that structured pipelines reduce bias when early roles correctly scope what the code should and should not consider. However, adding explicit fairness instructions to all agent roles produces worse outcomes than providing none, suggesting that diffused responsibility goes unaddressed. To address these limitations, we propose the Fairness Monitor Agent (FMA), a modular component that plugs into any existing code generation pipeline without modifying it. FMA analyzes the task description to determine which attributes should be considered or restricted, then detects and corrects violations through an iterative review process, without requiring an executable test suite. Evaluated on all 343 tasks, FMA reduces bias by 65.1% compared to a developer agent alone and improves functional correctness from 75.80% to 83.97%, outperforming all other studied approaches.

Related papers

• HarnessEval-W: Agentifying the Evaluation of Visual Worlds [149.63310396189317]
  HarnessEval-W is an agentified evaluation pipeline for benchmarking world models.<n>Rather than applying a fixed rubric, HarnessEval-W decomposes the evaluation question into measurable subproblems.<n>We apply HarnessEval-W to 18 representative world models over 330 evaluation cases.
  arXiv  Detail & Related papers  (2026-08-17T17:43:24Z)
• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• Regression Accumulation in Multi-Turn LLM Programming Conversations [25.94870617800846]
  regression accumulation appears across all six models.<n>Cross-Turn Conflict, where later code conflicts with earlier requirements, is the main failure class.<n> Verification Gate is the only strategy that consistently improves all models.
  arXiv  Detail & Related papers  (2026-07-02T08:15:40Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering [0.0]
  Large Language Models are increasingly used as judges to evaluate code artifacts.<n>Current practice lacks a principled account of reliability and bias.<n>This paper studies LLM-as-a-Judge for code through a measurement-first lens.
  arXiv  Detail & Related papers  (2026-04-18T02:35:05Z)
• Evaluating LLMs in Finance Requires Explicit Bias Consideration [88.38155218924999]
  Finance-specific biases can inflate performance, contaminate backtests, and make reported results useless for deployment claims.<n>No single bias is discussed in more than 28 percent of studies.<n>We propose a Structural Validity Framework and an evaluation checklist with minimal requirements for bias diagnosis and future system design.
  arXiv  Detail & Related papers  (2026-02-15T17:02:01Z)
• Summary-Mediated Repair: Can LLMs use code summarisation as a tool for program repair? [0.0]
  Large Language Models (LLMs) often produce code with subtle implementation-level bugs despite strong benchmark performance.<n>We propose summary-mediated repair, a prompt-only pipeline for program repair.
  arXiv  Detail & Related papers  (2025-11-24T05:33:38Z)
• HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment [52.374772443536045]
  HALF (Harm-Aware LLM Fairness) is a framework that assesses model bias in realistic applications and weighs the outcomes by harm severity.<n>We show that HALF exposes a clear gap between previous benchmarking success and deployment readiness.
  arXiv  Detail & Related papers  (2025-10-14T07:13:26Z)
• Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs [6.862079218077768]
  Testing Large Language Models on specific tasks is difficult and costly.<n>A key challenge is to assess input adequacy in a way that reflects the demands of the task.<n>We introduce CLOTHO, a task-specific, pre-generation adequacy measure.
  arXiv  Detail & Related papers  (2025-09-22T02:34:09Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• FairCoder: Evaluating Social Bias of LLMs in Code Generation [25.358230310973248]
  We introduce FairCoder, a novel benchmark for evaluating social bias in code generation.<n>Three metrics are designed to assess fairness performance on this benchmark.<n>The findings reveal that all tested LLMs exhibit social bias.
  arXiv  Detail & Related papers  (2025-01-09T17:42:23Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• See, Say, and Segment: Teaching LMMs to Overcome False Premises [67.36381001664635]
  We propose a cascading and joint training approach for LMMs to solve this task. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, and finally "segment" by outputting the mask of the desired objects if they exist.
  arXiv  Detail & Related papers  (2023-12-13T18:58:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.