Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks

• URL: http://arxiv.org/abs/2603.18765v1
• Date: Thu, 19 Mar 2026 11:20:07 GMT
• Title: Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks
• Abstract summary: This study investigates whether large language models (LLMs) exhibit implicit grading bias based on writing style when the underlying content correctness remains constant.<n>Two state-of-the-art open-source LLMs were prompted to grade responses on a 1-10 scale with explicit instructions to evaluate content correctness only and to disregard writing style.<n>Our results reveal statistically significant grading bias in Essay/Writing tasks across both models and all perturbation types.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) are increasingly deployed as automated graders in educational settings, concerns about fairness and bias in their evaluations have become critical. This study investigates whether LLMs exhibit implicit grading bias based on writing style when the underlying content correctness remains constant. We constructed a controlled dataset of 180 student responses across three subjects (Mathematics, Programming, and Essay/Writing), each with three surface-level perturbation types: grammar errors, informal language, and non-native phrasing. Two state-of-the-art open-source LLMs -- LLaMA 3.3 70B (Meta) and Qwen 2.5 72B (Alibaba) -- were prompted to grade responses on a 1-10 scale with explicit instructions to evaluate content correctness only and to disregard writing style. Our results reveal statistically significant grading bias in Essay/Writing tasks across both models and all perturbation types (p < 0.05), with effect sizes ranging from medium (Cohen's d = 0.64) to very large (d = 4.25). Informal language received the heaviest penalty, with LLaMA deducting an average of 1.90 points and Qwen deducting 1.20 points on a 10-point scale -- penalties comparable to the difference between a B+ and C+ letter grade. Non-native phrasing was penalized 1.35 and 0.90 points respectively. In sharp contrast, Mathematics and Programming tasks showed minimal bias, with most conditions failing to reach statistical significance. These findings demonstrate that LLM grading bias is subject-dependent, style-sensitive, and persists despite explicit counter-bias instructions in the grading prompt. We discuss implications for equitable deployment of LLM-based grading systems and recommend bias auditing protocols before institutional adoption.

Related papers

• Can LLMs in Draft-Verify-Revise Pipelines Resolve Deictic Ambiguity? [0.0]
  Draft-verify-revise is an orchestration pattern for scaling inference-time compute.<n>Draft-verify-revise pipelines should be wary of deictic shifts and make the intended referent explicit at each stage.
  arXiv  Detail & Related papers  (2026-09-10T19:49:06Z)
• LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability [0.0]
  Open-ended mathematics exams are valuable because they assess reasoning, proof construction, algorithmic thinking, and communication of intermediate steps.<n>This paper evaluates six contemporary large language model (LLM) configurations as grading assistants for an undergraduate discrete mathematics examination.
  arXiv  Detail & Related papers  (2026-06-01T18:00:34Z)
• Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs [2.727845763576068]
  Vision-capable large language models (LLMs) offer new opportunities.<n>We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics.
  arXiv  Detail & Related papers  (2026-05-18T19:06:15Z)
• HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing [84.84959282480641]
  We propose Tree-of-Writing (ToW) to resolve the implicit inconsistency often found when LLM-as-a-judge aggregates all sub-features in text evaluation.<n>We also present HowToBench, a large-scale Chinese writing benchmark encompassing 12 genres and 1302 instructions across three task categories.
  arXiv  Detail & Related papers  (2026-04-21T04:26:39Z)
• Leveraging Large Language Models for Predictive Analysis of Human Misery [1.2458057399345226]
  This study investigates the use of Large Language Models (LLMs) for predicting human-perceived misery scores.<n>We evaluate multiple prompting strategies, including zero-shot, fixed-context few-shot, and retrieval-based prompting.<n>To move beyond static evaluation, we introduce the "Misery Game Show", a novel gamified framework inspired by a television format.
  arXiv  Detail & Related papers  (2025-08-18T07:02:59Z)
• Implicit Bias in LLMs: A Survey [2.07180164747172]
  This paper provides a comprehensive review of the existing literature on implicit bias in Large language models.<n>We begin by introducing key concepts, theories and methods related to implicit bias in psychology.<n>We categorize detection methods into three primary approaches: word association, task-oriented text generation and decision-making.
  arXiv  Detail & Related papers  (2025-03-04T16:49:37Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Likelihood-based Mitigation of Evaluation Bias in Large Language Models [29.953609131069523]
  Large Language Models (LLMs) are widely used to evaluate natural language generation tasks as automated metrics.<n>It is possible that there might be a likelihood bias if LLMs are used for evaluation.
  arXiv  Detail & Related papers  (2024-02-25T04:52:02Z)
• GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models [83.30078426829627]
  Large language models (LLMs) have gained popularity and are being widely adopted by a large user community. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. We propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs to assess bias in models.
  arXiv  Detail & Related papers  (2023-12-11T12:02:14Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases [55.45617404586874]
  We propose a few-shot instruction-based method for prompting pre-trained language models (LMs) We show that large LMs can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models.
  arXiv  Detail & Related papers  (2021-12-15T04:19:52Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.