Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation

• URL: http://arxiv.org/abs/2605.02348v1
• Date: Mon, 04 May 2026 08:51:34 GMT
• Title: Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation
• Abstract summary: Large language models pick up social biases from the data they are trained on and carry those biases into downstream applications.<n>In this paper we take a different route: we debias the model at decoding time, treating bias mitigation as a structured search over candidate tokens without ever touching model weights.<n>We design three schemes of increasing sophistication (Best-of-N selection, Sequential critique-and-revise, and Constitutional self-audit) and evaluate them on four models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models pick up social biases from the data they are trained on and carry those biases into downstream applications, often reinforcing stereotypes around gender, race, religion, disability, age, and socioeconomic status. The standard fixes (retraining on curated data or fine-tuning with human feedback) are expensive, need access to model weights, and risk degrading the model on other tasks. In this paper we take a different route: we debias the model at decoding time, treating bias mitigation as a structured search over candidate tokens without ever touching model weights. A separate Process Reward Model (PRM) acts as a judge, scoring each candidate for both fairness and fluency. We design three schemes of increasing sophistication (Best-of-N selection, Sequential critique-and-revise, and Constitutional self-audit) and evaluate them on four models (GPT-4o-mini, Llama 3.2 3B, Gemma 3 4B, Qwen 2.5 3B) across a 200-prompt bilingual benchmark in English and Urdu covering eight bias categories. Sequential debiasing proves the most effective, raising mean bias scores by up to +0.40 over baseline while preserving (and sometimes improving) fluency. We then extend all three schemes to open-ended generation, where each token is debiased on the fly, and introduce a lightweight Bias Guard gate that fires only on potentially biased words, keeping overhead near 2x for well-calibrated models. A formal overhead metric that separates generator cost from judge cost reveals that Best-of-N is effectively free on the generator side in a native implementation. GPT-4o-mini, included as a strong proprietary anchor, confirms that the framework scales with model capability; the three open-weight models show where current small-scale LLMs still struggle.

Related papers

• LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning [6.241883798820154]
  We study cross-model consensus, the degree to which independently trained models, each solving the problem once, agree on a final answer.<n>We treat the panel as an LLM-jury, in which the structure of agreement, not any model's score of another, is the verification signal.<n>We make this precise with a parameter-free law, derived in closed form, that predicts consensus accuracy from three measured panel statistics to a mean absolute error of $0.03$.
  arXiv  Detail & Related papers  (2026-07-11T05:57:54Z)
• Primal Generation, Dual Judgment: Self-Training from Test-Time Scaling [29.008352630270192]
  We introduce DuST (Dual Self-Training), a framework for self-training from the dual judgment space.<n>We show that DuST consistently improves Best-of-4 test-time scaling on LiveCodeBench.<n>For Qwen3-30B-Thinking on LiveCodeBench v6, judgment quality improves by +6.2 NDCG, single-sample pass@1 improves by +3.1, and Best-of-4 accuracy improves by +4.1.
  arXiv  Detail & Related papers  (2026-05-11T22:34:45Z)
• ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment [49.93516092785756]
  Current reward modeling is heavily contingent upon experimental feedback data with high collection costs.<n>ImplicitRM aims to learn unbiased reward models from implicit preference data.<n> Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets.
  arXiv  Detail & Related papers  (2026-03-24T13:32:14Z)
• Bias Fitting to Mitigate Length Bias of Reward Model in RLHF [81.44256822500257]
  Reinforcement Learning from Human Feedback relies on reward models to align large language models with human preferences.<n>We propose FiMi-RM, a framework that autonomously learns and corrects underlying bias patterns.<n> Experimental results demonstrate that FiMi-RM achieves a more balanced length-reward distribution.
  arXiv  Detail & Related papers  (2025-05-19T08:29:28Z)
• Bias Similarity Measurement: A Black-Box Audit of Fairness Across LLMs [35.72288339965426]
  Bias Similarity Measurement (BSM) treats fairness as a relational property between models.<n>Open-weight models can match or exceed proprietary systems.<n>BSM offers an auditing workflow for procurement, regression testing, and lineage screening.
  arXiv  Detail & Related papers  (2024-10-15T19:21:14Z)
• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models [56.02275285521847]
  We propose to evaluate models using a Panel of LLm evaluators (PoLL) We find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.
  arXiv  Detail & Related papers  (2024-04-29T15:33:23Z)
• Fine-tuning Language Models for Factuality [96.5203774943198]
  Large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations' In this work, we fine-tune language models to be more factual, without human labeling.
  arXiv  Detail & Related papers  (2023-11-14T18:59:15Z)
• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)
• Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing [33.25539075550122]
  We extend an existing bias benchmark for NLI using a combination of LM-generated lexical variations, adversarial filtering, and human validation. We show that BBNLI-next reduces the accuracy of state-of-the-art NLI models from 95.3% to a strikingly low 57.5%. We propose bias measures that take into account both bias and model brittleness.
  arXiv  Detail & Related papers  (2023-05-22T01:02:45Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.