Bias at the End of the Score

• URL: http://arxiv.org/abs/2604.13305v1
• Date: Tue, 14 Apr 2026 21:20:47 GMT
• Title: Bias at the End of the Score
• Abstract summary: Reward models (RMs) are inherently non-neutral value functions designed to encode specific objectives.<n>We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation.
• Score: 42.96226771321315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward models (RMs) are inherently non-neutral value functions designed and trained to encode specific objectives, such as human preferences or text-image alignment. RMs have become crucial components of text-to-image (T2I) generation systems where they are used at various stages for dataset filtering, as evaluation metrics, as a supervisory signal during optimization of parameters, and for post-generation safety and quality filtering of T2I outputs. While specific problems with the integration of RMs into the T2I pipeline have been studied (e.g. reward hacking or mode collapse), their robustness and fairness as scoring functions remains largely unknown. We conduct a large scale audit of RM robustness with respect to demographic biases during T2I model training and generation. We provide quantitative and qualitative evidence that while originally developed as quality measures, RMs encode demographic biases, which cause reward-guided optimization to disproportionately sexualize female image subjects reinforce gender/racial stereotypes, and collapse demographic diversity. These findings highlight shortcomings in current reward models, challenge their reliability as quality metrics, and underscore the need for improved data collection and training procedures to enable more robust scoring.

Related papers

• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models [53.19726629537694]
  Post-training alignment of video generation models with human preferences is a critical goal.<n>Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise.<n>We propose SoliReward, a systematic framework for video RM training.
  arXiv  Detail & Related papers  (2025-12-17T14:28:23Z)
• Evaluating and Preserving High-level Fidelity in Super-Resolution [50.65679806442527]
  Super-Resolution (SR) models are achieving impressive effects in reconstructing details and delivering pleasant visually outputs.<n>However, the overpowering generative ability can sometimes hallucinate and thus change the image content.<n>This type of high-level change can be easily identified by humans yet not well-studied in existing low-level image quality metrics.
  arXiv  Detail & Related papers  (2025-12-07T22:53:34Z)
• Multi-Metric Preference Alignment for Generative Speech Restoration [15.696247605348383]
  We propose a multi-metric preference alignment strategy for generative models.<n>We observe consistent and significant performance gains across three diverse generative paradigms.<n>Our aligned models can serve as powerful ''data annotators'', generating high-quality pseudo-labels.
  arXiv  Detail & Related papers  (2025-08-24T07:05:10Z)
• Model-Agnostic Gender Bias Control for Text-to-Image Generation via Sparse Autoencoder [14.164976259534143]
  Text-to-image (T2I) diffusion models often exhibit gender bias, particularly by generating stereotypical associations between professions and gendered subjects.<n>This paper presents SAE Debias, a model-agnostic framework for mitigating such bias in T2I generation.<n>To the best of our knowledge, this is the first work to apply sparse autoencoders for identifying and intervening in gender bias within T2I models.
  arXiv  Detail & Related papers  (2025-07-28T16:36:13Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• FairT2I: Mitigating Social Bias in Text-to-Image Generation via Large Language Model-Assisted Detection and Attribute Rebalancing [32.01426831450348]
  We introduce FairT2I, a novel framework that harnesses large language models to detect and mitigate social biases in T2I generation.<n>Our results show that FairT2I successfully mitigates social biases and enhances the diversity of sensitive attributes in generated images.
  arXiv  Detail & Related papers  (2025-02-06T07:22:57Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)
• On Feature Learning in the Presence of Spurious Correlations [45.86963293019703]
  We show that the quality learned feature representations is greatly affected by the design decisions beyond the method. We significantly improve upon the best results reported in the literature on the popular Waterbirds, Celeb hair color prediction and WILDS-FMOW problems.
  arXiv  Detail & Related papers  (2022-10-20T16:10:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.