Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling

• URL: http://arxiv.org/abs/2607.15740v1
• Date: Fri, 17 Jul 2026 08:23:03 GMT
• Title: Debiasing Text-to-Image Evaluation via Implicit Cultural Alignment Reward Modeling
• Abstract summary: We introduce an Implicit Cultural Alignment Reward Model for generative AI.<n>Our framework integrates an Implicit Cultural Probe with a Skip-connection Cross-Attention mechanism.<n>Our model processes each evaluation in 0.21 seconds under our local inference setup, achieving a $10times$ speedup over standard VQA-based evaluators.
• Score: 1.1172382217477128
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Text-to-Image (T2I) systems rapidly advance, evaluating the cultural authenticity of synthesized content has become increasingly important for fair and trustworthy generative AI. Existing T2I evaluation metrics and multimodal judges often rely on visual-semantic representations that underrepresent implicit cultural norms, leading to biased preference judgments and the omission of fine-grained cultural cues. In addition, visual question answering (VQA)-based evaluators typically depend on autoregressive text generation, which limits their scalability for real-time reward modeling. To address these limitations, we introduce an Implicit Cultural Alignment Reward Model built upon a lightweight 4.2-billion-parameter Multimodal Large Language Model (MLLM). Our framework integrates an Implicit Cultural Probe with a Skip-connection Cross-Attention (SkipCA) mechanism, enabling late-stage semantic features to directly attend to early-stage visual representations and better preserve culturally salient details. Evaluations on 3,323 challenging and carefully curated image pairs from the CulturalFrames benchmark show that our approach achieves 80.54% pairwise accuracy, with Pearson and Kendall correlation coefficients of 0.546 and 0.377, respectively, outperforming representative vision-language metrics and MLLM-based evaluators. Moreover, by bypassing autoregressive text generation, our model processes each evaluation in 0.21 seconds under our local inference setup, achieving a $10\times$ speedup over standard VQA-based evaluators. These results suggest that the proposed reward model can provide an efficient and culturally aware scalar signal for preference optimization pipelines such as Reinforcement Learning from Human Feedback and Direct Preference Optimization.

Related papers

• AlignCultura: Towards Culturally Aligned Large Language Models? [19.134202394422285]
  Align-Cultura is a two-stage pipeline for cultural alignment.<n>CulTURAX is the HHH-English dataset grounded in the UNESCO cultural taxonomy.<n> Stage II benchmarks CULTURAX on general-purpose models, culturally fine-tuned models, and open-weight LLMs.<n> Empirically, culturally fine-tuned models improve joint HHH by 4%-6%, reduce cultural failures by 18%, achieve 10%-12% efficiency gains, and limit leakage to 0.3%.
  arXiv  Detail & Related papers  (2026-04-21T03:06:50Z)
• Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models [4.317017868438435]
  Vision-Language Models (VLMs) excel at visual perception, yet their ability to interpret cultural meaning in art remains under-validated.<n>Tier I computes automated coverage and risk indicators offline; Tier II applies rubric-based scoring using a single primary judge across five dimensions.<n>Tier III calibrates the Tier II aggregate score to human ratings via isotonic regression, yielding a 5.2% reduction in MAE on a 152-sample held-out set.
  arXiv  Detail & Related papers  (2026-01-12T20:33:35Z)
• CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluation [61.130639734982395]
  We introduce CAIRe, a novel evaluation metric that assesses the degree of cultural relevance of an image.<n>Our framework grounds entities and concepts in the image to a knowledge base and uses factual information to give independent graded judgments for each culture label.
  arXiv  Detail & Related papers  (2025-06-10T17:16:23Z)
• CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics [23.567641319277943]
  We quantify the alignment of text-to-image (T2I) models and evaluation metrics.<n>CulturalFrames is a novel benchmark for rigorous human evaluation of cultural representation.<n>We find that across models and countries, cultural expectations are missed an average of 44% of the time.
  arXiv  Detail & Related papers  (2025-06-10T14:21:46Z)
• T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation [60.620408007636016]
  We propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores.<n>Our approach integrates Group Relative Policy Optimization into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains.
  arXiv  Detail & Related papers  (2025-05-23T13:44:59Z)
• Multimodal Cultural Safety: Evaluation Frameworks and Alignment Strategies [58.88053690412802]
  Large vision-language models (LVLMs) are increasingly deployed in globally distributed applications, such as tourism assistants.<n> CROSS is a benchmark designed to assess the cultural safety reasoning capabilities of LVLMs.<n>We evaluate 21 leading LVLMs, including mixture-of-experts models and reasoning models.
  arXiv  Detail & Related papers  (2025-05-20T23:20:38Z)
• RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding [79.44246283490665]
  We introduce RAVENEA, a new benchmark designed to advance visual culture understanding through retrieval.<n>RAVENEA focuses on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC)<n>We train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art vision-language models.
  arXiv  Detail & Related papers  (2025-05-20T14:57:16Z)
• Deconstructing Bias: A Multifaceted Framework for Diagnosing Cultural and Compositional Inequities in Text-to-Image Generative Models [3.6335172274433414]
  This paper benchmarks the Component Inclusion Score (CIS), a metric designed to evaluate the fidelity of image generation across cultural contexts.<n>We quantify biases in terms of compositional fragility and contextual misalignment, revealing significant performance gaps between Western and non-Western cultural prompts.
  arXiv  Detail & Related papers  (2025-04-05T06:17:43Z)
• Towards Automatic Evaluation for Image Transcreation [52.71090829502756]
  We propose a suite of automatic evaluation metrics inspired by machine translation (MT) metrics.<n>We identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity.<n>Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity.
  arXiv  Detail & Related papers  (2024-12-18T10:55:58Z)
• On the Cultural Gap in Text-to-Image Generation [75.69755281031951]
  One challenge in text-to-image (T2I) generation is the inadvertent reflection of culture gaps present in the training data. There is no benchmark to systematically evaluate a T2I model's ability to generate cross-cultural images. We propose a Challenging Cross-Cultural (C3) benchmark with comprehensive evaluation criteria, which can assess how well-suited a model is to a target culture.
  arXiv  Detail & Related papers  (2023-07-06T13:17:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.