Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics

• URL: http://arxiv.org/abs/2604.02406v1
• Date: Thu, 02 Apr 2026 17:17:12 GMT
• Title: Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics
• Abstract summary: Measurement essential to improving AI performance and mitigating harms for marginalized groups.<n>Drive to automate measurement can be in tension with the ability for measurement instruments to capture expertise and perspectives of communities impacted by AI.<n>Recent work advocates for breaking measurement into several key stages: first moving from an abstract concept to be measured into a precise, "systematized" concept.
• Score: 18.0561997964007
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Measurement is essential to improving AI performance and mitigating harms for marginalized groups. As generative AI systems are rapidly deployed across geographies and contexts, AI measurement practices must be designed to support repeatable, automatable application across different models, datasets, and evaluation settings. But the drive to automate measurement can be in tension with the ability for measurement instruments to capture the expertise and perspectives of communities impacted by AI. Recent work advocates for breaking measurement into several key stages: first moving from an abstract concept to be measured into a precise, "systematized" concept; next operationalizing the systematized concept into a concrete measurement instrument; and finally applying the measurement instrument on data to produce measurements. This opens up an opportunity to concentrate community engagement in the systematization phase before operationalizing and applying measurement instruments. In this paper, we explore how to involve communities in systematizing the concept of "cultural appropriateness" in text-to-image models' representation of culturally significant artifacts through case studies with three communities: blind and low vision individuals residing in the UK, residents of Kerala, and residents of Tamil Nadu. Our systematized concepts reflect community members' lived experiences interacting with each artifact and how they want their material culture to be depicted, demonstrating the value of community involvement in defining valid measures. We explore how these systematized concepts can be operationalized into automated measurement instruments that could be applied using a multimodal LLM-as-a-judge approach and challenges that remain. We reflect on the benefits and limitations of such approaches.

Related papers

• Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge [78.35388859345056]
  We argue that the ML community would benefit from learning from and drawing on the social sciences when developing measurement instruments for evaluating GenAI systems.<n>We present a four-level framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, behaviors, and impacts of GenAI systems.
  arXiv  Detail & Related papers  (2025-02-01T21:09:51Z)
• Towards Automatic Evaluation for Image Transcreation [52.71090829502756]
  We propose a suite of automatic evaluation metrics inspired by machine translation (MT) metrics.<n>We identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity.<n>Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity.
  arXiv  Detail & Related papers  (2024-12-18T10:55:58Z)
• A Shared Standard for Valid Measurement of Generative AI Systems' Capabilities, Risks, and Impacts [38.66213773948168]
  The valid measurement of generative AI (GenAI) systems' capabilities, risks, and impacts forms the bedrock of our ability to evaluate these systems.<n>We introduce a shared standard for valid measurement that helps place many of the disparate-seeming evaluation practices in use today on a common footing.
  arXiv  Detail & Related papers  (2024-12-02T19:50:00Z)
• Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems [88.35461485731162]
  We identify four types of challenges that prevent practitioners from effectively using publicly available instruments for measuring representational harms. Our goal is to advance the development of instruments for measuring representational harms that are well-suited to practitioner needs.
  arXiv  Detail & Related papers  (2024-11-23T22:13:38Z)
• Evaluating Generative AI Systems is a Social Science Measurement Challenge [78.35388859345056]
  We present a framework for measuring concepts related to the capabilities, impacts, opportunities, and risks of GenAI systems. The framework distinguishes between four levels: the background concept, the systematized concept, the measurement instrument(s), and the instance-level measurements themselves.
  arXiv  Detail & Related papers  (2024-11-17T02:35:30Z)
• Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
  Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
  arXiv  Detail & Related papers  (2024-07-22T17:52:12Z)
• The Cadaver in the Machine: The Social Practices of Measurement and Validation in Motion Capture Technology [1.1650821883155187]
  We argue that the measurement of bodies and the validation of measurements for motion capture systems can be understood as social practices. We show how these practices become ingrained in motion capture design and innovation over time. We suggest that social practices of measurement and validation are ubiquitous in the development of data- and sensor-driven systems.
  arXiv  Detail & Related papers  (2024-01-19T18:41:53Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Towards Fair and Explainable AI using a Human-Centered AI Approach [5.888646114353372]
  We present 5 research projects that aim to enhance explainability and fairness in classification systems and word embeddings. The first project explores the utility/downsides of introducing local model explanations as interfaces for machine teachers. The second project presents D-BIAS, a causality-based human-in-the-loop visual tool for identifying and mitigating social biases in datasets. The third project presents WordBias, a visual interactive tool that helps audit pre-trained static word embeddings for biases against groups. The fourth project presents DramatVis Personae, a visual analytics tool that helps identify social
  arXiv  Detail & Related papers  (2023-06-12T21:08:55Z)
• Assessing the State of Self-Supervised Human Activity Recognition using Wearables [6.777825307593778]
  Self-supervised learning in the field of wearables-based human activity recognition (HAR) Self-supervised methods enable a host of new application domains such as, for example, domain adaptation and transfer across sensor positions, activities etc.
  arXiv  Detail & Related papers  (2022-02-22T02:21:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.