NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures

• URL: http://arxiv.org/abs/2609.06831v1
• Date: Sun, 06 Sep 2026 21:02:47 GMT
• Title: NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures
• Abstract summary: We introduce a benchmark of 3,268 contrastive image pairs spanning 16 countries.<n>Each pair varies only in the culturally relevant behavior that alters how each image is interpreted.<n>We introduce NormViz-Train, a training dataset of 64k images paired with explanations.
• Score: 58.53196987370368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems are used worldwide, but they struggle to serve the needs of culturally diverse populations. Prior work on cultural understanding evaluates AI systems on text-only settings or on visual artifact recognition (e.g. foods, clothing). The ability to reason about visually observable behaviors through local social norms, which we call visual norm understanding, remains unexamined. We introduce NormViz-Bench, a high quality, human-validated benchmark of 3,268 contrastive image pairs (6,536 images) spanning 16 countries. Each pair varies only in the culturally relevant behavior (e.g., objects, attributes, spatial relations, and actions) that alters how each image is interpreted. Each image is labeled as conforming to, violating, or irrelevant to local social norms, and pair-level evaluation requires both images to be correctly classified, thereby preventing reliance on superficial visual shortcuts. Even the strongest VLMs, Gemini 3.0 Flash and Qwen2.5 VL 7B, succeed on only 26.6% and 21.6% of pairs, struggling most with identifying violating and culturally benign visual behaviors. Towards bridging this, we introduce NormViz-Train, a training dataset of 64k images paired with explanations. Though absolute performance remains low (<30%), finetuning on NormViz-Train improves pair accuracy relatively by up to 125% and 36% Qwen3-VL 4B and 8B respectively, showing a path forward to teach models to connect visual perception to cultural significance. Together, NormViz-Bench and NormViz-Train establish visual norm understanding as a challenging and consequential frontier for multimodal AI.

Related papers

• CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries [42.15271207948674]
  We introduce CCBENCH, a framework for evaluating cultural competency in large language models (LLMs)<n>As a case study on health, we create CCBENCH-Health, which includes 60 theoretically grounded personas exhibiting varied norm-adherence states across six cultures.<n> Benchmarking five leading models reveals that even the best achieve culturally appropriate responses only 20-30% of the time.
  arXiv  Detail & Related papers  (2026-06-08T22:52:34Z)
• Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models [14.992895369883504]
  Prior work has examined cultural bias mainly in text-to-image (T2I) systems.<n>We bridge this gap with a unified evaluation across six countries.<n>We derive cross-country, cross-era, and cross-category evaluations.
  arXiv  Detail & Related papers  (2025-10-22T21:42:59Z)
• VideoNorms: Benchmarking Cultural Awareness of Video Language Models [19.29068943180369]
  We introduce VideoNorms, a benchmark of over 1000 (video clip, norm) pairs from US and Chinese cultures.<n>We use a human-AI collaboration framework, where a teacher model using theoretically-grounded prompting provides candidate annotations.<n>We benchmark a variety of open-weight VideoLLMs on the new dataset which highlight several common trends.
  arXiv  Detail & Related papers  (2025-10-09T17:54:55Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluation [61.130639734982395]
  We introduce CAIRe, a novel evaluation metric that assesses the degree of cultural relevance of an image.<n>Our framework grounds entities and concepts in the image to a knowledge base and uses factual information to give independent graded judgments for each culture label.
  arXiv  Detail & Related papers  (2025-06-10T17:16:23Z)
• Multimodal Cultural Safety: Evaluation Frameworks and Alignment Strategies [58.88053690412802]
  Large vision-language models (LVLMs) are increasingly deployed in globally distributed applications, such as tourism assistants.<n> CROSS is a benchmark designed to assess the cultural safety reasoning capabilities of LVLMs.<n>We evaluate 21 leading LVLMs, including mixture-of-experts models and reasoning models.
  arXiv  Detail & Related papers  (2025-05-20T23:20:38Z)
• RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding [79.44246283490665]
  We introduce RAVENEA, a new benchmark designed to advance visual culture understanding through retrieval.<n>RAVENEA focuses on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC)<n>We train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art vision-language models.
  arXiv  Detail & Related papers  (2025-05-20T14:57:16Z)
• Probing Visual Language Priors in VLMs [51.016683265437536]
  We introduce ViLP, a benchmark featuring deliberately out-of-distribution images.<n>Each question in ViLP is coupled with three potential answers and three corresponding images.<n>We propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training.
  arXiv  Detail & Related papers  (2024-12-31T17:54:29Z)
• On the Cultural Gap in Text-to-Image Generation [75.69755281031951]
  One challenge in text-to-image (T2I) generation is the inadvertent reflection of culture gaps present in the training data. There is no benchmark to systematically evaluate a T2I model's ability to generate cross-cultural images. We propose a Challenging Cross-Cultural (C3) benchmark with comprehensive evaluation criteria, which can assess how well-suited a model is to a target culture.
  arXiv  Detail & Related papers  (2023-07-06T13:17:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.