Mathematics Isn't Culture-Free: Probing Cultural Gaps via Entity and Scenario Perturbations

• URL: http://arxiv.org/abs/2507.00883v2
• Date: Fri, 31 Oct 2025 06:03:55 GMT
• Title: Mathematics Isn't Culture-Free: Probing Cultural Gaps via Entity and Scenario Perturbations
• Abstract summary: We create culturally adapted variants of the GSM8K test set for five regions Africa, India, China, Korea, and Japan.<n>We evaluate six large language models (LLMs), ranging from 8B to 72B parameters, across five prompting strategies to assess their robustness to cultural variation in math problem presentation.
• Score: 40.515279067334596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although mathematics is often considered culturally neutral, the way mathematical problems are presented can carry implicit cultural context. Existing benchmarks like GSM8K are predominantly rooted in Western norms, including names, currencies, and everyday scenarios. In this work, we create culturally adapted variants of the GSM8K test set for five regions Africa, India, China, Korea, and Japan using prompt-based transformations followed by manual verification. We evaluate six large language models (LLMs), ranging from 8B to 72B parameters, across five prompting strategies to assess their robustness to cultural variation in math problem presentation. Our findings reveal a consistent performance gap: models perform best on the original US-centric dataset and comparatively worse on culturally adapted versions. However, models with reasoning capabilities are more resilient to these shifts, suggesting that deeper reasoning helps bridge cultural presentation gaps in mathematical tasks

Related papers

• CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia [72.20656906184539]
  We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue.<n>The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues.<n>Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment.
  arXiv  Detail & Related papers  (2026-08-28T14:56:46Z)
• CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries [42.15271207948674]
  We introduce CCBENCH, a framework for evaluating cultural competency in large language models (LLMs)<n>As a case study on health, we create CCBENCH-Health, which includes 60 theoretically grounded personas exhibiting varied norm-adherence states across six cultures.<n> Benchmarking five leading models reveals that even the best achieve culturally appropriate responses only 20-30% of the time.
  arXiv  Detail & Related papers  (2026-06-08T22:52:34Z)
• CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs [59.09971492475217]
  We introduce CultureForest, a benchmark for textitCultural Norm Grounded Reasoning<n>CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions.<n>Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings.
  arXiv  Detail & Related papers  (2026-06-01T08:25:12Z)
• From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation [16.809989616664605]
  We introduce CulT-Eval, a benchmark designed to evaluate how models handle different types of culturally grounded expressions.<n>CulT-Eval comprises over 7,959 carefully curated instances spanning multiple types of culturally grounded expressions.<n>We propose a complementary evaluation metric that targets culturally induced meaning deviations.
  arXiv  Detail & Related papers  (2026-03-18T02:59:44Z)
• Do Large Language Models Truly Understand Cross-cultural Differences? [53.481048019144644]
  We develop a scenario-based benchmark to evaluate large language models' cross-cultural understanding and reasoning.<n>Grounded in cultural theory, we categorize cross-cultural capabilities into nine dimensions.<n>The dataset supports continuous expansion, and experiments confirm its transferability to other languages.
  arXiv  Detail & Related papers  (2025-12-08T01:21:58Z)
• Culture in Action: Evaluating Text-to-Image Models through Social Activities [40.874302288116304]
  Text-to-image (T2I) models achieve impressive photorealism by training on large-scale web data, but models inherit cultural biases and fail to depict underrepresented regions faithfully.<n>We introduce CULTIVate, a benchmark for evaluating T2I models on cross-cultural activities.<n>We propose four metrics to measure cultural alignment, hallucination, exaggerated elements, and diversity.
  arXiv  Detail & Related papers  (2025-11-07T19:51:11Z)
• Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens [9.000522371422628]
  We introduce a four-part framework that categorizes how benchmarks frame culture.<n>We qualitatively examine 20 cultural benchmarks and identify six recurring methodological issues.<n>Our aim is to guide the development of cultural benchmarks that go beyond static recall tasks.
  arXiv  Detail & Related papers  (2025-10-07T13:42:44Z)
• Everyday Physics in Korean Contexts: A Culturally Grounded Physical Reasoning Benchmark [12.080497081698299]
  Existing physical commonsense reasoning benchmarks predominantly focus on Western contexts, overlooking cultural variations in physical problem-solving.<n>EPiK (Everyday Physics in Korean Contexts) is a novel benchmark comprising 181 binary-choice problems that test physical reasoning within Korean cultural contexts.<n>EPiK is constructed using a two-stage generation and verification pipeline to create culturally-authentic problems.
  arXiv  Detail & Related papers  (2025-09-22T14:01:14Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluation [61.130639734982395]
  We introduce CAIRe, a novel evaluation metric that assesses the degree of cultural relevance of an image.<n>Our framework grounds entities and concepts in the image to a knowledge base and uses factual information to give independent graded judgments for each culture label.
  arXiv  Detail & Related papers  (2025-06-10T17:16:23Z)
• CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation [25.213316704661352]
  We introduce CaMMT, a benchmark of over 5,800 triples of images along with parallel captions in English and regional languages.<n>We find that visual context generally improves translation quality, especially in handling Culturally-Specific Items (CSIs) and correct gender usage.
  arXiv  Detail & Related papers  (2025-05-30T10:42:44Z)
• Fluent but Culturally Distant: Can Regional Training Teach Cultural Understanding? [17.231806929840015]
  We evaluate five Indic and five global LLMs along two key dimensions: values and practices.<n>Across all four tasks, we find that Indic models do not align more closely with Indian cultural norms than global models.<n>We trace this failure to the scarcity of high-quality, untranslated, and culturally grounded pretraining and fine-tuning data.
  arXiv  Detail & Related papers  (2025-05-25T01:59:23Z)
• From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMs [57.43233760384488]
  Adapting cultural values in Large Language Models (LLMs) presents significant challenges.<n>Prior work primarily aligns LLMs with different cultural values using World Values Survey (WVS) data.<n>In this paper, we investigate WVS-based training for cultural value adaptation and find that relying solely on survey data cane cultural norms and interfere with factual knowledge.
  arXiv  Detail & Related papers  (2025-05-22T09:00:01Z)
• CAReDiO: Cultural Alignment of LLM via Representativeness and Distinctiveness Guided Data Optimization [50.90288681622152]
  Large Language Models (LLMs) more deeply integrate into human life across various regions.<n>Existing approaches develop culturally aligned LLMs through fine-tuning with culture-specific corpora.<n>We introduce CAReDiO, a novel cultural data construction framework.
  arXiv  Detail & Related papers  (2025-04-09T13:40:13Z)
• Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts? [0.7329170469548214]
  This study highlights the impact of cultural context on the mathematical reasoning abilities of Large Language Models (LLMs)<n>Our findings reveal that LLMs struggle with math problems when cultural references change, even though the underlying mathematical structure remains constant.<n>Even models with no explicit mathematical training but exposure to relevant cultural contexts sometimes outperform larger, mathematically proficient models on culturally embedded math problems.
  arXiv  Detail & Related papers  (2025-03-23T10:35:39Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.