Language, Caste, and Context: Demographic Disparities in AI-Generated Explanations Across Indian and American STEM Educational Systems

• URL: http://arxiv.org/abs/2601.14506v1
• Date: Tue, 20 Jan 2026 21:58:45 GMT
• Title: Language, Caste, and Context: Demographic Disparities in AI-Generated Explanations Across Indian and American STEM Educational Systems
• Abstract summary: This paper asks how do LLMs perceive the intellectual capabilities of student profiles from intersecting marginalized identities across different cultural contexts?<n>We construct profiles combining caste, medium of instruction, and school boards in India, and race, HBCU attendance, and school type in America, alongside universal factors like income and college tier.<n>We observe biases providing lower-quality outputs to profiles with marginalized backgrounds in both contexts.
• Score: 2.0722928522838693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The popularization of AI chatbot usage globally has created opportunities for research into their benefits and drawbacks, especially for students using AI assistants for coursework support. This paper asks: how do LLMs perceive the intellectual capabilities of student profiles from intersecting marginalized identities across different cultural contexts? We conduct one of the first large-scale intersectional analyses on LLM explanation quality for Indian and American undergraduate profiles preparing for engineering entrance examinations. By constructing profiles combining multiple demographic dimensions including caste, medium of instruction, and school boards in India, and race, HBCU attendance, and school type in America, alongside universal factors like income and college tier, we examine how quality varies across these factors. We observe biases providing lower-quality outputs to profiles with marginalized backgrounds in both contexts. LLMs such as Qwen2.5-32B-Instruct and GPT-4o demonstrate granular understandings of context-specific discrimination, systematically providing simpler explanations to Hindi/Regional-medium students in India and HBCU profiles in America, treating these as proxies for lower capability. Even when marginalized profiles attain social mobility by getting accepted into elite institutions, they still receive more simplistic explanations, showing how demographic information is inextricably linked to LLM biases. Different models (Qwen2.5-32B-Instruct, GPT-4o, GPT-4o-mini, GPT-OSS 20B) embed similar biases against historically marginalized populations in both contexts, preventing profiles from switching between AI assistants for better results. Our findings have strong implications for AI incorporation into global engineering education.

Related papers

• Sociodemographic Biases in Educational Counselling by Large Language Models [61.30199414932204]
  This study examines sociodemographic biases in Large Language Models (LLMs)<n>We evaluate responses from six LLMs answering questions about 900 vignettes describing students in diverse circumstances.
  arXiv  Detail & Related papers  (2026-04-03T14:08:22Z)
• Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models [2.48490797934472]
  Artificial Intelligence (AI) is increasingly used in hiring, with large language models (LLMs) having the potential to influence or even make hiring decisions.<n>This raises pressing concerns about bias, fairness, and trust, particularly across diverse cultural contexts.<n>We conduct a systematic analysis of how LLMs assess job interviews across cultural and identity dimensions.
  arXiv  Detail & Related papers  (2025-08-21T07:45:00Z)
• Addressing Bias in LLMs: Strategies and Application to Fair AI-based Recruitment [49.81946749379338]
  This work seeks to analyze the capacity of Transformers-based systems to learn demographic biases present in the data.<n>We propose a privacy-enhancing framework to reduce gender information from the learning pipeline as a way to mitigate biased behaviors in the final tools.
  arXiv  Detail & Related papers  (2025-06-13T15:29:43Z)
• What Does a Software Engineer Look Like? Exploring Societal Stereotypes in LLMs [9.007321855123882]
  This study investigates how OpenAI's GPT-4 and Microsoft Copilot can reinforce gender and racial stereotypes.<n>We used each LLM to generate 300 profiles, consisting of 100 gender-based and 50 gender-neutral profiles.<n>Our analysis reveals that both models preferred male and Caucasian profiles, particularly for senior roles.
  arXiv  Detail & Related papers  (2025-01-07T06:44:41Z)
• Embracing AI in Education: Understanding the Surge in Large Language Model Use by Secondary Students [53.20318273452059]
  Large language models (LLMs) like OpenAI's ChatGPT have opened up new avenues in education.<n>Despite school restrictions, our survey of over 300 middle and high school students revealed that a remarkable 70% of students have utilized LLMs.<n>We propose a few ideas to address such issues, including subject-specific models, personalized learning, and AI classrooms.
  arXiv  Detail & Related papers  (2024-11-27T19:19:34Z)
• Large Language Models Reflect the Ideology of their Creators [71.65505524599888]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language.<n>This paper shows that the ideological stance of an LLM appears to reflect the worldview of its creators.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMs [58.27353205269664]
  Social biases can manifest in language agency in Large Language Model (LLM)-generated content.<n>We introduce the Language Agency Bias Evaluation benchmark, which comprehensively evaluates biases in LLMs.<n>Using LABE, we unveil language agency social biases in 3 recent LLMs: ChatGPT, Llama3, and Mistral.
  arXiv  Detail & Related papers  (2024-04-16T12:27:54Z)
• Laissez-Faire Harms: Algorithmic Biases in Generative Language Models [0.0]
  We show that synthetically generated texts from five of the most pervasive LMs perpetuate harms of omission, subordination, and stereotyping for minoritized individuals. We find widespread evidence of bias to an extent that such individuals are hundreds to thousands of times more likely to encounter LM-generated outputs. Our findings highlight the urgent need to protect consumers from discriminatory harms caused by language models.
  arXiv  Detail & Related papers  (2024-04-11T05:09:03Z)
• Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs [13.744746481528711]
  Large Language Models (LLMs) are widely used to simulate human responses across diverse contexts.<n>We evaluate nine popular LLMs on their ability to understand demographic differences in two subjective judgment tasks: politeness and offensiveness.<n>We find that in zero-shot settings, most models' predictions for both tasks align more closely with labels from White participants than those from Asian or Black participants.
  arXiv  Detail & Related papers  (2023-11-16T10:02:24Z)
• Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs [67.51906565969227]
  We study the unintended side-effects of persona assignment on the ability of LLMs to perform basic reasoning tasks. Our study covers 24 reasoning datasets, 4 LLMs, and 19 diverse personas (e.g. an Asian person) spanning 5 socio-demographic groups.
  arXiv  Detail & Related papers  (2023-11-08T18:52:17Z)
• Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models [0.0]
  This paper investigates bias along less-studied but still consequential, dimensions, such as age and beauty. We ask whether LLMs hold wide-reaching biases of positive or negative sentiment for specific social groups similar to the "what is beautiful is good" bias found in people in experimental psychology.
  arXiv  Detail & Related papers  (2023-09-16T07:07:04Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.