Analysis of LLM Bias (Chinese Propaganda & Anti-US Sentiment) in DeepSeek-R1 vs. ChatGPT o3-mini-high

• URL: http://arxiv.org/abs/2506.01814v1
• Date: Mon, 02 Jun 2025 15:54:06 GMT
• Title: Analysis of LLM Bias (Chinese Propaganda & Anti-US Sentiment) in DeepSeek-R1 vs. ChatGPT o3-mini-high
• Authors: PeiHsuan Huang, ZihWei Lin, Simon Imbot, WenCheng Fu, Ethan Tu,
• Abstract summary: DeepSeek-R1 consistently exhibited substantially higher proportions of both propaganda and anti-U.S. sentiment.<n>These biases were not confined to overtly political topics but also permeated cultural and lifestyle content.
• Score: 0.40329768057075643
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) increasingly shape public understanding and civic decisions, yet their ideological neutrality is a growing concern. While existing research has explored various forms of LLM bias, a direct, cross-lingual comparison of models with differing geopolitical alignments-specifically a PRC-system model versus a non-PRC counterpart-has been lacking. This study addresses this gap by systematically evaluating DeepSeek-R1 (PRC-aligned) against ChatGPT o3-mini-high (non-PRC) for Chinese-state propaganda and anti-U.S. sentiment. We developed a novel corpus of 1,200 de-contextualized, reasoning-oriented questions derived from Chinese-language news, presented in Simplified Chinese, Traditional Chinese, and English. Answers from both models (7,200 total) were assessed using a hybrid evaluation pipeline combining rubric-guided GPT-4o scoring with human annotation. Our findings reveal significant model-level and language-dependent biases. DeepSeek-R1 consistently exhibited substantially higher proportions of both propaganda and anti-U.S. bias compared to ChatGPT o3-mini-high, which remained largely free of anti-U.S. sentiment and showed lower propaganda levels. For DeepSeek-R1, Simplified Chinese queries elicited the highest bias rates; these diminished in Traditional Chinese and were nearly absent in English. Notably, DeepSeek-R1 occasionally responded in Simplified Chinese to Traditional Chinese queries and amplified existing PRC-aligned terms in its Chinese answers, demonstrating an "invisible loudspeaker" effect. Furthermore, such biases were not confined to overtly political topics but also permeated cultural and lifestyle content, particularly in DeepSeek-R1.

Related papers

• Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation [89.52571224447111]
  Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization.<n>We provide an LLM judge bias analysis as a function of overlap with human-written responses in the domain of summarization.
  arXiv  Detail & Related papers  (2026-02-07T19:39:28Z)
• Bilingual Bias in Large Language Models: A Taiwan Sovereignty Benchmark Study [0.0]
  Large Language Models (LLMs) are increasingly deployed in multilingual contexts, yet their consistency across languages on politically sensitive topics remains understudied.<n>This paper presents a systematic benchmark study examining how 17 LLMs respond to questions concerning the sovereignty of the Republic of China (Taiwan) when queried in Chinese versus English.<n>We discover significant language bias -- the phenomenon where the same model produces substantively different political stances depending on the query language.
  arXiv  Detail & Related papers  (2026-02-06T03:57:21Z)
• Is DeepSeek a New Voice Among LLMs in Public Opinion Simulation? [6.489711597270606]
  This study evaluates the ability of an open-source large language model (LLM) to simulate public opinions in comparison to models developed by tech companies.<n>Our findings indicate that DeepSeek-V3 performs best in simulating U.S. opinions on the abortion issue.<n>For Chinese samples, DeepSeek-V3 performs best in simulating opinions on foreign aid and individualism but shows limitations in modeling views on capitalism.
  arXiv  Detail & Related papers  (2025-06-17T19:19:14Z)
• Geopolitical biases in LLMs: what are the "good" and the "bad" countries according to contemporary language models [52.00270888041742]
  We introduce a novel dataset with neutral event descriptions and contrasting viewpoints from different countries.<n>Our findings show significant geopolitical biases, with models favoring specific national narratives.<n>Simple debiasing prompts had a limited effect on reducing these biases.
  arXiv  Detail & Related papers  (2025-06-07T10:45:17Z)
• Exploring Multimodal Challenges in Toxic Chinese Detection: Taxonomy, Benchmark, and Findings [48.841514684592426]
  We highlight the multimodal nature of Chinese language as a key challenge for deploying language models in toxic Chinese detection.<n>First, we propose a taxonomy of 3 perturbation strategies and 8 specific approaches in toxic Chinese content.<n>Then, we curate a dataset based on this taxonomy, and benchmark 9 SOTA LLMs (from both the US and China) to assess if they can detect perturbed toxic Chinese text.
  arXiv  Detail & Related papers  (2025-05-30T08:32:45Z)
• Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs [50.07451351559251]
  We present a study across five typologically distinct languages (English, Russian, German, Hindi, and Vietnamese)<n>We examine how position bias interacts with prompt strategies and affects output entropy.
  arXiv  Detail & Related papers  (2025-05-22T02:23:00Z)
• R1dacted: Investigating Local Censorship in DeepSeek's R1 Language Model [17.402774424821814]
  Reports suggest R1 refuses to answer certain prompts related to politically sensitive topics in China.<n>We introduce a large-scale set of heavily curated prompts that get censored by R1, but are not censored by other models.<n>We conduct a comprehensive analysis of R1's censorship patterns, examining their consistency, triggers, and variations across topics, prompt phrasing, and context.
  arXiv  Detail & Related papers  (2025-05-19T02:16:56Z)
• Mapping Geopolitical Bias in 11 Large Language Models: A Bilingual, Dual-Framing Analysis of U.S.-China Tensions [2.8202443616982884]
  This study systematically analyzes geopolitical bias across 11 prominent Large Language Models (LLMs)<n>We generated 19,712 prompts designed to detect ideological leanings in model outputs.<n>U.S.-based models predominantly favored Pro-U.S. stances, while Chinese-origin models exhibited pronounced Pro-China biases.
  arXiv  Detail & Related papers  (2025-03-31T03:38:17Z)
• White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMs [58.27353205269664]
  Social biases can manifest in language agency in Large Language Model (LLM)-generated content.<n>We introduce the Language Agency Bias Evaluation benchmark, which comprehensively evaluates biases in LLMs.<n>Using LABE, we unveil language agency social biases in 3 recent LLMs: ChatGPT, Llama3, and Mistral.
  arXiv  Detail & Related papers  (2024-04-16T12:27:54Z)
• Red AI? Inconsistent Responses from GPT3.5 Models on Political Issues in the US and China [13.583047010078648]
  This study investigates political biases in GPT's multilingual models. We posed the same question about political issues in the U.S. and China to GPT in both English and simplified Chinese. Our analysis of the bilingual responses revealed that GPT's bilingual models' political "knowledge" (content) and the political "attitude" (sentiment) are significantly more inconsistent on political issues in China.
  arXiv  Detail & Related papers  (2023-12-15T16:25:56Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models [52.25049362267279]
  We present a Chinese Bias Benchmark dataset that consists of over 100K questions jointly constructed by human experts and generative language models. The testing instances in the dataset are automatically derived from 3K+ high-quality templates manually authored with stringent quality control. Extensive experiments demonstrate the effectiveness of the dataset in detecting model bias, with all 10 publicly available Chinese large language models exhibiting strong bias in certain categories.
  arXiv  Detail & Related papers  (2023-06-28T14:14:44Z)
• CHBias: Bias Evaluation and Mitigation of Chinese Conversational Language Models [30.400023506841503]
  We introduce a new Chinese dataset, CHBias, for bias evaluation and mitigation of Chinese conversational language models. We evaluate two popular pretrained Chinese conversational models, CDial-GPT and EVA2.0, using CHBias.
  arXiv  Detail & Related papers  (2023-05-18T18:58:30Z)
• CORGI-PM: A Chinese Corpus For Gender Bias Probing and Mitigation [28.38578407487603]
  We propose a Chinese cOrpus foR Gender bIas Probing and Mitigation CORGI-PM, which contains 32.9k sentences with high-quality labels. We address three challenges for automatic textual gender bias mitigation, which requires the models to detect, classify, and mitigate textual gender bias. CORGI-PM is the first sentence-level Chinese corpus for gender bias probing and mitigation.
  arXiv  Detail & Related papers  (2023-01-01T12:48:12Z)
• COLD: A Benchmark for Chinese Offensive Language Detection [54.60909500459201]
  We use COLDataset, a Chinese offensive language dataset with 37k annotated sentences. We also propose textscCOLDetector to study output offensiveness of popular Chinese language models. Our resources and analyses are intended to help detoxify the Chinese online communities and evaluate the safety performance of generative language models.
  arXiv  Detail & Related papers  (2022-01-16T11:47:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.