A Benchmark for Zero-Shot Belief Inference in Large Language Models

• URL: http://arxiv.org/abs/2511.18616v1
• Date: Sun, 23 Nov 2025 21:13:20 GMT
• Title: A Benchmark for Zero-Shot Belief Inference in Large Language Models
• Authors: Joseph Malone, Rachith Aiyappa, Byunghwee Lee, Haewoon Kwak, Jisun An, Yong-Yeol Ahn,
• Abstract summary: We introduce a benchmark that evaluates the ability of large language models to predict individuals' stances on a wide range of topics.<n>We find that providing more background information about an individual improves predictive accuracy, but performance varies substantially across belief domains.
• Score: 3.669506952334741
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Beliefs are central to how humans reason, communicate, and form social connections, yet most computational approaches to studying them remain confined to narrow sociopolitical contexts and rely on fine-tuning for optimal performance. Despite the growing use of large language models (LLMs) across disciplines, how well these systems generalize across diverse belief domains remains unclear. We introduce a systematic, reproducible benchmark that evaluates the ability of LLMs to predict individuals' stances on a wide range of topics in a zero-shot setting using data from an online debate platform. The benchmark includes multiple informational conditions that isolate the contribution of demographic context and known prior beliefs to predictive success. Across several small- to medium-sized models, we find that providing more background information about an individual improves predictive accuracy, but performance varies substantially across belief domains. These findings reveal both the capacity and limitations of current LLMs to emulate human reasoning, advancing the study of machine behavior and offering a scalable framework for modeling belief systems beyond the sociopolitical sphere.

Related papers

• HumanLLM: Towards Personalized Understanding and Simulation of Human Nature [72.55730315685837]
  HumanLLM is a foundation model designed for personalized understanding and simulation of individuals.<n>We first construct the Cognitive Genome, a large-scale corpus curated from real-world user data on platforms like Reddit, Twitter, Blogger, and Amazon.<n>We then formulate diverse learning tasks and perform supervised fine-tuning to empower the model to predict a wide range of individualized human behaviors, thoughts, and experiences.
  arXiv  Detail & Related papers  (2026-01-22T09:27:27Z)
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs [8.00888290370075]
  It remains unclear which of these ingredients are most crucial for building models that develop human-like representations.<n>Most current benchmarks are not suited to measuring representational alignment between humans and models.
  arXiv  Detail & Related papers  (2025-10-01T15:37:19Z)
• Towards Safer AI Moderation: Evaluating LLM Moderators Through a Unified Benchmark Dataset and Advocating a Human-First Approach [0.9147875523270338]
  Large Language Models (LLMs) have demonstrated remarkable capabilities, surpassing earlier models in complexity and performance.<n>They struggle with detecting implicit hate, offensive language, and gender biases due to the subjective and context-dependent nature of these issues.<n>We develop an experimental framework based on state-of-the-art (SOTA) models to assess human emotions and offensive behaviors.
  arXiv  Detail & Related papers  (2025-08-09T18:00:27Z)
• Using cognitive models to reveal value trade-offs in language models [12.178109894945981]
  We use a cognitive model of polite speech to evaluate value trade-offs in two encompassing model settings.<n>Our results highlight patterns of higher informational utility than social utility in reasoning models' default behavior.<n>Our framework offers a flexible tool for probing value trade-offs across diverse model types.
  arXiv  Detail & Related papers  (2025-06-25T17:58:12Z)
• Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge [1.1666234644810893]
  Small models outperform larger ones in safety, suggesting that training and architecture may matter more than scale.<n>No model is fully robust to adversarial elicitation, with jailbreak attacks using low-resource languages or refusal suppression proving effective.
  arXiv  Detail & Related papers  (2025-04-10T16:00:59Z)
• LLM Generated Persona is a Promise with a Catch [18.45442859688198]
  Persona-based simulations hold promise for transforming disciplines that rely on population-level feedback.<n>Traditional methods to collect realistic persona data face challenges.<n>They are prohibitively expensive and logistically challenging due to privacy constraints.
  arXiv  Detail & Related papers  (2025-03-18T03:11:27Z)
• An Overview of Large Language Models for Statisticians [109.38601458831545]
  Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence (AI)<n>This paper explores potential areas where statisticians can make important contributions to the development of LLMs.<n>We focus on issues such as uncertainty quantification, interpretability, fairness, privacy, watermarking and model adaptation.
  arXiv  Detail & Related papers  (2025-02-25T03:40:36Z)
• Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning [88.68573198200698]
  We introduce ExploreToM, the first framework to allow large-scale generation of diverse and challenging theory of mind data.<n>Our approach leverages an A* search over a custom domain-specific language to produce complex story structures and novel, diverse, yet plausible scenarios.<n>Our evaluation reveals that state-of-the-art LLMs, such as Llama-3.1-70B and GPT-4o, show accuracies as low as 0% and 9% on ExploreToM-generated data.
  arXiv  Detail & Related papers  (2024-12-12T21:29:00Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context.<n>We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters.<n>We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Designing Domain-Specific Large Language Models: The Critical Role of Fine-Tuning in Public Opinion Simulation [0.0]
  This paper introduces a novel fine-tuning approach that integrates socio-demographic data from the UK Household Longitudinal Study.<n>By emulating diverse synthetic profiles, the fine-tuned models significantly outperform pre-trained counterparts.<n>Its broader implications include deploying LLMs in domains like healthcare and education, fostering inclusive and data-driven decision-making.
  arXiv  Detail & Related papers  (2024-09-28T10:39:23Z)
• Social Debiasing for Fair Multi-modal LLMs [59.61512883471714]
  Multi-modal Large Language Models (MLLMs) have dramatically advanced the research field and delivered powerful vision-language understanding capabilities.<n>These models often inherit deep-rooted social biases from their training data, leading to uncomfortable responses with respect to attributes such as race and gender.<n>This paper addresses the issue of social biases in MLLMs by introducing a comprehensive counterfactual dataset with multiple social concepts.
  arXiv  Detail & Related papers  (2024-08-13T02:08:32Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.