The Qualitative Laboratory: Theory Prototyping and Hypothesis Generation with Large Language Models

• URL: http://arxiv.org/abs/2601.00797v1
• Date: Tue, 25 Nov 2025 08:31:48 GMT
• Title: The Qualitative Laboratory: Theory Prototyping and Hypothesis Generation with Large Language Models
• Authors: Hugues Draelants,
• Abstract summary: We argue that for this specific task, persona simulation offers a distinct advantage over established methods.<n>By generating naturalistic discourse, it overcomes the lack of discursive depth common in vignette surveys.<n>We present a protocol where personas derived from a sociological theory of climate reception react to policy messages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: A central challenge in social science is to generate rich qualitative hypotheses about how diverse social groups might interpret new information. This article introduces and illustrates a novel methodological approach for this purpose: sociological persona simulation using Large Language Models (LLMs), which we frame as a "qualitative laboratory". We argue that for this specific task, persona simulation offers a distinct advantage over established methods. By generating naturalistic discourse, it overcomes the lack of discursive depth common in vignette surveys, and by operationalizing complex worldviews through natural language, it bypasses the formalization bottleneck of rule-based agent-based models (ABMs). To demonstrate this potential, we present a protocol where personas derived from a sociological theory of climate reception react to policy messages. The simulation produced nuanced and counter-intuitive hypotheses - such as a conservative persona's rejection of a national security frame - that challenge theoretical assumptions. We conclude that this method, used as part of a "simulation then validation" workflow, represents a superior tool for generating deeply textured hypotheses for subsequent empirical testing.

Related papers

• On Emergent Social World Models -- Evidence for Functional Integration of Theory of Mind and Pragmatic Reasoning in Language Models [4.5373666852176715]
  This paper investigates whether LMs recruit shared computational mechanisms for general Theory of Mind (ToM) and language-specific pragmatic reasoning.<n>We analyze LMs' performance across seven subcategories of ToM abilities on a substantially larger localizer dataset.<n>Results from stringent hypothesis-driven statistical testing offer suggestive evidence for the functional integration hypothesis.
  arXiv  Detail & Related papers  (2026-02-10T21:12:12Z)
• MASim: Multilingual Agent-Based Simulation for Social Science [68.04129327237963]
  Multi-agent role-playing has recently shown promise for studying social behavior with language agents.<n>Existing simulations are mostly monolingual and fail to model cross-lingual interaction.<n>We introduce MASim, the first multilingual agent-based simulation framework.
  arXiv  Detail & Related papers  (2025-12-08T06:12:48Z)
• Large language models replicate and predict human cooperation across experiments in game theory [0.8166364251367626]
  How closely large language models mirror actual human decision-making remains poorly understood.<n>We develop a digital twin of game-theoretic experiments and introduce a systematic prompting and probing framework for machine-behavioral evaluation.<n>We find that Llama reproduces human cooperation patterns with high fidelity, capturing human deviations from rational choice theory.
  arXiv  Detail & Related papers  (2025-11-06T16:21:27Z)
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies [46.27915760967977]
  We find that many recent studies adopt experimental designs that systematically undermine the validity of their claims.<n>From a survey of over 40 papers, we identify six recurring methodological flaws.<n>We formalize these six requirements as the PIMMUR principles and argue they are necessary conditions for credible LLM-based social simulation.
  arXiv  Detail & Related papers  (2025-09-22T17:27:29Z)
• Population-Aligned Persona Generation for LLM-based Social Simulation [58.84363795421489]
  We propose a systematic framework for synthesizing high-quality, population-aligned persona sets for social simulation.<n>Our approach begins by leveraging large language models to generate narrative personas from long-term social media data.<n>To address the needs of specific simulation contexts, we introduce a task-specific module that adapts the globally aligned persona set to targeted subpopulations.
  arXiv  Detail & Related papers  (2025-09-12T10:43:47Z)
• Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models [93.1043186636177]
  We explore the hypothesis that people use a combination of distributed and symbolic representations to construct bespoke mental models tailored to novel situations.<n>We propose a computational implementation of this idea -- a Model Synthesis Architecture''<n>We evaluate our MSA as a model of human judgments on a novel reasoning dataset.
  arXiv  Detail & Related papers  (2025-07-16T18:01:03Z)
• Hypothesis Testing for Quantifying LLM-Human Misalignment in Multiple Choice Settings [7.284860523651357]
  We assess the misalignment between Large Language Models (LLMs)-simulated and actual human behaviors in multiple-choice survey settings.<n>We apply this framework to a popular language model for simulating people's opinions in various public surveys.<n>This raises questions about the alignment of this language model with the tested populations.
  arXiv  Detail & Related papers  (2025-06-17T22:04:55Z)
• MOOSE-Chem3: Toward Experiment-Guided Hypothesis Ranking via Simulated Experimental Feedback [136.27567671480156]
  We introduce experiment-guided ranking, which prioritizes hypotheses based on feedback from prior tests.<n>We frame experiment-guided ranking as a sequential decision-making problem.<n>Our approach significantly outperforms pre-experiment baselines and strong ablations.
  arXiv  Detail & Related papers  (2025-05-23T13:24:50Z)
• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage.<n>Models may behave unreliably due to poorly explored failure modes.<n> causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
  This work proposes the first dataset for social science academic hypotheses discovery. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
  arXiv  Detail & Related papers  (2023-09-06T05:19:41Z)
• Simulation as Experiment: An Empirical Critique of Simulation Research on Recommender Systems [4.006331916849688]
  We argue that simulation studies of recommender system (RS) evolution are conceptually similar to empirical experimental approaches. By adopting standards and practices common in empirical disciplines, simulation researchers can mitigate many of these weaknesses.
  arXiv  Detail & Related papers  (2021-07-29T21:05:01Z)
• Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference [5.283529004179579]
  Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) is the task of predicting the entailment relation between a pair of sentences. Models that understand entailment should encode both, the premise and the hypothesis. Experiments by Poliak et al. revealed a strong preference of these models towards patterns observed only in the hypothesis.
  arXiv  Detail & Related papers  (2021-01-19T01:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.