Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates

• URL: http://arxiv.org/abs/2606.03029v1
• Date: Tue, 02 Jun 2026 02:07:46 GMT
• Title: Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates
• Abstract summary: We introduce conditional hypothesis generation to steer hypothesis discovery toward differences within relevant subgroups.<n>Two challenges arise: the target subgroup may be underrepresented, and the direction of a difference may reverse across subgroups.<n>We propose two econometrics-inspired methods to detect sign reversals, and the other applies within-stratum demeaning and inverse-frequency reweighting to equalize underrepresented strata.
• Score: 11.185598864391103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A core goal of computational social science is to discover interpretable differences in how language varies across outcomes of interest, such as political affiliation or instructional quality. Recent LLM-based hypothesis generation methods describe such differences in natural language, but select for globally discriminative patterns without accounting for covariates that shape the data based on researchers' domain knowledge. When covariates are ignored, selected patterns can reflect confounds rather than differences of substantive interest. We introduce conditional hypothesis generation, a framework that incorporates researcher-specified covariates to steer hypothesis discovery toward differences that hold within relevant subgroups. Two challenges arise: the target subgroup may be underrepresented (stratum imbalance), and the direction of a difference may reverse across subgroups (sign reversal). We propose two econometrics-inspired methods: one introduces feature--covariate interactions to detect sign reversals, and the other applies within-stratum demeaning and inverse-frequency reweighting to equalize underrepresented strata. Synthetic experiments show each method outperforms global baselines in its targeted setting, and expert evaluation on two real-world datasets confirms that covariate-aware generation surfaces more useful hypotheses within relevant subgroups.

Related papers

• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Differential Subgroup Discovery: Characterizing Where Two Populations Differ, and Why [36.94047915240195]
  We study the problem of understanding where two populations differ within a feature space.<n> Differential subgroups reveal the regions of the feature space where population-level gaps are most pronounced.
  arXiv  Detail & Related papers  (2026-04-30T11:31:42Z)
• Can I guess where you are from? Modeling dialectal morphosyntactic similarities in Brazilian Portuguese [0.0]
  This paper investigates morphosyntactic covariation in Brazilian Portuguese (BP) to assess whether dialectal origin can be inferred from the combined behavior of linguistic variables.<n> Focusing on four grammatical phenomena related to pronouns, correlation and clustering methods are applied to model covariation and dialectal distribution.
  arXiv  Detail & Related papers  (2026-03-21T07:40:17Z)
• Beyond Interaction Effects: Two Logics for Studying Population Inequalities [2.710807780228189]
  We show that the choice between deduction and induction reflects a tradeoff between interpretability and flexibility.<n>Our framework is particularly relevant for inequality research, where understanding how treatment effects vary across social subpopulation is substantively central.
  arXiv  Detail & Related papers  (2025-12-26T19:25:39Z)
• Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective [24.54292750583169]
  Large Language Models (LLMs) often generate responses with inherent biases, undermining their reliability in real-world applications.<n>We propose FiSCo (Fine-grained Semantic Comparison), a novel statistical framework to evaluate group-level fairness in LLMs.<n>We decompose model outputs into semantically distinct claims and apply statistical hypothesis testing to compare inter- and intra-group similarities.
  arXiv  Detail & Related papers  (2025-06-23T18:31:22Z)
• On Reference (In-)Determinacy in Natural Language Inference [62.904689974282334]
  We revisit the reference determinacy (RD) assumption in the task of natural language inference (NLI)<n>We observe that current NLI models fail in downstream applications such as fact verification, where the input premise and hypothesis may refer to different contexts.<n>We introduce RefNLI, a diagnostic benchmark for identifying reference ambiguity in NLI examples.
  arXiv  Detail & Related papers  (2025-02-09T06:58:13Z)
• Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? [90.35044668396591]
  A recurring theme in machine learning is algorithmic monoculture: the same systems, or systems that share components, are deployed by multiple decision-makers. We propose the component-sharing hypothesis: if decision-makers share components like training data or specific models, then they will produce more homogeneous outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that sharing training data reliably exacerbates homogenization. We conclude with philosophical analyses of and societal challenges for outcome homogenization, with an eye towards implications for deployed machine learning systems.
  arXiv  Detail & Related papers  (2022-11-25T09:33:11Z)
• Modelling Heterogeneity Using Bayesian Structured Sparsity [0.0]
  How to estimate the effect of some variable differing across observations is a key question in political science. This paper allows a common way of simplifying complex phenomenon (placing observations with similar effects into discrete groups) to be integrated into regression analysis.
  arXiv  Detail & Related papers  (2021-03-29T19:54:25Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Learning Causal Semantic Representation for Out-of-Distribution Prediction [125.38836464226092]
  We propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately. We show that CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error.
  arXiv  Detail & Related papers  (2020-11-03T13:16:05Z)
• LOGAN: Local Group Bias Detection by Clustering [86.38331353310114]
  We argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. We propose LOGAN, a new bias detection technique based on clustering. Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region.
  arXiv  Detail & Related papers  (2020-10-06T16:42:51Z)
• L2R2: Leveraging Ranking for Abductive Reasoning [65.40375542988416]
  The abductive natural language inference task ($alpha$NLI) is proposed to evaluate the abductive reasoning ability of a learning system. A novel $L2R2$ approach is proposed under the learning-to-rank framework. Experiments on the ART dataset reach the state-of-the-art in the public leaderboard.
  arXiv  Detail & Related papers  (2020-05-22T15:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.