Hypothesis Generation with Large Language Models

• URL: http://arxiv.org/abs/2404.04326v2
• Date: Fri, 23 Aug 2024 18:00:00 GMT
• Title: Hypothesis Generation with Large Language Models
• Authors: Yangqiaoyu Zhou, Haokun Liu, Tejes Srivastava, Hongyuan Mei, Chenhao Tan,
• Abstract summary: We focus on hypothesis generation based on data (i.e., labeled examples) Inspired by multi-armed bandits, we design a reward function to inform the exploitation-exploration tradeoff in the update process. Our algorithm is able to generate hypotheses that enable much better predictive performance than few-shot prompting in classification tasks.
• Score: 28.73562677221476
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Effective generation of novel hypotheses is instrumental to scientific progress. So far, researchers have been the main powerhouse behind hypothesis generation by painstaking data analysis and thinking (also known as the Eureka moment). In this paper, we examine the potential of large language models (LLMs) to generate hypotheses. We focus on hypothesis generation based on data (i.e., labeled examples). To enable LLMs to handle arbitrarily long contexts, we generate initial hypotheses from a small number of examples and then update them iteratively to improve the quality of hypotheses. Inspired by multi-armed bandits, we design a reward function to inform the exploitation-exploration tradeoff in the update process. Our algorithm is able to generate hypotheses that enable much better predictive performance than few-shot prompting in classification tasks, improving accuracy by 31.7% on a synthetic dataset and by 13.9%, 3.3% and, 24.9% on three real-world datasets. We also outperform supervised learning by 12.8% and 11.2% on two challenging real-world datasets. Furthermore, we find that the generated hypotheses not only corroborate human-verified theories but also uncover new insights for the tasks.

Related papers

• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on causal inferences on a target experiment with unlabeled factual outcomes, retrieved by a predictive model fine-tuned on a labeled similar experiment. First, we show that factual outcome estimation via Empirical Risk Minimization (ERM) may fail to yield valid causal inferences on the target population. We propose Deconfounded Empirical Risk Minimization (DERM), a new simple learning procedure minimizing the risk over a fictitious target population.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Sparse Autoencoders for Hypothesis Generation [1.5450225594635711]
  HypotheSAEs is a method to hypothesize relationships between text data (e.g., headlines) and a target variable (e.g., clicks) We train a sparse autoencoder on text embeddings to produce interpretable features describing the data distribution. We select features that predict the target variable, and (3) generate a natural language interpretation of each feature.
  arXiv  Detail & Related papers  (2025-02-05T18:58:02Z)
• Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning [88.68573198200698]
  We introduce ExploreToM, the first framework to allow large-scale generation of diverse and challenging theory of mind data. Our approach leverages an A* search over a custom domain-specific language to produce complex story structures and novel, diverse, yet plausible scenarios. Our evaluation reveals that state-of-the-art LLMs, such as Llama-3.1-70B and GPT-4o, show accuracies as low as 0% and 9% on ExploreToM-generated data.
  arXiv  Detail & Related papers  (2024-12-12T21:29:00Z)
• Improving the Natural Language Inference robustness to hard dataset by data augmentation and preprocessing [1.7487745673871375]
  Natural Language Inference (NLI) is the task of inferring whether the hypothesis can be justified by the given premise. We propose the data augmentation and preprocessing methods to solve the word overlap, numerical reasoning and length mismatch problems.
  arXiv  Detail & Related papers  (2024-12-10T01:49:23Z)
• Literature Meets Data: A Synergistic Approach to Hypothesis Generation [24.98928229927995]
  We develop the first method that combines literature-based insights with data to perform hypothesis generation. We also conduct the first human evaluation to assess the utility of LLM-generated hypotheses in assisting human decision-making.
  arXiv  Detail & Related papers  (2024-10-22T18:00:00Z)
• Hypothesizing Missing Causal Variables with LLMs [55.28678224020973]
  We formulate a novel task where the input is a partial causal graph with missing variables, and the output is a hypothesis about the missing variables to complete the partial graph. We show the strong ability of LLMs to hypothesize the mediation variables between a cause and its effect. We also observe surprising results where some of the open-source models outperform the closed GPT-4 model.
  arXiv  Detail & Related papers  (2024-09-04T10:37:44Z)
• Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale [53.152460508207184]
  Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
  arXiv  Detail & Related papers  (2024-02-02T05:53:22Z)
• Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
  This work proposes the first dataset for social science academic hypotheses discovery. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
  arXiv  Detail & Related papers  (2023-09-06T05:19:41Z)
• Exploring Lexical Irregularities in Hypothesis-Only Models of Natural Language Inference [5.283529004179579]
  Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) is the task of predicting the entailment relation between a pair of sentences. Models that understand entailment should encode both, the premise and the hypothesis. Experiments by Poliak et al. revealed a strong preference of these models towards patterns observed only in the hypothesis.
  arXiv  Detail & Related papers  (2021-01-19T01:08:06Z)
• Empirically Verifying Hypotheses Using Reinforcement Learning [58.09414653169534]
  This paper formulates hypothesis verification as an RL problem. We aim to build an agent that, given a hypothesis about the dynamics of the world, can take actions to generate observations which can help predict whether the hypothesis is true or false.
  arXiv  Detail & Related papers  (2020-06-29T01:01:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.