Explicit Inductive Inference using Large Language Models

• URL: http://arxiv.org/abs/2408.14467v1
• Date: Mon, 26 Aug 2024 17:58:17 GMT
• Title: Explicit Inductive Inference using Large Language Models
• Authors: Tianyang Liu, Tianyi Li, Liang Cheng, Mark Steedman,
• Abstract summary: Large Language Models (LLMs) are reported to hold undesirable attestation bias on inference tasks. We propose a pipeline that exploits this bias to do explicit inductive inference.
• Score: 13.767536248988268
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) are reported to hold undesirable attestation bias on inference tasks: when asked to predict if a premise P entails a hypothesis H, instead of considering H's conditional truthfulness entailed by P, LLMs tend to use the out-of-context truth label of H as a fragile proxy. In this paper, we propose a pipeline that exploits this bias to do explicit inductive inference. Our pipeline uses an LLM to transform a premise into a set of attested alternatives, and then aggregate answers of the derived new entailment inquiries to support the original inference prediction. On a directional predicate entailment benchmark, we demonstrate that by applying this simple pipeline, we can improve the overall performance of LLMs on inference and substantially alleviate the impact of their attestation bias.

Related papers

• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Quantitative Certification of Bias in Large Language Models [18.91760463088026]
  Large Language Models (LLMs) can produce biased responses that can cause representational harms. We propose QuaCer-B that certifies LLMs for bias on distributions of prompts.
  arXiv  Detail & Related papers  (2024-05-29T05:39:37Z)
• Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction [56.17020601803071]
  Recent research shows that pre-trained language models (PLMs) suffer from "prompt bias" in factual knowledge extraction. This paper aims to improve the reliability of existing benchmarks by thoroughly investigating and mitigating prompt bias.
  arXiv  Detail & Related papers  (2024-03-15T02:04:35Z)
• Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment [32.12998469814097]
  A novel causal prompting method based on front-door adjustment is proposed to effectively mitigate Large Language Models (LLMs) biases. Experimental results show that the proposed causal prompting approach achieves excellent performance across seven natural language processing datasets.
  arXiv  Detail & Related papers  (2024-03-05T07:47:34Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Hypothesis Search: Inductive Reasoning with Language Models [39.03846394586811]
  Recent work evaluates large language models on inductive reasoning tasks by directly prompting them yielding "in context learning" This works well for straightforward inductive tasks but performs poorly on complex tasks such as the Abstraction and Reasoning Corpus (ARC) In this work, we propose to improve the inductive reasoning ability of LLMs by generating explicit hypotheses at multiple levels of abstraction.
  arXiv  Detail & Related papers  (2023-09-11T17:56:57Z)
• Sources of Hallucination by Large Language Models on Inference Tasks [16.644096408742325]
  Large Language Models (LLMs) are claimed to be capable of Natural Language Inference (NLI) We present a series of behavioral studies on several LLM families which probe their behavior using controlled experiments.
  arXiv  Detail & Related papers  (2023-05-23T22:24:44Z)
• ADEPT: A DEbiasing PrompT Framework [49.582497203415855]
  Finetuning is an applicable approach for debiasing contextualized word embeddings. discrete prompts with semantic meanings have shown to be effective in debiasing tasks. We propose ADEPT, a method to debias PLMs using prompt tuning while maintaining the delicate balance between removing biases and ensuring representation ability.
  arXiv  Detail & Related papers  (2022-11-10T08:41:40Z)
• Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence [45.9949173746044]
  We show that large-size pre-trained language models (PLMs) do not satisfy the logical negation property (LNP) We propose a novel intermediate training task, names meaning-matching, designed to directly learn a meaning-text correspondence. We find that the task enables PLMs to learn lexical semantic information.
  arXiv  Detail & Related papers  (2022-05-08T08:37:36Z)
• Probing as Quantifying the Inductive Bias of Pre-trained Representations [99.93552997506438]
  We present a novel framework for probing where the goal is to evaluate the inductive bias of representations for a particular task. We apply our framework to a series of token-, arc-, and sentence-level tasks.
  arXiv  Detail & Related papers  (2021-10-15T22:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.