Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification

• URL: http://arxiv.org/abs/2402.03686v3
• Date: Mon, 27 May 2024 18:44:14 GMT
• Title: Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification
• Authors: Soumya Sanyal, Tianyi Xiao, Jiacheng Liu, Wenya Wang, Xiang Ren,
• Abstract summary: We study the entailment verification problem of multi-sentence premises. Modern NLP problems, such as detecting inconsistent model-generated rationales, require complex multi-hop reasoning.
• Score: 41.330719056639616
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Making inferences in text comprehension to understand the meaning is essential in language processing. This work studies the entailment verification (EV) problem of multi-sentence premises that requires a system to make multiple inferences implicitly. Studying EV for such complex premises is important because modern NLP problems, such as detecting inconsistent model-generated rationales, require complex multi-hop reasoning. However, current textual inference datasets mostly contain short premises that only partially focus on these challenges. To address this, we compile an EV benchmark that includes datasets from three NLP domains (NLI, contextual QA, and rationales) containing multi-sentence premises. On benchmarking humans and LLMs, we find that LLMs are better than humans in multi-hop reasoning across extended contexts, while humans perform better in simple deductive reasoning tasks. We also finetune a Flan-T5 model for EV using two training objectives to obtain a strong open-source model that outperforms GPT-3.5 and rivals GPT-4. Finally, we use this model to filter out inconsistent model-generated rationales in self-consistency decoding, resulting in a 6% accuracy improvement on average across three MCQ datasets.

Related papers

• Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design [63.24275274981911]
  Compound AI Systems consisting of many language model inference calls are increasingly employed. In this work, we construct systems, which we call Networks of Networks (NoNs) organized around the distinction between generating a proposed answer and verifying its correctness. We introduce a verifier-based judge NoN with K generators, an instantiation of "best-of-K" or "judge-based" compound AI systems.
  arXiv  Detail & Related papers  (2024-07-23T20:40:37Z)
• From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI [0.0]
  We study the effectiveness of large language models (LLMs) on different question answering tasks. We demonstrate the ability of LLMs to reason with commonsense as the models outperform humans on different datasets. Our questionnaire revealed that 66% of participants rated GPT-3.5's explanations as either "good" or "excellent"
  arXiv  Detail & Related papers  (2024-07-04T09:38:49Z)
• Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models [46.26140720993383]
  Multi-LogiEval is a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths. We conduct evaluations on a range of Large Language Models including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral.
  arXiv  Detail & Related papers  (2024-06-24T23:02:56Z)
• InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models [50.03163753638256]
  Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark.
  arXiv  Detail & Related papers  (2023-11-20T07:06:31Z)
• Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond [32.797832207443896]
  Large Language Models (LLMs) have emerged as a noteworthy innovation in natural language processing (NLP) We aim to bridge this gap and provide comprehensive evaluations in this paper.
  arXiv  Detail & Related papers  (2023-06-16T13:39:35Z)
• STREET: A Multi-Task Structured Reasoning and Explanation Benchmark [56.555662318619135]
  We introduce a unified multi-task and multi-domain natural language reasoning and explanation benchmark. We expect models to not only answer questions, but also produce step-by-step structured explanations describing how premises in the question are used to produce intermediate conclusions that can prove the correctness of a certain answer.
  arXiv  Detail & Related papers  (2023-02-13T22:34:02Z)
• Teaching Broad Reasoning Skills via Decomposition-Guided Contexts [50.114651561111245]
  Question-answering datasets require a broad set of reasoning skills. We show how to use question decompositions to teach these broad reasoning skills in a robust fashion.
  arXiv  Detail & Related papers  (2022-05-25T05:13:21Z)
• The Right Tool for the Job: Matching Model and Instance Complexities [62.95183777679024]
  As NLP models become larger, executing a trained model requires significant computational resources incurring monetary and environmental costs. We propose a modification to contextual representation fine-tuning which, during inference, allows for an early (and fast) "exit" We test our proposed modification on five different datasets in two tasks: three text classification datasets and two natural language inference benchmarks.
  arXiv  Detail & Related papers  (2020-04-16T04:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.