Are Large Language Models Really Good Logical Reasoners? A Comprehensive
Evaluation and Beyond
- URL: http://arxiv.org/abs/2306.09841v3
- Date: Tue, 8 Aug 2023 12:57:18 GMT
- Title: Are Large Language Models Really Good Logical Reasoners? A Comprehensive
Evaluation and Beyond
- Authors: Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhe Zhao, Jun Liu, Erik Cambria
- Abstract summary: Large Language Models (LLMs) have emerged as a noteworthy innovation in natural language processing (NLP)
We aim to bridge this gap and provide comprehensive evaluations in this paper.
- Score: 32.797832207443896
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Logical reasoning consistently plays a fundamental and significant role in
the domains of knowledge engineering and artificial intelligence. Recently,
Large Language Models (LLMs) have emerged as a noteworthy innovation in natural
language processing (NLP), exhibiting impressive achievements across various
classic NLP tasks. However, the question of whether LLMs can effectively
address the task of logical reasoning, which requires gradual cognitive
inference similar to human intelligence, remains unanswered. To this end, we
aim to bridge this gap and provide comprehensive evaluations in this paper.
Firstly, to offer systematic evaluations, we select fifteen typical logical
reasoning datasets and organize them into deductive, inductive, abductive and
mixed-form reasoning settings. Considering the comprehensiveness of
evaluations, we include three representative LLMs (i.e., text-davinci-003,
ChatGPT and BARD) and evaluate them on all selected datasets under zero-shot,
one-shot and three-shot settings. Secondly, different from previous evaluations
relying only on simple metrics (e.g., accuracy), we propose fine-level
evaluations from objective and subjective manners, covering both answers and
explanations. Additionally, to uncover the logical flaws of LLMs, problematic
cases will be attributed to five error types from two dimensions, i.e.,
evidence selection process and reasoning process. Thirdly, to avoid the
influences of knowledge bias and purely focus on benchmarking the logical
reasoning capability of LLMs, we propose a new dataset with neutral content. It
contains 3,000 samples and covers deductive, inductive and abductive settings.
Based on the in-depth evaluations, this paper finally forms a general
evaluation scheme of logical reasoning capability from six dimensions. It
reflects the pros and cons of LLMs and gives guiding directions for future
works.
Related papers
- Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models [46.26140720993383]
Multi-LogiEval is a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths.
We conduct evaluations on a range of Large Language Models including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral.
arXiv Detail & Related papers (2024-06-24T23:02:56Z) - LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models [52.03659714625452]
Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks.
But, can they really "reason" over the natural language?
This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied.
arXiv Detail & Related papers (2024-04-23T21:08:49Z) - LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models [25.537725151112387]
We introduce AutoRace for fully automated reasoning chain evaluation.
We also develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms.
arXiv Detail & Related papers (2024-04-08T06:35:09Z) - Can Large Language Models Identify Authorship? [18.378744138365537]
Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving.
This paper conducts a comprehensive evaluation of LLMs in authorship analysis.
arXiv Detail & Related papers (2024-03-13T03:22:02Z) - A & B == B & A: Triggering Logical Reasoning Failures in Large Language
Models [65.86149763739141]
We introduce LogicAsker, an automatic approach that comprehensively evaluates and improves the logical reasoning abilities of LLMs.
We evaluate LogicAsker on six widely deployed LLMs, including GPT-3, ChatGPT, GPT-4, Bard, Vicuna, and Guanaco.
The results show that test cases from LogicAsker can find logical reasoning failures in different LLMs with a rate of 25% - 94%.
arXiv Detail & Related papers (2024-01-01T13:53:53Z) - CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark.
In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship.
We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
arXiv Detail & Related papers (2023-11-29T08:29:54Z) - LINC: A Neurosymbolic Approach for Logical Reasoning by Combining
Language Models with First-Order Logic Provers [60.009969929857704]
Logical reasoning is an important task for artificial intelligence with potential impacts on science, mathematics, and society.
In this work, we reformulating such tasks as modular neurosymbolic programming, which we call LINC.
We observe significant performance gains on FOLIO and a balanced subset of ProofWriter for three different models in nearly all experimental conditions we evaluate.
arXiv Detail & Related papers (2023-10-23T17:58:40Z) - Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models [56.34029644009297]
Large language models (LLMs) have demonstrated the ability to overcome various limitations of formal Knowledge Representation (KR) systems.
LLMs excel most in abductive reasoning, followed by deductive reasoning, while they are least effective at inductive reasoning.
We study single-task training, multi-task training, and "chain-of-thought" knowledge distillation fine-tuning technique to assess the performance of model.
arXiv Detail & Related papers (2023-10-02T01:00:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.