Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs

• URL: http://arxiv.org/abs/2303.12810v1
• Date: Wed, 22 Mar 2023 22:53:44 GMT
• Title: Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
• Authors: Shrivats Agrawal
• Abstract summary: This study aims to investigate the performance of large language models (LLMs) on different reasoning tasks. My findings indicate that LLMs excel at analogical and moral reasoning, yet struggle to perform as proficiently on spatial reasoning tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The potential of large language models (LLMs) to reason like humans has been a highly contested topic in Machine Learning communities. However, the reasoning abilities of humans are multifaceted and can be seen in various forms, including analogical, spatial and moral reasoning, among others. This fact raises the question whether LLMs can perform equally well across all these different domains. This research work aims to investigate the performance of LLMs on different reasoning tasks by conducting experiments that directly use or draw inspirations from existing datasets on analogical and spatial reasoning. Additionally, to evaluate the ability of LLMs to reason like human, their performance is evaluted on more open-ended, natural language questions. My findings indicate that LLMs excel at analogical and moral reasoning, yet struggle to perform as proficiently on spatial reasoning tasks. I believe these experiments are crucial for informing the future development of LLMs, particularly in contexts that require diverse reasoning proficiencies. By shedding light on the reasoning abilities of LLMs, this study aims to push forward our understanding of how they can better emulate the cognitive abilities of humans.

Related papers

• A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment [0.9188951403098383]
  Large Language Models (LLMs) are increasingly used as reasoning engines in agentic systems. We present the first embodied and cognitively meaningful evaluation of physical common-sense reasoning in LLMs. We employ the Animal-AI environment, a simulated 3D virtual laboratory, to study physical common-sense reasoning in LLMs.
  arXiv  Detail & Related papers  (2024-10-30T17:28:28Z)
• Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina [7.155982875107922]
  Studies suggest large language models (LLMs) can exhibit human-like reasoning, aligning with human behavior in economic experiments, surveys, and political discourse. This has led many to propose that LLMs can be used as surrogates or simulations for humans in social science research. We assess the reasoning depth of LLMs using the 11-20 money request game.
  arXiv  Detail & Related papers  (2024-10-25T14:46:07Z)
• Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models [71.93366651585275]
  Large language models (LLMs) have exhibited impressive performance in language comprehension and various reasoning tasks. We propose Visualization-of-Thought (VoT) to elicit spatial reasoning of LLMs by visualizing their reasoning traces. VoT significantly enhances the spatial reasoning abilities of LLMs.
  arXiv  Detail & Related papers  (2024-04-04T17:45:08Z)
• Can Language Models Recognize Convincing Arguments? [12.458437450959416]
  Large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives. We study their performance in detecting convincing arguments to gain insights into their persuasive capabilities.
  arXiv  Detail & Related papers  (2024-03-31T17:38:33Z)
• Should We Fear Large Language Models? A Structural Analysis of the Human Reasoning System for Elucidating LLM Capabilities and Risks Through the Lens of Heidegger's Philosophy [0.0]
  This study investigates the capabilities and risks of Large Language Models (LLMs) It uses the innovative parallels between the statistical patterns of word relationships within LLMs and Martin Heidegger's concepts of "ready-to-hand" and "present-at-hand" Our findings reveal that while LLMs possess the capability for Direct Explicative Reasoning and Pseudo Rational Reasoning, they fall short in authentic rational reasoning and have no creative reasoning capabilities.
  arXiv  Detail & Related papers  (2024-03-05T19:40:53Z)
• Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks. The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
  arXiv  Detail & Related papers  (2024-01-30T17:38:54Z)
• Democratizing Reasoning Ability: Tailored Learning from Large Language Model [97.4921006089966]
  We propose a tailored learning approach to distill such reasoning ability to smaller LMs. We exploit the potential of LLM as a reasoning teacher by building an interactive multi-round learning paradigm. To exploit the reasoning potential of the smaller LM, we propose self-reflection learning to motivate the student to learn from self-made mistakes.
  arXiv  Detail & Related papers  (2023-10-20T07:50:10Z)
• Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models [56.34029644009297]
  Large language models (LLMs) have demonstrated the ability to overcome various limitations of formal Knowledge Representation (KR) systems. LLMs excel most in abductive reasoning, followed by deductive reasoning, while they are least effective at inductive reasoning. We study single-task training, multi-task training, and "chain-of-thought" knowledge distillation fine-tuning technique to assess the performance of model.
  arXiv  Detail & Related papers  (2023-10-02T01:00:50Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [109.8527403904657]
  We show that large language models (LLMs) possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We propose a simple method to dynamically utilize supporting documents with our judgement strategy.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)
• Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [75.85554779782048]
  Large Language Models (LLMs) have excited the natural language and machine learning community over recent years. Despite of numerous successful applications, the underlying mechanism of such in-context capabilities still remains unclear. In this work, we hypothesize that the learned textitsemantics of language tokens do the most heavy lifting during the reasoning process.
  arXiv  Detail & Related papers  (2023-05-24T07:33:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.