Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large
Language Models
- URL: http://arxiv.org/abs/2402.03877v2
- Date: Wed, 14 Feb 2024 19:33:19 GMT
- Title: Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large
Language Models
- Authors: Spyridon Mouselinos, Henryk Michalewski, Mateusz Malinowski
- Abstract summary: Large Language Models (LLMs) demonstrate ever-increasing abilities in mathematical and algorithmic tasks, yet their geometric reasoning skills are underexplored.
We investigate LLMs' abilities in constructive geometric problem-solving one of the most fundamental steps in the development of human mathematical reasoning.
Our work reveals notable challenges that the state-of-the-art LLMs face in this domain despite many successes in similar areas.
- Score: 28.819559978685806
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Large Language Models (LLMs) demonstrate ever-increasing abilities in
mathematical and algorithmic tasks, yet their geometric reasoning skills are
underexplored. We investigate LLMs' abilities in constructive geometric
problem-solving one of the most fundamental steps in the development of human
mathematical reasoning. Our work reveals notable challenges that the
state-of-the-art LLMs face in this domain despite many successes in similar
areas. LLMs exhibit biases in target variable selection and struggle with 2D
spatial relationships, often misrepresenting and hallucinating objects and
their placements. To this end, we introduce a framework that formulates an
LLMs-based multi-agents system that enhances their existing reasoning potential
by conducting an internal dialogue. This work underscores LLMs' current
limitations in geometric reasoning and improves geometric reasoning
capabilities through self-correction, collaboration, and diverse role
specializations.
Related papers
- Reasoning in Large Language Models: A Geometric Perspective [4.2909314120969855]
We explore the reasoning abilities of large language models (LLMs) through their geometrical understanding.
We establish a connection between the expressive power of LLMs and the density of their self-attention graphs.
arXiv Detail & Related papers (2024-07-02T21:39:53Z) - MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions [58.57255822646756]
This paper introduces MathChat, a benchmark designed to evaluate large language models (LLMs) across a broader spectrum of mathematical tasks.
We evaluate the performance of various SOTA LLMs on the MathChat benchmark, and we observe that while these models excel in single turn question answering, they significantly underperform in more complex scenarios.
We develop MathChat sync, a synthetic dialogue based math dataset for LLM finetuning, focusing on improving models' interaction and instruction following capabilities in conversations.
arXiv Detail & Related papers (2024-05-29T18:45:55Z) - Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models [71.93366651585275]
We propose visualization-of-Thought (VoT) prompting for large language models (LLMs)
VoT elicits spatial reasoning of LLMs by visualizing their reasoning traces, thereby guiding subsequent reasoning steps.
We employ VoT for multi-hop spatial reasoning tasks, including natural language navigation, visual navigation, and visual tiling in 2D grid worlds.
arXiv Detail & Related papers (2024-04-04T17:45:08Z) - Adversarial Math Word Problem Generation [6.92510069380188]
We propose a new paradigm for ensuring fair evaluation of large language models (LLMs)
We generate adversarial examples which preserve the structure and difficulty of the original questions aimed for assessment, but are unsolvable by LLMs.
We conduct experiments on various open- and closed-source LLMs, quantitatively and qualitatively demonstrating that our method significantly degrades their math problem-solving ability.
arXiv Detail & Related papers (2024-02-27T22:07:52Z) - Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks.
The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human.
These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
arXiv Detail & Related papers (2024-01-30T17:38:54Z) - Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventions [47.83142414018448]
Large Language Models (LLMs) have showcased striking results on logical reasoning benchmarks.
We focus on two popular reasoning tasks: arithmetic reasoning and code generation.
We show a significant performance drop across all the models against the questions.
arXiv Detail & Related papers (2024-01-17T18:13:07Z) - G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model [124.68242155098189]
Large language models (LLMs) have shown remarkable proficiency in human-level reasoning and generation capabilities.
G-LLaVA demonstrates exceptional performance in solving geometric problems, significantly outperforming GPT-4-V on the MathVista benchmark with only 7B parameters.
arXiv Detail & Related papers (2023-12-18T17:36:20Z) - Evaluating Spatial Understanding of Large Language Models [26.436450329727645]
Large language models show remarkable capabilities across a variety of tasks.
Recent studies suggest that LLM representations implicitly capture aspects of the underlying grounded concepts.
We design natural-language navigation tasks and evaluate the ability of LLMs to represent and reason about spatial structures.
arXiv Detail & Related papers (2023-10-23T03:44:40Z) - Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration [83.4031923134958]
Corex is a suite of novel general-purpose strategies that transform Large Language Models into autonomous agents.
Inspired by human behaviors, Corex is constituted by diverse collaboration paradigms including Debate, Review, and Retrieve modes.
We demonstrate that orchestrating multiple LLMs to work in concert yields substantially better performance compared to existing methods.
arXiv Detail & Related papers (2023-09-30T07:11:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.