A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoning

• URL: http://arxiv.org/abs/2302.09068v1
• Date: Wed, 15 Feb 2023 05:04:49 GMT
• Title: A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoning
• Abstract summary: Two recently released generative transformer models, ChatGPT and DALL-E 2, are evaluated. DALL-E 2 is able to generate at least one correct image for each spatial reasoning prompt, but most images generated are incorrect. In evaluating ChatGPT on the axioms developed under the classical Von Neumann-Morgenstern utility theorem, we find that, although it demonstrates some level of rational decision-making, many of its decisions violate at least one of the axioms.
• Score: 8.185725740857594
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We conduct a pilot study selectively evaluating the cognitive abilities (decision making and spatial reasoning) of two recently released generative transformer models, ChatGPT and DALL-E 2. Input prompts were constructed following neutral a priori guidelines, rather than adversarial intent. Post hoc qualitative analysis of the outputs shows that DALL-E 2 is able to generate at least one correct image for each spatial reasoning prompt, but most images generated are incorrect (even though the model seems to have a clear understanding of the objects mentioned in the prompt). Similarly, in evaluating ChatGPT on the rationality axioms developed under the classical Von Neumann-Morgenstern utility theorem, we find that, although it demonstrates some level of rational decision-making, many of its decisions violate at least one of the axioms even under reasonable constructions of preferences, bets, and decision-making prompts. ChatGPT's outputs on such problems generally tended to be unpredictable: even as it made irrational decisions (or employed an incorrect reasoning process) for some simpler decision-making problems, it was able to draw correct conclusions for more complex bet structures. We briefly comment on the nuances and challenges involved in scaling up such a 'cognitive' evaluation or conducting it with a closed set of answer keys ('ground truth'), given that these models are inherently generative and open-ended in responding to prompts.

Related papers

• Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny? [0.0]
  We investigate an alternative standard designed to function despite realistic ambiguity.<n>We use a four-phase dialectical protocol grounded in Walton's theory of argumentation schemes and Govier's criteria for argument cogency.<n>It is adaptive to different frames of reasoning, extends beyond multiple-choice framing, and treats both the reasoning that precedes a verdict and its post-hoc justification.
  arXiv  Detail & Related papers  (2026-09-04T12:40:39Z)
• Heuristic Classification of Thoughts Prompting (HCoT): Integrating Expert System Heuristics for Structured Reasoning into Large Language Models [9.545957160908115]
  Heuristic- Classification-of-Thoughts prompting schema (HCoT)<n>Heuristic- Classification-of-Thoughts prompting schema (HCoT)<n>Heuristic- Classification-of-Thoughts prompting schema (HCoT)
  arXiv  Detail & Related papers  (2026-04-14T07:24:08Z)
• Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models [72.4149653187766]
  We propose a Reasoner-Verifier framework named Adrialversa Reasoning RAG (ARR)<n>The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage.<n> Experiments on multiple benchmarks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2026-01-08T06:57:03Z)
• ArgRAG: Explainable Retrieval Augmented Generation using Quantitative Bipolar Argumentation [37.816652871790474]
  We propose ArgRAG, an explainable, and contestable alternative that replaces black-box reasoning with structured inference.<n>ArgRAG constructs a QBAF from retrieved documents and performs deterministic reasoning under gradual semantics.
  arXiv  Detail & Related papers  (2025-08-26T13:54:51Z)
• Thinking About Thinking: SAGE-nano's Inverse Reasoning for Self-Aware Language Models [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities at solving complex reasoning tasks with Chain-of-Thought prompting.<n>We introduce textbfinverse reasoning, a novel paradigm enabling LLMs to decompose and explain their own reasoning chains post-hoc.<n>Our work creates new avenues for transparent AI systems and closes significant gaps in AI safety, education, and scientific discovery.
  arXiv  Detail & Related papers  (2025-06-30T09:53:41Z)
• PixelThink: Towards Efficient Chain-of-Pixel Reasoning [70.32510083790069]
  PixelThink is a simple yet effective scheme that integrates externally estimated task difficulty and internally measured model uncertainty.<n>It learns to compress reasoning length in accordance with scene complexity and predictive confidence.<n> Experimental results demonstrate that the proposed approach improves both reasoning efficiency and overall segmentation performance.
  arXiv  Detail & Related papers  (2025-05-29T17:55:49Z)
• Implicit Reasoning in Transformers is Reasoning through Shortcuts [10.351525484558376]
  Test-time compute is emerging as a new paradigm for enhancing language models' complex multi-step reasoning capabilities. We investigate how language models perform implicit reasoning in multi-step tasks.
  arXiv  Detail & Related papers  (2025-03-10T17:58:31Z)
• Unveiling Reasoning Thresholds in Language Models: Scaling, Fine-Tuning, and Interpretability through Attention Maps [3.8936716676293917]
  This study investigates the in-context learning capabilities of various decoder-only transformer-based language models with different model sizes and training data. We identify a critical parameter threshold (1.6 billion), beyond which reasoning performance improves significantly in tasks such as commonsense reasoning in multiple-choice question answering and deductive reasoning.
  arXiv  Detail & Related papers  (2025-02-21T00:48:32Z)
• Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs [86.79757571440082]
  Large language models (LLMs) such as OpenAI's o1 have demonstrated remarkable abilities in complex reasoning tasks. We identify a phenomenon we term underthinking, where o1-like LLMs frequently switch between different reasoning thoughts. We propose a decoding strategy with thought switching penalty TIP that discourages premature transitions between thoughts.
  arXiv  Detail & Related papers  (2025-01-30T18:58:18Z)
• Can ChatGPT Make Explanatory Inferences? Benchmarks for Abductive Reasoning [0.0]
  This paper proposes a set of benchmarks for assessing the ability of AI programs to perform explanatory inference. Tests on the benchmarks reveal that ChatGPT performs creative and evaluative inferences in many domains. Claims that ChatGPT and similar models are incapable of explanation, understanding, causal reasoning, meaning, and creativity are rebutted.
  arXiv  Detail & Related papers  (2024-04-29T15:19:05Z)
• Conceptual and Unbiased Reasoning in Language Models [98.90677711523645]
  We propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions. We show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making.
  arXiv  Detail & Related papers  (2024-03-30T00:53:53Z)
• How Ambiguous are the Rationales for Natural Language Reasoning? A Simple Approach to Handling Rationale Uncertainty [0.0]
  Rationales behind answers not only explain model decisions but boost language models to reason well on complex reasoning tasks. It is non-trivial to estimate the degree to which the rationales are faithful enough to encourage model performance. We propose how to deal with imperfect rationales causing aleatoric uncertainty.
  arXiv  Detail & Related papers  (2024-02-22T07:12:34Z)
• Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA [7.141288053123662]
  Natural language explanation in visual question answer (VQA-NLE) aims to explain the decision-making process of models by generating natural language sentences to increase users' trust in the black-box systems. Existing post-hoc explanations are not always aligned with human logical inference, suffering from the issues on: 1) Deductive unsatisfiability, the generated explanations do not logically lead to the answer; 2) Factual inconsistency, the model falsifies its counterfactual explanation for answers without considering the facts in images; and 3) Semantic perturbation insensitivity, the model can not recognize the semantic changes caused by small perturbations
  arXiv  Detail & Related papers  (2023-12-21T05:51:55Z)
• HOP, UNION, GENERATE: Explainable Multi-hop Reasoning without Rationale Supervision [118.0818807474809]
  This work proposes a principled, probabilistic approach for training explainable multi-hop QA systems without rationale supervision. Our approach performs multi-hop reasoning by explicitly modeling rationales as sets, enabling the model to capture interactions between documents and sentences within a document.
  arXiv  Detail & Related papers  (2023-05-23T16:53:49Z)
• MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure [129.8481568648651]
  We propose a benchmark to investigate models' logical reasoning capabilities in complex real-life scenarios. Based on the multi-hop chain of reasoning, the explanation form includes three main components. We evaluate the current best models' performance on this new explanation form.
  arXiv  Detail & Related papers  (2022-10-22T16:01:13Z)
• Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
  We study the evaluation of counterfactual statements through neural models. First, we show that neural causal models (NCMs) are expressive enough. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
  arXiv  Detail & Related papers  (2022-09-30T18:29:09Z)
• An Interpretable Neuro-Symbolic Reasoning Framework for Task-Oriented Dialogue Generation [21.106357884651363]
  We introduce neuro-symbolic to perform explicit reasoning that justifies model decisions by reasoning chains. We propose a two-phase approach that consists of a hypothesis generator and a reasoner. The whole system is trained by exploiting raw textual dialogues without using any reasoning chain annotations.
  arXiv  Detail & Related papers  (2022-03-11T10:44:08Z)
• Abstract Reasoning via Logic-guided Generation [65.92805601327649]
  Abstract reasoning, i.e., inferring complicated patterns from given observations, is a central building block of artificial general intelligence. This paper aims to design a framework for the latter approach and bridge the gap between artificial and human intelligence. We propose logic-guided generation (LoGe), a novel generative DNN framework that reduces abstract reasoning as an optimization problem in propositional logic.
  arXiv  Detail & Related papers  (2021-07-22T07:28:24Z)
• Contrastive Explanations for Model Interpretability [77.92370750072831]
  We propose a methodology to produce contrastive explanations for classification models. Our method is based on projecting model representation to a latent space. Our findings shed light on the ability of label-contrastive explanations to provide a more accurate and finer-grained interpretability of a model's decision.
  arXiv  Detail & Related papers  (2021-03-02T00:36:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.