Eliciting Problem Specifications via Large Language Models

• URL: http://arxiv.org/abs/2405.12147v2
• Date: Mon, 10 Jun 2024 19:05:57 GMT
• Title: Eliciting Problem Specifications via Large Language Models
• Authors: Robert E. Wray, James R. Kirk, John E. Laird,
• Abstract summary: Large language models (LLMs) can be utilized to map a problem class into a semi-formal specification. A cognitive system can then use the problem-space specification to solve multiple instances of problems from the problem class.
• Score: 4.055489363682198
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cognitive systems generally require a human to translate a problem definition into some specification that the cognitive system can use to attempt to solve the problem or perform the task. In this paper, we illustrate that large language models (LLMs) can be utilized to map a problem class, defined in natural language, into a semi-formal specification that can then be utilized by an existing reasoning and learning system to solve instances from the problem class. We present the design of LLM-enabled cognitive task analyst agent(s). Implemented with LLM agents, this system produces a definition of problem spaces for tasks specified in natural language. LLM prompts are derived from the definition of problem spaces in the AI literature and general problem-solving strategies (Polya's How to Solve It). A cognitive system can then use the problem-space specification, applying domain-general problem solving strategies ("weak methods" such as search), to solve multiple instances of problems from the problem class. This result, while preliminary, suggests the potential for speeding cognitive systems research via disintermediation of problem formulation while also retaining core capabilities of cognitive systems, such as robust inference and online learning.

Related papers

• BloomWise: Enhancing Problem-Solving capabilities of Large Language Models using Bloom's-Taxonomy-Inspired Prompts [59.83547898874152]
  We introduce BloomWise, a new prompting technique, inspired by Bloom's taxonomy, to improve the performance of Large Language Models (LLMs) The decision regarding the need to employ more sophisticated cognitive skills is based on self-evaluation performed by the LLM. In extensive experiments across 4 popular math reasoning datasets, we have demonstrated the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2024-10-05T09:27:52Z)
• Knowledge Tagging with Large Language Model based Multi-Agent System [17.53518487546791]
  This paper investigates the use of a multi-agent system to address the limitations of previous algorithms. We highlight the significant potential of an LLM-based multi-agent system in overcoming the challenges that previous methods have encountered.
  arXiv  Detail & Related papers  (2024-09-12T21:39:01Z)
• Interactively Diagnosing Errors in a Semantic Parser [7.136205674624813]
  We present work in progress on an interactive error diagnosis system for the CNLU. We show how the first two stages of the INLD pipeline can be cast as a model-based diagnosis problem. We demonstrate our system's ability to diagnose semantic errors on synthetic examples.
  arXiv  Detail & Related papers  (2024-07-08T21:16:09Z)
• Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever [48.5585921817745]
  Large Language Models (LLMs) are used to automate the knowledge tagging task. We show the strong performance of zero- and few-shot results over math questions knowledge tagging tasks. By proposing a reinforcement learning-based demonstration retriever, we successfully exploit the great potential of different-sized LLMs.
  arXiv  Detail & Related papers  (2024-06-19T23:30:01Z)
• Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? [140.9751389452011]
  We study the biases of large language models (LLMs) in relation to those known in children when solving arithmetic word problems. We generate a novel set of word problems for each of these tests, using a neuro-symbolic approach that enables fine-grained control over the problem features.
  arXiv  Detail & Related papers  (2024-01-31T18:48:20Z)
• MacGyver: Are Large Language Models Creative Problem Solvers? [87.70522322728581]
  We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. We create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems. We present our collection to both LLMs and humans to compare and contrast their problem-solving abilities.
  arXiv  Detail & Related papers  (2023-11-16T08:52:27Z)
• ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Models [20.039580079339537]
  Autotelic CodE Search (ACES) jointly optimize for the diversity and difficulty of generated problems. We represent problems in a space of semantic descriptors describing the programming skills required to solve them. ACES iteratively prompts a large language model to generate difficult problems achieving a diversity of target semantic descriptors.
  arXiv  Detail & Related papers  (2023-10-15T14:57:14Z)
• Learning Physical Concepts in Cyber-Physical Systems: A Case Study [72.74318982275052]
  We provide an overview of the current state of research regarding methods for learning physical concepts in time series data. We also analyze the most important methods from the current state of the art using the example of a three-tank system.
  arXiv  Detail & Related papers  (2021-11-28T14:24:52Z)
• Probably Approximately Correct Constrained Learning [135.48447120228658]
  We develop a generalization theory based on the probably approximately correct (PAC) learning framework. We show that imposing a learner does not make a learning problem harder in the sense that any PAC learnable class is also a constrained learner. We analyze the properties of this solution and use it to illustrate how constrained learning can address problems in fair and robust classification.
  arXiv  Detail & Related papers  (2020-06-09T19:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.