Eliciting Problem Specifications via Large Language Models
- URL: http://arxiv.org/abs/2405.12147v2
- Date: Mon, 10 Jun 2024 19:05:57 GMT
- Title: Eliciting Problem Specifications via Large Language Models
- Authors: Robert E. Wray, James R. Kirk, John E. Laird,
- Abstract summary: Large language models (LLMs) can be utilized to map a problem class into a semi-formal specification.
A cognitive system can then use the problem-space specification to solve multiple instances of problems from the problem class.
- Score: 4.055489363682198
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Cognitive systems generally require a human to translate a problem definition into some specification that the cognitive system can use to attempt to solve the problem or perform the task. In this paper, we illustrate that large language models (LLMs) can be utilized to map a problem class, defined in natural language, into a semi-formal specification that can then be utilized by an existing reasoning and learning system to solve instances from the problem class. We present the design of LLM-enabled cognitive task analyst agent(s). Implemented with LLM agents, this system produces a definition of problem spaces for tasks specified in natural language. LLM prompts are derived from the definition of problem spaces in the AI literature and general problem-solving strategies (Polya's How to Solve It). A cognitive system can then use the problem-space specification, applying domain-general problem solving strategies ("weak methods" such as search), to solve multiple instances of problems from the problem class. This result, while preliminary, suggests the potential for speeding cognitive systems research via disintermediation of problem formulation while also retaining core capabilities of cognitive systems, such as robust inference and online learning.
Related papers
- Interactively Diagnosing Errors in a Semantic Parser [7.136205674624813]
We present work in progress on an interactive error diagnosis system for the CNLU.
We show how the first two stages of the INLD pipeline can be cast as a model-based diagnosis problem.
We demonstrate our system's ability to diagnose semantic errors on synthetic examples.
arXiv Detail & Related papers (2024-07-08T21:16:09Z) - Knowledge Tagging System on Math Questions via LLMs with Flexible Demonstration Retriever [48.5585921817745]
Large Language Models (LLMs) are used to automate the knowledge tagging task.
We show the strong performance of zero- and few-shot results over math questions knowledge tagging tasks.
By proposing a reinforcement learning-based demonstration retriever, we successfully exploit the great potential of different-sized LLMs.
arXiv Detail & Related papers (2024-06-19T23:30:01Z) - Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? [140.9751389452011]
We study the biases of large language models (LLMs) in relation to those known in children when solving arithmetic word problems.
We generate a novel set of word problems for each of these tests, using a neuro-symbolic approach that enables fine-grained control over the problem features.
arXiv Detail & Related papers (2024-01-31T18:48:20Z) - MacGyver: Are Large Language Models Creative Problem Solvers? [87.70522322728581]
We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting.
We create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems.
We present our collection to both LLMs and humans to compare and contrast their problem-solving abilities.
arXiv Detail & Related papers (2023-11-16T08:52:27Z) - ACES: Generating Diverse Programming Puzzles with with Autotelic Generative Models [20.039580079339537]
Autotelic CodE Search (ACES) jointly optimize for the diversity and difficulty of generated problems.
We represent problems in a space of semantic descriptors describing the programming skills required to solve them.
ACES iteratively prompts a large language model to generate difficult problems achieving a diversity of target semantic descriptors.
arXiv Detail & Related papers (2023-10-15T14:57:14Z) - MRKL Systems: A modular, neuro-symbolic architecture that combines large
language models, external knowledge sources and discrete reasoning [50.40151403246205]
Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks.
We define a flexible architecture with multiple neural models, complemented by discrete knowledge and reasoning modules.
We describe this neuro-symbolic architecture, dubbed the Modular Reasoning, Knowledge and Language (MRKL) system.
arXiv Detail & Related papers (2022-05-01T11:01:28Z) - Learning Physical Concepts in Cyber-Physical Systems: A Case Study [72.74318982275052]
We provide an overview of the current state of research regarding methods for learning physical concepts in time series data.
We also analyze the most important methods from the current state of the art using the example of a three-tank system.
arXiv Detail & Related papers (2021-11-28T14:24:52Z) - Problem Learning: Towards the Free Will of Machines [19.365648708008624]
This paper describes Problem Learning, which aims at learning to discover and define valid and ethical problems from data or from the machine's interaction with the environment.
In a broader sense, problem learning is an approach towards the free will of intelligent machines.
arXiv Detail & Related papers (2021-09-01T04:08:09Z) - Specialization in Hierarchical Learning Systems [0.0]
We investigate in how far information constraints in hierarchies of experts not only provide a principled method for regularization but also to enforce specialization.
We devise an information-theoretically motivated on-line learning rule that allows partitioning of the problem space into multiple sub-problems that can be solved by the individual experts.
We show the broad applicability of our approach on a range of problems including classification, regression, density estimation, and reinforcement learning problems.
arXiv Detail & Related papers (2020-11-03T17:00:31Z) - Probably Approximately Correct Constrained Learning [135.48447120228658]
We develop a generalization theory based on the probably approximately correct (PAC) learning framework.
We show that imposing a learner does not make a learning problem harder in the sense that any PAC learnable class is also a constrained learner.
We analyze the properties of this solution and use it to illustrate how constrained learning can address problems in fair and robust classification.
arXiv Detail & Related papers (2020-06-09T19:59:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.