Large Language Models are Fixated by Red Herrings: Exploring Creative
Problem Solving and Einstellung Effect using the Only Connect Wall Dataset
- URL: http://arxiv.org/abs/2306.11167v4
- Date: Wed, 8 Nov 2023 20:47:00 GMT
- Title: Large Language Models are Fixated by Red Herrings: Exploring Creative
Problem Solving and Einstellung Effect using the Only Connect Wall Dataset
- Authors: Saeid Naeini, Raeid Saqur, Mozhgan Saeidi, John Giorgi and Babak Taati
- Abstract summary: The quest for human imitative AI has been an enduring topic in AI research since its inception.
Creative problem solving in humans is a well-studied topic in cognitive neuroscience.
Only Connect Wall segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings.
- Score: 4.789429120223149
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The quest for human imitative AI has been an enduring topic in AI research
since its inception. The technical evolution and emerging capabilities of the
latest cohort of large language models (LLMs) have reinvigorated the subject
beyond academia to the cultural zeitgeist. While recent NLP evaluation
benchmark tasks test some aspects of human-imitative behaviour (e.g.,
BIG-bench's 'human-like behavior' tasks), few, if not none, examine creative
problem solving abilities. Creative problem solving in humans is a well-studied
topic in cognitive neuroscience with standardized tests that predominantly use
the ability to associate (heterogeneous) connections among clue words as a
metric for creativity. Exposure to misleading stimuli - distractors dubbed red
herrings - impede human performance in such tasks via the fixation effect and
Einstellung paradigm. In cognitive neuroscience studies, such fixations are
experimentally induced by pre-exposing participants to orthographically similar
incorrect words to subsequent word-fragments or clues. The popular British quiz
show Only Connect's Connecting Wall segment essentially mimics Mednick's Remote
Associates Test (RAT) formulation with built-in, deliberate red herrings, which
makes it an ideal proxy dataset to explore and study fixation effect and
Einstellung paradigm from cognitive neuroscience in LLMs. In this paper we
present the novel Only Connect Wall (OCW) dataset and report results from our
evaluation of selected pre-trained language models and LLMs on creative problem
solving tasks like grouping clue words by heterogeneous connections, and
identifying correct open knowledge domain connections in respective groups. We
synthetically generate two additional datasets: OCW-Randomized, OCW-WordNet to
further analyze our red-herrings hypothesis in language models. The code and
link to the dataset are available at https://github.com/TaatiTeam/OCW.
Related papers
- tagE: Enabling an Embodied Agent to Understand Human Instructions [3.943519623674811]
We introduce a novel system known as task and argument grounding for Embodied agents (tagE)
At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language.
Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions.
arXiv Detail & Related papers (2023-10-24T08:17:48Z) - Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration [116.09561564489799]
Solo Performance Prompting transforms a single LLM into a cognitive synergist by engaging in multi-turn self-collaboration with multiple personas.
A cognitive synergist is an intelligent agent that collaboratively combines multiple minds' strengths and knowledge to enhance problem-solving in complex tasks.
Our in-depth analysis shows that assigning multiple fine-grained personas in LLMs improves problem-solving abilities compared to using a single or fixed number of personas.
arXiv Detail & Related papers (2023-07-11T14:45:19Z) - Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics
Interface of LMs Through Agentivity [68.8204255655161]
We present the semantic notion of agentivity as a case study for probing such interactions.
This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
arXiv Detail & Related papers (2023-05-29T16:24:01Z) - ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational
Finance Question Answering [70.6359636116848]
We propose a new large-scale dataset, ConvFinQA, to study the chain of numerical reasoning in conversational question answering.
Our dataset poses great challenge in modeling long-range, complex numerical reasoning paths in real-world conversations.
arXiv Detail & Related papers (2022-10-07T23:48:50Z) - Negation, Coordination, and Quantifiers in Contextualized Language
Models [4.46783454797272]
We explore whether the semantic constraints of function words are learned and how the surrounding context impacts their embeddings.
We create suitable datasets, provide new insights into the inner workings of LMs vis-a-vis function words and implement an assisting visual web interface for qualitative analysis.
arXiv Detail & Related papers (2022-09-16T10:01:11Z) - A Survey On Neural Word Embeddings [0.4822598110892847]
The study of meaning in natural language processing relies on the distributional hypothesis.
The revolutionary idea of distributed representation for a concept is close to the working of a human mind.
Neural word embeddings transformed the whole field of NLP by introducing substantial improvements in all NLP tasks.
arXiv Detail & Related papers (2021-10-05T03:37:57Z) - CogAlign: Learning to Align Textual Neural Representations to Cognitive
Language Processing Signals [60.921888445317705]
We propose a CogAlign approach to integrate cognitive language processing signals into natural language processing models.
We show that CogAlign achieves significant improvements with multiple cognitive features over state-of-the-art models on public datasets.
arXiv Detail & Related papers (2021-06-10T07:10:25Z) - Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as
a Target for NLP [5.447716844779342]
Cryptic crosswords are the dominant English-language crossword variety in the United Kingdom.
We present a dataset of cryptic crossword clues that can be used as a benchmark and train a sequence-to-sequence model to solve them.
We show that performance can be substantially improved using a novel curriculum learning approach.
arXiv Detail & Related papers (2021-04-17T18:54:00Z) - Probing Linguistic Features of Sentence-Level Representations in Neural
Relation Extraction [80.38130122127882]
We introduce 14 probing tasks targeting linguistic properties relevant to neural relation extraction (RE)
We use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets.
We find that the bias induced by the architecture and the inclusion of linguistic features are clearly expressed in the probing task performance.
arXiv Detail & Related papers (2020-04-17T09:17:40Z) - Information-Theoretic Probing for Linguistic Structure [74.04862204427944]
We propose an information-theoretic operationalization of probing as estimating mutual information.
We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research.
arXiv Detail & Related papers (2020-04-07T01:06:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.