Language Models are Crossword Solvers

• URL: http://arxiv.org/abs/2406.09043v3
• Date: Sun, 09 Feb 2025 14:26:45 GMT
• Title: Language Models are Crossword Solvers
• Authors: Soumadeep Saha, Sutanoya Chakraborty, Saptarshi Saha, Utpal Garain,
• Abstract summary: We tackle the challenge of solving crosswords with large language models (LLMs) We demonstrate that the current generation of language models shows significant competence at deciphering cryptic crossword clues. We also develop a search algorithm that builds off this performance to tackle the problem of solving full crossword grids with out-of-the-box LLMs.
• Score: 1.53744306569115
• License:
• Abstract: Crosswords are a form of word puzzle that require a solver to demonstrate a high degree of proficiency in natural language understanding, wordplay, reasoning, and world knowledge, along with adherence to character and length constraints. In this paper we tackle the challenge of solving crosswords with large language models (LLMs). We demonstrate that the current generation of language models shows significant competence at deciphering cryptic crossword clues and outperforms previously reported state-of-the-art (SoTA) results by a factor of 2-3 in relevant benchmarks. We also develop a search algorithm that builds off this performance to tackle the problem of solving full crossword grids with out-of-the-box LLMs for the very first time, achieving an accuracy of 93% on New York Times crossword puzzles. Additionally, we demonstrate that LLMs generalize well and are capable of supporting answers with sound rationale.

Related papers

• What Makes Cryptic Crosswords Challenging for LLMs? [4.463184061618504]
  Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2024-12-12T07:23:52Z)
• Are LLMs Good Cryptic Crossword Solvers? [4.463184061618504]
  Cryptic crosswords are puzzles that rely on the solver's ability to manipulate language on different levels and deal with various types of wordplay. Previous research suggests that solving such puzzles is a challenge even for modern NLP models.
  arXiv  Detail & Related papers  (2024-03-15T06:57:08Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners? [140.9751389452011]
  We study the biases of large language models (LLMs) in relation to those known in children when solving arithmetic word problems. We generate a novel set of word problems for each of these tests, using a neuro-symbolic approach that enables fine-grained control over the problem features.
  arXiv  Detail & Related papers  (2024-01-31T18:48:20Z)
• The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics [74.99898531299148]
  This research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. We apply two languages to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different language families and sizes. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed.
  arXiv  Detail & Related papers  (2023-11-16T09:35:50Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)
• Shortcut Learning of Large Language Models in Natural Language Understanding [119.45683008451698]
  Large language models (LLMs) have achieved state-of-the-art performance on a series of natural language understanding tasks. They might rely on dataset bias and artifacts as shortcuts for prediction. This has significantly affected their generalizability and adversarial robustness.
  arXiv  Detail & Related papers  (2022-08-25T03:51:39Z)
• Down and Across: Introducing Crossword-Solving as a New NLP Benchmark [11.194615436370507]
  We release the specification of a corpus of crossword puzzles collected from the New York Times daily crossword spanning 25 years. These puzzles include a diverse set of clues: historic, factual, word meaning, synonyms/antonyms, fill-in-the-blank, abbreviations, prefixes/suffixes, wordplay, and cross-lingual.
  arXiv  Detail & Related papers  (2022-05-20T21:16:44Z)
• Automated Crossword Solving [38.36920665368784]
  Our system improves exact puzzle accuracy from 57% to 82% on crosswords from The New York Times. Our system also won first place at the top human crossword tournament.
  arXiv  Detail & Related papers  (2022-05-19T16:28:44Z)
• Visual Keyword Spotting with Attention [82.79015266453533]
  We investigate Transformer-based models that ingest two streams, a visual encoding of the video and a phonetic encoding of the keyword. We show through extensive evaluations that our model outperforms the prior state-of-the-art visual keyword spotting and lip reading methods. We demonstrate the ability of our model to spot words under the extreme conditions of isolated mouthings in sign language videos.
  arXiv  Detail & Related papers  (2021-10-29T17:59:04Z)
• Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP [28.479149974110463]
  Cryptic crosswords, the dominant crossword variety in the UK, are a promising target for advancing NLP systems. We present a dataset of cryptic clues as a challenging new benchmark for NLP systems. We also introduce a challenging data split, examine the meta-linguistic capabilities of subword-tokenized models, and investigate model systematicity by perturbing the wordplay part of clues.
  arXiv  Detail & Related papers  (2021-04-17T18:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.