ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization

• URL: http://arxiv.org/abs/2607.05185v1
• Date: Mon, 06 Jul 2026 15:02:23 GMT
• Title: ClassicLogic: A Knowledge-Driven Benchmark of Classic Puzzle Games for Evaluating Compositional Generalization
• Abstract summary: ClassicLogic is a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies.<n>The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.

Related papers

• PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzles [53.47227295854126]
  This work investigates the reasoning and planning capabilities of foundation models and their scalability in complex, dynamic environments.<n>We introduce PuzzlePlex, a benchmark designed to assess these capabilities through a diverse set of puzzles.
  arXiv  Detail & Related papers  (2025-10-07T21:24:29Z)
• OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization [88.76091817642963]
  Recent large-scale language models (LLMs) with long Chain-of-such reasoning as DeepSeek-R1-have achieved impressive results on Olympiad-level mathematics.<n>We introduce OMEGA-Out-of-distribution Math Problems Evaluation with 3 Generalization Axes-a benchmark designed to evaluate three axes of out-of-distribution generalization.
  arXiv  Detail & Related papers  (2025-06-23T17:51:40Z)
• LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning [29.047063129464494]
  Large language models (LLMs) excel at many supervised tasks but often struggle with structured reasoning unfamiliar settings.<n>This discrepancy suggests that standard fine-tuning pipelines may instill narrow, domain-specifics rather than fostering general-purpose thinking strategies.<n>We propose a "play to learn" framework that fine-tunes LLMs through reinforcement learning on a suite of seven custom logic puzzles.
  arXiv  Detail & Related papers  (2025-06-05T09:40:47Z)
• Mathematical Definition and Systematization of Puzzle Rules [0.0]
  We introduce a mathematical framework for defining and systematizing pencil puzzle rules.<n>This framework formalizes grid elements, their positional relationships, and iterative composition operations.<n>Applying this framework, we successfully formalized the rules of well-known Nikoli puzzles, including Slitherlink and Sudoku.
  arXiv  Detail & Related papers  (2024-12-18T02:00:53Z)
• Improving Complex Reasoning over Knowledge Graph with Logic-Aware Curriculum Tuning [89.89857766491475]
  We propose a curriculum-based logical-aware instruction tuning framework, named LACT.<n>Specifically, we augment the arbitrary first-order logical queries via binary tree decomposition.<n> Experiments across widely used datasets demonstrate that LACT has substantial improvements(brings an average +5.5% MRR score) over advanced methods, achieving the new state-of-the-art.
  arXiv  Detail & Related papers  (2024-05-02T18:12:08Z)
• A Neural Rewriting System to Solve Algorithmic Problems [47.129504708849446]
  We propose a modular architecture designed to learn a general procedure for solving nested mathematical formulas. Inspired by rewriting systems, a classic framework in symbolic artificial intelligence, we include in the architecture three specialized and interacting modules. We benchmark our system against the Neural Data Router, a recent model specialized for systematic generalization, and a state-of-the-art large language model (GPT-4) probed with advanced prompting strategies.
  arXiv  Detail & Related papers  (2024-02-27T10:57:07Z)
• Generalization in Text-based Games via Hierarchical Reinforcement Learning [42.70991837415775]
  We introduce a hierarchical framework built upon the knowledge graph-based RL agent. In the high level, a meta-policy is executed to decompose the whole game into a set of subtasks specified by textual goals. In the low level, a sub-policy is executed to conduct goal-conditioned reinforcement learning.
  arXiv  Detail & Related papers  (2021-09-21T05:27:33Z)
• HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solving [104.79156980475686]
  Humans learn compositional and causal abstraction, ie, knowledge, in response to the structure of naturalistic tasks. We argue there shall be three levels of generalization in how an agent represents its knowledge: perceptual, conceptual, and algorithmic. This benchmark is centered around a novel task domain, HALMA, for visual concept development and rapid problem-solving.
  arXiv  Detail & Related papers  (2021-02-22T20:37:01Z)
• Compositional Generalization by Learning Analytical Expressions [87.15737632096378]
  A memory-augmented neural model is connected with analytical expressions to achieve compositional generalization. Experiments on the well-known benchmark SCAN demonstrate that our model seizes a great ability of compositional generalization.
  arXiv  Detail & Related papers  (2020-06-18T15:50:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.