GenPuzzle: Benchmarking Visual Reasoning in Image Generation Models

• URL: http://arxiv.org/abs/2609.05902v1
• Date: Sat, 05 Sep 2026 05:56:15 GMT
• Title: GenPuzzle: Benchmarking Visual Reasoning in Image Generation Models
• Abstract summary: GenPuzzle is a benchmark for reasoning-centric image generation.<n>It contains 2,005 problems across 12 tracks, spanning pattern completion, spatial construction, mazes, Sudoku, nonograms, tangrams, board games, matchstick puzzles, orthographic projection, and mathematical visual proof.
• Score: 14.487084010911095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent image generation systems increasingly combine multimodal understanding, reasoning, and synthesis, suggesting that they may do more than render plausible scenes. Yet existing evaluations emphasize aesthetics, prompt alignment, compositionality, or text-based answers, leaving unclear whether these systems can solve visual problems and faithfully express solutions in pixels. We introduce GenPuzzle, a benchmark for reasoning-centric image generation. GenPuzzle contains 2,005 problems across 12 tracks, spanning pattern completion, spatial construction, mazes, Sudoku, nonograms, tangrams, board games, matchstick puzzles, orthographic projection, and mathematical visual proof. Each task provides a visual puzzle and requires an image output that preserves the input state while executing a logically valid solution. GenPuzzle uses task-specific evaluation protocols: discrete grid outputs are transcribed and verified programmatically, while visually complex outputs are assessed with tiered, multidimensional, or binary multimodal large language model (MLLM) rubrics. We further select the automatic judge by measuring agreement with human reference scores. Across three frontier generators, the strongest model reaches only 40.57 Macro Overall, revealing frequent failures in logic, geometry, state preservation, and instruction execution. GenPuzzle provides a testbed for measuring progress from image rendering toward visual problem solving.

Related papers

• Seq2Seq Models Reconstruct Visual Jigsaw Puzzles without Seeing Them [2.8834483859625952]
  We introduce a specialized tokenizer that converts each puzzle piece into a discrete sequence of tokens.<n>Treated as "blind" solvers, encoder-decoder transformers accurately reconstruct the original layout.<n>Despite being deliberately restricted from accessing visual input, our models achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2025-11-09T10:43:16Z)
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought [118.71264263478083]
  We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning.<n>We include 546 multimodal problems, annotated with intermediate visual images and final answers.
  arXiv  Detail & Related papers  (2025-11-04T18:00:51Z)
• Leveraging Hierarchical Image-Text Misalignment for Universal Fake Image Detection [58.927873049646024]
  We show that fake images cannot be properly aligned with corresponding captions compared to real images.<n>We propose a simple yet effective ITEM by leveraging the image-text misalignment in a joint visual-language space as discriminative clues.
  arXiv  Detail & Related papers  (2025-11-01T06:51:14Z)
• RECODE: Reasoning Through Code Generation for Visual Question Answering [68.86938437188964]
  We propose to leverage derendering -- the process of reverse-engineering visuals into executable code -- as a new modality for verifiable visual reasoning.<n>Our work demonstrates that grounding visual perception in executable code provides a new path toward more accurate and verifiable multimodal reasoning.
  arXiv  Detail & Related papers  (2025-10-15T17:05:37Z)
• GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzles [13.001531516689035]
  We challenge text-to-image models with generating escape room puzzle images that are visually appealing, logically solid, and intellectually stimulating.<n>We propose a hierarchical multi-agent framework that decomposes this task into structured stages: functional design, symbolic scene graph reasoning, layout synthesis, and local image editing.
  arXiv  Detail & Related papers  (2025-06-27T01:08:37Z)
• ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoning [76.2503352325492]
  ControlThinker is a novel framework that employs a "comprehend-then-generate" paradigm.<n>Latent semantics from control images are mined to enrich text prompts.<n>This enriched semantic understanding then seamlessly aids in image generation without the need for additional complex modifications.
  arXiv  Detail & Related papers  (2025-06-04T05:56:19Z)
• Puzzled by Puzzles: When Vision-Language Models Can't Take a Hint [57.73346054360675]
  Rebus puzzles, visual riddles that encode language through imagery, spatial arrangement, and symbolic substitution, pose a unique challenge to current vision-language models (VLMs)<n>In this paper, we investigate the capacity of contemporary VLMs to interpret and solve rebus puzzles by constructing a hand-generated and annotated benchmark of diverse English-language rebus puzzles.
  arXiv  Detail & Related papers  (2025-05-29T17:59:47Z)
• GenVP: Generating Visual Puzzles with Contrastive Hierarchical VAEs [15.118234858274679]
  We propose Generative Visual Puzzles (GenVP) to model the entire RPM generation process.<n>Our model's capability spans from generating multiple solutions for one specific problem prompt to creating complete new puzzles out of the desired set of rules.
  arXiv  Detail & Related papers  (2025-03-30T21:35:26Z)
• Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning [24.386388107656334]
  This paper introduces the novel task of multimodal puzzle solving, framed within the context of visual question-answering. We present a new dataset, AlgoVQA, designed to challenge and evaluate the capabilities of multimodal language models in solving algorithmic puzzles.
  arXiv  Detail & Related papers  (2024-03-06T17:15:04Z)
• Video Anomaly Detection by Solving Decoupled Spatio-Temporal Jigsaw Puzzles [67.39567701983357]
  Video Anomaly Detection (VAD) is an important topic in computer vision. Motivated by the recent advances in self-supervised learning, this paper addresses VAD by solving an intuitive yet challenging pretext task. Our method outperforms state-of-the-art counterparts on three public benchmarks.
  arXiv  Detail & Related papers  (2022-07-20T19:49:32Z)
• Pictorial and apictorial polygonal jigsaw puzzles: The lazy caterer model, properties, and solvers [14.08706290287121]
  We formalize a new type of jigsaw puzzle where the pieces are general convex polygons generated by cutting through a global polygonal shape/image with an arbitrary number of straight cuts. We analyze the theoretical properties of such puzzles, including the inherent challenges in solving them once pieces are contaminated with geometrical noise.
  arXiv  Detail & Related papers  (2020-08-17T22:07:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.