JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles
Abstract Overview
JigShape is a benchmark for evaluating visual-geometric reasoning in vision-language models through jigsaw puzzle completion. Unlike prior rectangular-cut benchmarks that can produce ambiguous ground truth in repeated-texture regions, it uses tab-and-blank interlocking pieces so that local geometric compatibility and image content jointly determine a unique solution. The benchmark contains 95,468 instances derived from 23,742 source images across four grid sizes from 4×4 to 16×16, plus a paired 4×4 no-shape ablation set. The evaluation studies both zero-shot frontier VLMs and supervised fine-tuned models to test whether current systems can use geometric constraints and how performance changes as puzzle size grows.
Novelty
The paper's main novelty is the design of a jigsaw benchmark with explicit tab-and-blank shape constraints, aimed at turning puzzle solving from an ambiguous visual matching problem into a well-posed test of joint visual and geometric reasoning. It also introduces a controlled shape-versus-no-shape ablation to isolate whether models genuinely exploit boundary geometry.
Results
Empirically, only GPT-5.5 exceeds the random baseline in zero-shot evaluation on 4×4 puzzles, reaching 69.65% piece accuracy and 26.40% exact match, while the other tested frontier models remain near chance. Supervised fine-tuning makes the task learnable at 4×4, with Qwen3-VL-8B and Gemma3-12B surpassing 97% piece accuracy, but all models show a sharp scaling collapse on denser grids, falling to near-random levels by 12×12 or 16×16. The 4×4 ablation further shows that removing shape constraints causes large drops, especially for fine-tuned models, indicating heavy reliance on geometric cues.
Key Points
- JigShape replaces rectangular puzzle cuts with tab-and-blank interlocking shapes to provide strong local compatibility constraints and unambiguous ground truth.
- The benchmark scales from 4×4 to 16×16 and includes 95K+ instances, enabling analysis of how visual-geometric reasoning degrades with combinatorial complexity.
- Results show limited zero-shot geometric reasoning, a pronounced scaling cliff for both zero-shot and fine-tuned models, and strong dependence on shape cues in the ablation setting.