TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models

• URL: http://arxiv.org/abs/2603.00206v1
• Date: Fri, 27 Feb 2026 11:45:26 GMT
• Title: TACIT Benchmark: A Programmatic Visual Reasoning Benchmark for Generative and Discriminative Models
• Abstract summary: TACIT Benchmark is a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains.<n>The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing visual reasoning benchmarks predominantly rely on natural language prompts, evaluate narrow reasoning modalities, or depend on subjective scoring procedures such as LLM-as-judge. We introduce the TACIT Benchmark, a programmatic visual reasoning benchmark comprising 10 tasks across 6 reasoning domains: spatial navigation, abstract pattern completion, causal simulation, logical constraint satisfaction, graph theory, and topology. The benchmark provides dual-track evaluation: a generative track in which models must produce solution images verified through deterministic computer-vision pipelines, and a discriminative track offering five-way multiple choice with structurally plausible near-miss distractors. Each distractor violates exactly one structural constraint, requiring models to reason about fine-grained visual differences rather than exploit superficial cues. Version 0.1.0 distributes 6,000 puzzles (108,000 PNG images across three resolutions) with fully deterministic seeded generation and reproducible verification. The dataset, generation code, and evaluation harness are released under the Apache 2.0 license on HuggingFace (DOI: 10.57967/hf/7904).

Related papers

• ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI [14.957780321740394]
  ERQA-Plus is a diagnostic benchmark for reasoning in embodied AI.<n>It contains 1,766 question-answer instances grounded in 711 robot-centric images.
  arXiv  Detail & Related papers  (2026-06-16T07:56:33Z)
• LADBench: A Benchmark for Logical Fault Detection in Images [1.6882040908691864]
  Large Vision Language Models (VLMs) excel at visual question answering and semantic grounding, but their capacity for autonomous logical reasoning remains underexplored.<n>Existing anomaly benchmarks emphasize visual errors or direct prompting rather than the physical and social common sense needed for open-world deployment.<n>We introduce LAD-bench, a benchmark of more than 1,000 curated synthetic images with logical anomalies across four domains.
  arXiv  Detail & Related papers  (2026-06-16T02:32:38Z)
• The Abstraction Gap in Vision-Language Causal Reasoning [0.5156484100374058]
  Vision-language models (VLMs) generate fluent causal explanations.<n>Current evaluations cannot distinguish linguistic plausibility from faithful causal reasoning.<n>We introduce a dual-probe methodology that isolates these properties.
  arXiv  Detail & Related papers  (2026-05-27T17:38:10Z)
• OrgForge-IT: A Verifiable Synthetic Benchmark for LLM-Based Insider Threat Detection [0.0]
  We present a verifiable synthetic benchmark in which a deterministic simulation engine maintains ground truth and language models generate only surface prose.<n>The corpus spans 51 simulated days, 2,904 telemetry records at a 96.4% noise rate, and four detection scenarios designed to defeat single-surface and single-day triage strategies.
  arXiv  Detail & Related papers  (2026-03-23T19:03:53Z)
• LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification [24.91906506651266]
  We introduce LogicGraph, the first benchmark aimed to systematically evaluate multi-path logical reasoning.<n>This pipeline yields solver-verified reasoning problems formalized by high-depth multi-path reasoning.<n>We propose a reference-free evaluation framework to rigorously assess model performance in both convergent and divergent regimes.
  arXiv  Detail & Related papers  (2026-02-24T16:04:26Z)
• Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies [40.03295633717008]
  We introduce VIA-Bench, a benchmark designed to probe model performance on visual illusions and anomalies.<n>We construct over 1K high-quality question-answer pairs that require nuanced visual reasoning.<n>Our findings reveal a fundamental divergence between machine and human perception, suggesting that resolving such perceptual bottlenecks is critical for the advancement of artificial general intelligence.
  arXiv  Detail & Related papers  (2026-02-02T08:48:03Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demand [7.223768474709172]
  We introduce OpenExempt, a framework and benchmark for diagnostic evaluation of legal reasoning.<n>The OpenExempt Framework uses expert-crafted symbolic representations of U.S. Bankruptcy Code statutes.<n>Using this system, we construct the OpenExempt Benchmark, a diagnostic benchmark for legal reasoning with 9,765 samples.
  arXiv  Detail & Related papers  (2026-01-19T16:07:47Z)
• SliceLens: Fine-Grained and Grounded Error Slice Discovery for Multi-Instance Vision Tasks [16.05135819343667]
  We propose SliceLens, a hypothesis-driven framework to generate and verify diverse failure hypotheses.<n>We also introduce FeSD, the first benchmark specifically designed for evaluating fine-grained error slice discovery.
  arXiv  Detail & Related papers  (2025-12-31T03:28:41Z)
• Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark [112.46338388724116]
  We introduce the Visual Reasoning Tracer (VRT) task, which requires models to not only localize the target object but also explicitly predict the intermediate objects that form the reasoning path.<n>To advance research in this area, we contribute: (1) VRT-Bench, a human-annotated benchmark for evaluating visual reasoning; (2) a new metric for assessing the quality of reasoning traces; and (3) VRT-80k, a large-scale dataset for reasoning model training.
  arXiv  Detail & Related papers  (2025-12-04T18:55:34Z)
• VCoT-Grasp: Grasp Foundation Models with Visual Chain-of-Thought Reasoning for Language-driven Grasp Generation [67.98487725287835]
  VCoT-Grasp is an end-to-end grasp foundation model that incorporates visual chain-of-thought reasoning to enhance visual understanding for grasp generation.<n>For training, we refine and introduce a large-scale dataset, VCoT-GraspSet, comprising 167K synthetic images with over 1.36M grasps.<n>Our method significantly improves grasp success rates and generalizes effectively to unseen objects, backgrounds, and distractors.
  arXiv  Detail & Related papers  (2025-10-07T11:50:26Z)
• MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval [86.35779264575154]
  Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios.<n>We introduce MR$2$-Bench, a reasoning-intensive benchmark for multimodal retrieval.
  arXiv  Detail & Related papers  (2025-09-30T15:09:14Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps [56.76175383189738]
  We introduce ReasonMap, a benchmark designed to assess the fine-grained visual understanding and spatial reasoning abilities of MLLMs.<n>ReasonMap encompasses high-resolution transit maps from 30 cities across 13 countries and includes 1,008 question-answer pairs spanning two question types and three templates.<n> Comprehensive evaluations of 15 popular MLLMs, including both base and reasoning variants, reveal a counterintuitive pattern.
  arXiv  Detail & Related papers  (2025-05-24T12:33:52Z)
• Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models [58.64449765678416]
  We introduce landscape of thoughts (LoT) to inspect the reasoning trajectories with certain reasoning methods on any multi-choice dataset.<n>LoT distinguishes between strong and weak models, correct and incorrect answers, as well as different reasoning tasks.<n>We showcase this advantage by adapting LoT to a lightweight verifier that evaluates the correctness of trajectories.
  arXiv  Detail & Related papers  (2025-03-28T06:09:51Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.