Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning

• URL: http://arxiv.org/abs/2606.24548v3
• Date: Wed, 01 Jul 2026 03:29:55 GMT
• Title: Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
• Abstract summary: We investigate whether text-to-image models can generate images under rules that systematically contradict real-world priors.<n>CF-World organizes each scenario into three progressive levels: factual generation under ordinary world knowledge, explicit counterfactual generation with direct visual instructions, and implicit counterfactual generation requiring causal deduction from altered rules.
• Score: 39.26120643494408
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-image (T2I) generation models have achieved remarkable progress in producing visually realistic images from natural language prompts. Yet it remains unclear whether their success reflects genuine causal understanding or sophisticated pattern matching over visual-textual correlations. Inspired by Russell's inductivist turkey, we introduce Counterfactual-World (CF-World), a counterfactual benchmark designed to investigate whether text-to-image models can generate images under rules that systematically contradict real-world priors. CF-World organizes each scenario into three progressive levels: factual generation under ordinary world knowledge, explicit counterfactual generation with direct visual instructions, and implicit counterfactual generation requiring causal deduction from altered rules. We evaluate both open-source and closed-source T2I models using a Vision Language Model (VLM)-based evaluator (CF-Eval). Furthermore, we introduce two metrics: Prior Resistance Rate (PRR), which measures a models' ability to overcome entrenched real-world priors, and Reasoning Retention Rate (RRR), which assesses whether models can maintain reasoning-dependent counterfactual generation without explicit visual cues. Experiments show that all models exhibit sharp degradation from factual to counterfactual settings. Further analyses suggest that these failures arise because current T2I models encode world knowledge and visual appearances as tightly coupled patterns. Consequently, their heavy reliance on frequent visual co-occurrences within the training data forces them to default to familiar commonsense priors when tasked with rendering counterfactual worlds.

Related papers

• Benchmarking and Enhancing Text-to-Image Models for Generating Visual Representations in Early Arithmetic Education [57.241885377686486]
  We introduce equation-to-visual generation, a task that requires producing meaningful visuals from arithmetic equations.<n>We construct E2V-Bench, a benchmark spanning four pedagogically grounded visual types, along with automatic metrics for evaluating visual correctness.<n>Our evaluation reveals that recent text-to-image (T2I) models frequently fail on this task, with errors dominated by incorrect object counts and broken relational structure.
  arXiv  Detail & Related papers  (2026-05-29T12:18:08Z)
• FakeVLM-R1: Internalizing Physical Laws via CoT for Synthetic Image Detection [39.012577080756735]
  We propose FakeVLM-R1, aiming to endow the model with human-like critical thinking capabilities.<n>FakeVLM-R1 achieves high-precision, logically interpretable detection.<n>It also resolves the over-rejection bias of existing methods against real images.
  arXiv  Detail & Related papers  (2026-05-28T15:13:31Z)
• When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators [8.125052877783043]
  This work revisits the promise of synthetic data as a scalable substitute for real training sets.<n>We generate large-scale synthetic datasets using state-of-the-art T2I models released between 2022 and 2025.<n>Despite observable advances in visual fidelity and prompt adherence, classification accuracy on real test data consistently declines with newer T2I models as training data generators.
  arXiv  Detail & Related papers  (2026-02-23T15:15:53Z)
• Envision: Benchmarking Unified Understanding & Generation for Causal World Process Insights [41.385614383367205]
  Current models aim to transcend the limitations of single-modality representations by unifying understanding and generation.<n>Their reliance on static single-image generation in training and evaluation leads to overfitting to static pattern matching and semantic fusion.<n>We propose Envision-a causal event progression benchmark for chained text-to-multi-image generation.
  arXiv  Detail & Related papers  (2025-12-01T15:52:31Z)
• Beyond Words and Pixels: A Benchmark for Implicit World Knowledge Reasoning in Generative Models [15.983959465314749]
  We introduce PicWorld, the first comprehensive benchmark that assesses the grasp of implicit world knowledge and physical causal reasoning of T2I models.<n>This benchmark consists of 1,100 prompts across three core categories.<n>We conduct a thorough analysis of 17 mainstream T2I models on PicWorld, illustrating that they universally exhibit a fundamental limitation in their capacity for implicit world knowledge and physical causal reasoning to varying degrees.
  arXiv  Detail & Related papers  (2025-11-23T03:44:54Z)
• AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models [58.85362281293525]
  We introduce AcT2I, a benchmark designed to evaluate the performance of T2I models in generating images from action-centric prompts.<n>We experimentally validate that leading T2I models do not fare well on AcT2I.<n>We build upon this by developing a training-free, knowledge distillation technique utilizing Large Language Models to address this limitation.
  arXiv  Detail & Related papers  (2025-09-19T16:41:39Z)
• ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL [54.100889131719626]
  Chain-of-thought reasoning and reinforcement learning have driven breakthroughs in NLP.<n>We introduce ReasonGen-R1, a framework that imbues an autoregressive image generator with explicit text-based "thinking" skills.<n>We show that ReasonGen-R1 consistently outperforms strong baselines and prior state-of-the-art models.
  arXiv  Detail & Related papers  (2025-05-30T17:59:48Z)
• WorldGenBench: A World-Knowledge-Integrated Benchmark for Reasoning-Driven Text-to-Image Generation [38.196609962452655]
  textbfWorldGenBench is a benchmark designed to evaluate T2I models' world knowledge grounding and implicit inferential capabilities.<n>We propose the textbfKnowledge Checklist Score, a structured metric that measures how well generated images satisfy key semantic expectations.<n>Our findings highlight the need for deeper understanding and inference capabilities in next-generation T2I systems.
  arXiv  Detail & Related papers  (2025-05-02T17:59:06Z)
• RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning [54.07026389388881]
  We present the first real-object-based retrieval-augmented generation framework (RealRAG)<n>RealRAG augments fine-grained and unseen novel object generation by learning and retrieving real-world images to overcome the knowledge gaps of generative models.<n>Our framework integrates fine-grained visual knowledge for the generative models, tackling the distortion problem and improving the realism for fine-grained object generation.
  arXiv  Detail & Related papers  (2025-02-02T16:41:54Z)
• PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models [50.33699462106502]
  Text-to-image (T2I) models frequently fail to produce images consistent with physical commonsense. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge. We introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties.
  arXiv  Detail & Related papers  (2024-06-17T17:49:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.