Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

• URL: http://arxiv.org/abs/2604.15210v1
• Date: Thu, 16 Apr 2026 16:41:13 GMT
• Title: Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding
• Abstract summary: We introduce IRS, a framework that decomposes humor understanding into three components.<n>IRS supervises intermediate reasoning process through structured traces.<n>Our results suggest that supervising reasoning structure, rather than scale alone, is key for reasoning-centric tasks.
• Score: 13.744630706260743
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humor is one of the few cognitive tasks where getting the reasoning right matters as much as getting the answer right. While recent work evaluates humor understanding on benchmarks such as the New Yorker Cartoon Caption Contest (NYCC), it largely treats it as black-box prediction, overlooking the structured reasoning processes underlying humor comprehension. We introduce IRS (Incongruity-Resolution Supervision), a framework that decomposes humor understanding into three components: incongruity modeling, which identifies mismatches in the visual scene; resolution modeling, which constructs coherent reinterpretations of these mismatches; and preference alignment, which evaluates candidate interpretations under human judgments. Grounded in incongruity-resolution theory and expert captionist practice, IRS supervises intermediate reasoning process through structured traces that make the path from visual perception to humorous interpretation explicit and learnable. Across 7B, 32B, and 72B models on NYCC, IRS outperforms strong open and closed multimodal baselines across caption matching and ranking tasks, with our largest model approaching expert-level performance on ranking. Zero-shot transfer to external benchmarks shows that IRS learns generalizable reasoning patterns. Our results suggest that supervising reasoning structure, rather than scale alone, is key for reasoning-centric tasks.

Related papers

• CausalSplat: Towards Comprehensive Hierarchical Reasoning in 3D Gaussian Splatting [19.87557814775208]
  CausalSplat is a framework that integrates vision-language models with 3D scene graphs to disentangle explicit structural perception from implicit logical inference.<n>Our benchmarks evaluate commonsense, spatial, affordance, and counterfactual reasoning.<n>Experiments demonstrate that CausalSplat achieves state of the art performance on our reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-08-11T17:09:49Z)
• Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment [67.2775785848366]
  Two key failure modes are prominent: hallucinated chain-of-thought (CoT) reasoning that contradicts physical reality, and misalignment between the model's reasoning and actions.<n>We present VAORA, a novel reward design that directly addresses both issues.<n>VAORA introduces two complementary rewards: Visual Alignment Reward, which anchors VLM reasoning to the visual context independent of the agent action itself, and Visual-Action Alignment Reward, which grounds reasoning in the visual outcome induced by the model's action.
  arXiv  Detail & Related papers  (2026-07-07T17:27:59Z)
• H-GRPO: Permutation-Invariant Reinforcement Learning for Grounded Visual Reasoning [14.276463740862825]
  Vision-Language Models (VLMs) often achieve high performance on benchmarks while remaining "black boxes"<n>We propose a framework designed to enhance both performance and interpretability through De-compositional Evidence Grounding.
  arXiv  Detail & Related papers  (2026-06-29T07:51:25Z)
• Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning [82.86343313807158]
  We argue that a central challenge is how and when visual evidence is introduced into the reasoning process.<n>Motivated by this insight, we propose CSMR, a multimodal reasoning framework in which a language model controls the reasoning process.
  arXiv  Detail & Related papers  (2026-05-27T08:43:13Z)
• Think Visually, Reason Textually: Vision-Language Synergy in ARC [94.15522924153264]
  ARC-AGI is a rigorous testbed for conceptual rule induction and transfer to novel tasks.<n>Most existing methods treat ARC-AGI as a purely textual reasoning task, overlooking the fact that humans rely heavily on visual abstraction.<n>We introduce two synergistic strategies: Vision-Language Synergy Reasoning (VLSR) and Modality-Switch Self-Correction (MSSC)<n>Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence.
  arXiv  Detail & Related papers  (2025-11-19T18:59:04Z)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception [67.89135437537179]
  We introduce BLINK-Twice, a vision-centric reasoning benchmark grounded in challenging perceptual tasks.<n>Instead of relying on external knowledge, our tasks require models to reason from visual content alone.<n>Compared to prior perception benchmarks, it moves beyond shallow perception and requires fine-grained observation and analytical reasoning.
  arXiv  Detail & Related papers  (2025-10-10T13:14:13Z)
• Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens [19.316594303998667]
  Reasoning tasks have long been regarded as rigorous benchmarks for assessing the capabilities of machine learning models.<n>We revisit reasoning tasks from a causal perspective, seeking to understand their behavior in latent space.<n>We introduce a framework, called SR$2$, that incorporates the estimated latent variables as feedback into the selection mechanism.
  arXiv  Detail & Related papers  (2025-10-09T13:45:31Z)
• Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench [16.929265302194782]
  HumorBench is a benchmark designed to evaluate large language models' (LLMs) ability to reason about and explain sophisticated humor in cartoon captions.<n>LLMs are evaluated based on their explanations towards the humor and abilities in identifying the joke elements.
  arXiv  Detail & Related papers  (2025-07-29T03:44:43Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• How do Transformers Learn Implicit Reasoning? [67.02072851088637]
  We study how implicit multi-hop reasoning emerges by training transformers from scratch in a controlled symbolic environment.<n>We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures.
  arXiv  Detail & Related papers  (2025-05-29T17:02:49Z)
• A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs [3.2228025627337864]
  This paper introduces a structured evaluation framework to dissect the perception-reasoning interface in Vision-Language Models (VLMs)<n>We propose three distinct evaluation paradigms, mirroring human problem-solving strategies.<n>Applying this framework, we demonstrate that CA, leveraging powerful language models for reasoning over rich, independently generated descriptions, achieves new state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2025-01-23T12:42:42Z)
• Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models [36.119299938503936]
  Large vision-language models (LVLMs) have shown promising performance on a variety of vision-language tasks. They remain susceptible to hallucinations, generating outputs misaligned with visual content or instructions. We propose reflective instruction tuning, which integrates rationale learning into visual instruction tuning.
  arXiv  Detail & Related papers  (2024-07-16T06:32:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.