Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions

• URL: http://arxiv.org/abs/2405.19088v2
• Date: Mon, 28 Oct 2024 08:15:20 GMT
• Title: Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions
• Authors: Zhe Hu, Tuo Liang, Jing Li, Yiren Lu, Yunlai Zhou, Yiran Qiao, Jing Ma, Yu Yin,
• Abstract summary: This paper focuses on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics. Our results show that even state-of-the-art models still lag behind human performance on this task.
• Score: 16.23585043442914
• License:
• Abstract: Recent advancements in large multimodal language models have demonstrated remarkable proficiency across a wide range of tasks. Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particularly when it involves nonlinear narratives that underpin many jokes and humor cues. This paper investigates this challenge by focusing on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial or open-sourced large (vision) language models, we assess their capability to comprehend the complex interplay of the narrative humor inherent in these comics. Our results show that even state-of-the-art models still lag behind human performance on this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.

Related papers

• Visual-O1: Understanding Ambiguous Instructions via Multi-modal Multi-turn Chain-of-thoughts Reasoning [53.45295657891099]
  This paper proposes Visual-O1, a multi-modal multi-turn chain-of-thought reasoning framework. It simulates human multi-modal multi-turn reasoning, providing instantial experience for highly intelligent models. Our work highlights the potential of artificial intelligence to work like humans in real-world scenarios with uncertainty and ambiguity.
  arXiv  Detail & Related papers  (2024-10-04T11:18:41Z)
• Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! [14.84123301554462]
  We present UNPIE, a novel benchmark designed to assess the impact of multimodal inputs in resolving lexical ambiguities. Our dataset includes 1,000 puns, each accompanied by an image that explains both meanings. The results indicate that various Socratic Models and Visual-Language Models improve over the text-only models when given visual context.
  arXiv  Detail & Related papers  (2024-10-01T19:32:57Z)
• One missing piece in Vision and Language: A Survey on Comics Understanding [13.766672321462435]
  This survey is the first to propose a task-oriented framework for comics intelligence. It aims to guide future research by addressing critical gaps in data availability and task definition.
  arXiv  Detail & Related papers  (2024-09-14T18:26:26Z)
• LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models [60.67899965748755]
  We present LLaVA-Read, a multimodal large language model that utilizes dual visual encoders along with a visual text encoder. Our research suggests visual text understanding remains an open challenge and an efficient visual text encoder is crucial for future successful multimodal systems.
  arXiv  Detail & Related papers  (2024-07-27T05:53:37Z)
• Zero-Shot Character Identification and Speaker Prediction in Comics via Iterative Multimodal Fusion [35.25298023240529]
  We propose a novel zero-shot approach to identify characters and predict speaker names based solely on unannotated comic images. Our method requires no training data or annotations, it can be used as-is on any comic series.
  arXiv  Detail & Related papers  (2024-04-22T08:59:35Z)
• Exploring Chinese Humor Generation: A Study on Two-Part Allegorical Sayings [0.76146285961466]
  This paper investigates the capability of state-of-the-art language models to comprehend and generate Chinese humor. We employ two prominent training methods: fine-tuning a medium-sized language model and prompting a large one. Human-annotated results show that these models can generate humorous allegorical sayings, with prompting proving to be a practical and effective method.
  arXiv  Detail & Related papers  (2024-03-16T02:58:57Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest [70.40189243067857]
  Large neural networks can now generate jokes, but do they really "understand" humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest. We find that both types of models struggle at all three tasks.
  arXiv  Detail & Related papers  (2022-09-13T20:54:00Z)
• Visualizing and Explaining Language Models [0.0]
  Natural Language Processing has become, after Computer Vision, the second field of Artificial Intelligence. This paper showcases the techniques used in some of the most popular Deep Learning for NLP visualizations, with a special focus on interpretability and explainability.
  arXiv  Detail & Related papers  (2022-04-30T17:23:33Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.