Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions

• URL: http://arxiv.org/abs/2405.19088v1
• Date: Wed, 29 May 2024 13:51:43 GMT
• Title: Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions
• Authors: Zhe Hu, Tuo Liang, Jing Li, Yiren Lu, Yunlai Zhou, Yiran Qiao, Jing Ma, Yu Yin,
• Abstract summary: This paper focuses on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics. Our results show that even state-of-the-art models still lag behind human performance on this task.
• Score: 16.23585043442914
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advancements in large multimodal language models have demonstrated remarkable proficiency across a wide range of tasks. Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particularly when it involves nonlinear narratives that underpin many jokes and humor cues. This paper investigates this challenge by focusing on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial or open-sourced large (vision) language models, we assess their capability to comprehend the complex interplay of the narrative humor inherent in these comics. Our results show that even state-of-the-art models still lag behind human performance on this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.

Related papers

• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models [26.839159541015597]
  We develop novel benchmarks that cover diverse aspects of spatial reasoning. Our findings reveal several counter-intuitive insights that have been overlooked in the literature. We hope our study will inform the development of multimodal models to improve spatial intelligence.
  arXiv  Detail & Related papers  (2024-06-21T03:53:37Z)
• Zero-Shot Character Identification and Speaker Prediction in Comics via Iterative Multimodal Fusion [35.25298023240529]
  We propose a novel zero-shot approach to identify characters and predict speaker names based solely on unannotated comic images. Our method requires no training data or annotations, it can be used as-is on any comic series.
  arXiv  Detail & Related papers  (2024-04-22T08:59:35Z)
• Exploring Chinese Humor Generation: A Study on Two-Part Allegorical Sayings [0.76146285961466]
  This paper investigates the capability of state-of-the-art language models to comprehend and generate Chinese humor. We employ two prominent training methods: fine-tuning a medium-sized language model and prompting a large one. Human-annotated results show that these models can generate humorous allegorical sayings, with prompting proving to be a practical and effective method.
  arXiv  Detail & Related papers  (2024-03-16T02:58:57Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• Dense Multitask Learning to Reconfigure Comics [63.367664789203936]
  We develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels. Our method can successfully identify the semantic units as well as the notion of 3D in comic panels.
  arXiv  Detail & Related papers  (2023-07-16T15:10:34Z)
• Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest [70.40189243067857]
  Large neural networks can now generate jokes, but do they really "understand" humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest. We find that both types of models struggle at all three tasks.
  arXiv  Detail & Related papers  (2022-09-13T20:54:00Z)
• On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization [89.94078728495423]
  We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs. Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
  arXiv  Detail & Related papers  (2022-05-24T00:52:40Z)
• Visualizing and Explaining Language Models [0.0]
  Natural Language Processing has become, after Computer Vision, the second field of Artificial Intelligence. This paper showcases the techniques used in some of the most popular Deep Learning for NLP visualizations, with a special focus on interpretability and explainability.
  arXiv  Detail & Related papers  (2022-04-30T17:23:33Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
  We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing. Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
  arXiv  Detail & Related papers  (2021-12-16T05:36:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.