Chumor 2.0: Towards Benchmarking Chinese Humor Understanding

• URL: http://arxiv.org/abs/2412.17729v1
• Date: Mon, 23 Dec 2024 17:19:58 GMT
• Title: Chumor 2.0: Towards Benchmarking Chinese Humor Understanding
• Authors: Ruiqi He, Yushu He, Longju Bai, Jiarui Liu, Zhenjie Sun, Zenghao Tang, He Wang, Hanchen Xia, Rada Mihalcea, Naihao Deng,
• Abstract summary: Chumor is the first Chinese humor dataset that exceeds the size of existing humor datasets. Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes.
• Score: 23.370445567734798
• License:
• Abstract: Existing humor datasets and evaluations predominantly focus on English, leaving limited resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, the first Chinese humor explanation dataset that exceeds the size of existing humor datasets. Chumor is sourced from Ruo Zhi Ba, a Chinese Reddit-like platform known for sharing intellectually challenging and culturally specific jokes. We test ten LLMs through direct and chain-of-thought prompting, revealing that Chumor poses significant challenges to existing LLMs, with their accuracy slightly above random and far below human. In addition, our analysis highlights that human-annotated humor explanations are significantly better than those generated by GPT-4o and ERNIE-4-turbo. We release Chumor at https://huggingface.co/datasets/dnaihao/Chumor, our project page is at https://dnaihao.github.io/Chumor-dataset/, our leaderboard is at https://huggingface.co/spaces/dnaihao/Chumor, and our codebase is at https://github.com/dnaihao/Chumor-dataset.

Related papers

• Can Pre-trained Language Models Understand Chinese Humor? [74.96509580592004]
  This paper is the first work that systematically investigates the humor understanding ability of pre-trained language models (PLMs) We construct a comprehensive Chinese humor dataset, which can fully meet all the data requirements of the proposed evaluation framework. Our empirical study on the Chinese humor dataset yields some valuable observations, which are of great guiding value for future optimization of PLMs in humor understanding and generation.
  arXiv  Detail & Related papers  (2024-07-04T18:13:38Z)
• Is AI fun? HumorDB: a curated dataset and benchmark to investigate graphical humor [8.75275650545552]
  HumorDB is an image-only dataset specifically designed to advance visual humor understanding. The dataset enables evaluation through binary classification, range regression, and pairwise comparison tasks. HumorDB shows potential as a valuable benchmark for powerful large multimodal models.
  arXiv  Detail & Related papers  (2024-06-19T13:51:40Z)
• Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Ba [7.878358092927338]
  We construct Chumor, a dataset sourced from Ruo Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing intellectually challenging and culturally specific jokes. We annotate explanations for each joke and evaluate human explanations against two state-of-the-art LLMs, GPT-4o and ERNIE Bot. Our evaluation shows that Chumor is challenging even for SOTA LLMs, and the human explanations for Chumor jokes are significantly better than explanations generated by the LLMs.
  arXiv  Detail & Related papers  (2024-06-18T16:22:05Z)
• Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models [27.936545041302377]
  Large language models (LLMs) can generate synthetic data for humor detection via editing texts. We benchmark LLMs on an existing human dataset and show that current LLMs display an impressive ability to 'unfun' jokes. We extend our approach to a code-mixed English-Hindi humor dataset, where we find that GPT-4's synthetic data is highly rated by bilingual annotators.
  arXiv  Detail & Related papers  (2024-02-23T02:58:12Z)
• ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models [19.399535453449488]
  OpenAI's ChatGPT model almost seems to communicate on a human level and can even tell jokes. In a series of exploratory experiments around jokes, i.e., generation, explanation, and detection, we seek to understand ChatGPT's capability to grasp and reproduce human humor. Our empirical evidence indicates that jokes are not hard-coded but mostly also not newly generated by the model.
  arXiv  Detail & Related papers  (2023-06-07T16:10:21Z)
• The Naughtyformer: A Transformer Understands Offensive Humor [63.05016513788047]
  We introduce a novel jokes dataset filtered from Reddit and solve the subtype classification task using a finetuned Transformer dubbed the Naughtyformer. We show that our model is significantly better at detecting offensiveness in jokes compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-11-25T20:37:58Z)
• ExPUNations: Augmenting Puns with Keywords and Explanations [88.58174386894913]
  We augment an existing dataset of puns with detailed crowdsourced annotations of keywords. This is the first humor dataset with such extensive and fine-grained annotations specifically for puns. We propose two tasks: explanation generation to aid with pun classification and keyword-conditioned pun generation.
  arXiv  Detail & Related papers  (2022-10-24T18:12:02Z)
• Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results [84.37263300062597]
  Humor is a substantial element of human social behavior, affect, and cognition. Current methods of humor detection have been exclusively based on staged data, making them inadequate for "real-world" applications. We contribute to addressing this deficiency by introducing the novel Passau-Spontaneous Football Coach Humor dataset, comprising about 11 hours of recordings.
  arXiv  Detail & Related papers  (2022-09-28T17:36:47Z)
• Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest [70.40189243067857]
  Large neural networks can now generate jokes, but do they really "understand" humor? We challenge AI models with three tasks derived from the New Yorker Cartoon Caption Contest. We find that both types of models struggle at all three tasks.
  arXiv  Detail & Related papers  (2022-09-13T20:54:00Z)
• M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations [72.81164101048181]
  We propose a dataset for Multimodal Multiparty Hindi Humor (M2H2) recognition in conversations containing 6,191 utterances from 13 episodes of a very popular TV series "Shrimaan Shrimati Phir Se" Each utterance is annotated with humor/non-humor labels and encompasses acoustic, visual, and textual modalities. The empirical results on M2H2 dataset demonstrate that multimodal information complements unimodal information for humor recognition.
  arXiv  Detail & Related papers  (2021-08-03T02:54:09Z)
• Let's be Humorous: Knowledge Enhanced Humor Generation [26.886255899651893]
  We explore how to generate a punchline given the set-up with the relevant knowledge. To our knowledge, this is the first attempt to generate punchlines with knowledge enhanced model. The experimental results demonstrate that our method can make use of knowledge to generate fluent, funny punchlines.
  arXiv  Detail & Related papers  (2020-04-28T06:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.