HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models

• URL: http://arxiv.org/abs/2604.19786v1
• Date: Tue, 31 Mar 2026 18:54:15 GMT
• Title: HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
• Abstract summary: We introduce HumorRank, a tournament-based evaluation framework and leaderboard for textual humor generation.<n>Using SemEval-2026 MWAHAHA test dataset, we conduct an automated pairwise evaluation across nine models spanning proprietary, open-weight, and specialized systems.<n>Our results demonstrate that HumorRank yields statistically grounded model stratifications, showing that humor quality is driven by mastery of comedic mechanisms rather than model scale alone.
• Score: 13.570863502498336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating humor in large language models (LLMs) is an open challenge because existing approaches yield isolated, incomparable metrics rather than unified model rankings, making it difficult to track progress across systems. We introduce HumorRank, a tournament-based evaluation framework and leaderboard for textual humor generation. Using SemEval-2026 MWAHAHA test dataset, we conduct an extensive automated pairwise evaluation across nine models spanning proprietary, open-weight, and specialized systems. Pairwise judgments grounded in the General Theory of Verbal Humor (GTVH) are aggregated via an Adaptive Swiss tournament, with Bradley-Terry Maximum Likelihood Estimation (MLE) producing globally consistent humor generation capability rankings. Our results demonstrate that HumorRank yields statistically grounded model stratifications, showing that humor quality is driven by mastery of comedic mechanisms rather than model scale alone. HumorRank thus provides a scalable, interpretable methodology for benchmarking and understanding LLM-generated humor.

Related papers

• Multilingual Humour-Aware Retrieval with Dense and Re-Ranking Models [0.4640835690336653]
  Team DUTH studies multilingual humour-aware information retrieval using the CLEF 2025 JOKER Task 1 benchmark.<n>Our approach combines multilingual XLM-RoBERTa-based dense retrieval with additional system variants, including neural re-ranking.
  arXiv  Detail & Related papers  (2026-05-24T16:50:15Z)
• HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation [13.570863502498336]
  We introduce the Cognitive Synergy Framework, a methodology for generating high-quality humor data.<n>We deploy six cognitive personas to synthesize diverse comedic perspectives for a given prompt.<n>We find that cognitive-driven data is far more critical than alignment algorithms or model scale for humor generation.
  arXiv  Detail & Related papers  (2026-03-19T13:12:53Z)
• Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models [57.33350664910483]
  We introduce Squid Game, a dynamic and adversarial evaluation environment with resource-constrained and asymmetric information settings.<n>We evaluate over 50 LLMs on Squid Game, presenting the largest behavioral evaluation study of general LLMs on dynamic adversarial scenarios.
  arXiv  Detail & Related papers  (2025-11-12T06:06:29Z)
• Engagement Undermines Safety: How Stereotypes and Toxicity Shape Humor in Language Models [55.98686105081078]
  Large language models are increasingly used for creative writing and engagement content, raising safety concerns about the outputs.<n>This work evaluates how funniness optimization in modern LLM pipelines couples with harmful content by measuring humor, stereotypicality, and toxicity.
  arXiv  Detail & Related papers  (2025-10-21T09:28:09Z)
• V-HUB: A Visual-Centric Humor Understanding Benchmark for Video LLMs [72.59885036868499]
  v-HUB is a visual-centric video humor understanding benchmark.<n>Each video clip is paired with rich annotations, including captions, descriptions, and explanations.<n>We evaluate a diverse set of MLLMs, from specialized Video-LLMs to versatile OmniLLMs that can process audio.
  arXiv  Detail & Related papers  (2025-09-30T04:33:52Z)
• Which LLMs Get the Joke? Probing Non-STEM Reasoning Abilities with HumorBench [16.929265302194782]
  HumorBench is a benchmark designed to evaluate large language models' (LLMs) ability to reason about and explain sophisticated humor in cartoon captions.<n>LLMs are evaluated based on their explanations towards the humor and abilities in identifying the joke elements.
  arXiv  Detail & Related papers  (2025-07-29T03:44:43Z)
• From Punchlines to Predictions: A Metric to Assess LLM Performance in Identifying Humor in Stand-Up Comedy [6.124881326867511]
  In light of the widespread adoption of Large Language Models, the intersection of humor and AI has become no laughing matter.<n>In this study, we assess the ability of models in accurately identifying humorous quotes from a stand-up comedy transcript.<n>We propose a novel humor detection metric designed to evaluate LLMs amongst various prompts on their capability to extract humorous punchlines.
  arXiv  Detail & Related papers  (2025-04-12T02:19:53Z)
• Can Pre-trained Language Models Understand Chinese Humor? [74.96509580592004]
  This paper is the first work that systematically investigates the humor understanding ability of pre-trained language models (PLMs) We construct a comprehensive Chinese humor dataset, which can fully meet all the data requirements of the proposed evaluation framework. Our empirical study on the Chinese humor dataset yields some valuable observations, which are of great guiding value for future optimization of PLMs in humor understanding and generation.
  arXiv  Detail & Related papers  (2024-07-04T18:13:38Z)
• Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models [27.936545041302377]
  Large language models (LLMs) can generate synthetic data for humor detection via editing texts. We benchmark LLMs on an existing human dataset and show that current LLMs display an impressive ability to 'unfun' jokes. We extend our approach to a code-mixed English-Hindi humor dataset, where we find that GPT-4's synthetic data is highly rated by bilingual annotators.
  arXiv  Detail & Related papers  (2024-02-23T02:58:12Z)
• Towards Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First Results [84.37263300062597]
  Humor is a substantial element of human social behavior, affect, and cognition. Current methods of humor detection have been exclusively based on staged data, making them inadequate for "real-world" applications. We contribute to addressing this deficiency by introducing the novel Passau-Spontaneous Football Coach Humor dataset, comprising about 11 hours of recordings.
  arXiv  Detail & Related papers  (2022-09-28T17:36:47Z)
• Predicting the Humorousness of Tweets Using Gaussian Process Preference Learning [56.18809963342249]
  We present a probabilistic approach that learns to rank and rate the humorousness of short texts by exploiting human preference judgments and automatically sourced linguistic annotations. We report system performance for the campaign's two subtasks, humour detection and funniness score prediction, and discuss some issues arising from the conversion between the numeric scores used in the HAHA@IberLEF 2019 data and the pairwise judgment annotations required for our method.
  arXiv  Detail & Related papers  (2020-08-03T13:05:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.