Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models

• URL: http://arxiv.org/abs/2608.25662v2
• Date: Fri, 28 Aug 2026 05:44:55 GMT
• Title: Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models
• Abstract summary: In 2026, we held the fourth iteration of the SHROOM-Visions task series: SHROOM-Visions (textbfShared-task on textbfHallucinations and textbfRelated textbfMistakes in textbfVision language modeltextbfs)<n>This time we aim to tackle hallucinations through a model-agnostic detection task focused on large vision-language models.<n>The evaluation uses a five-class taxonomy of hallucinations spanning four languages: Chinese, English, French
• Score: 19.79187126303009
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In 2026, we held the fourth iteration of the SHROOM Shared Task series: SHROOM-Visions (\textbf{S}hared-task on \textbf{H}allucinations and \textbf{R}elated \textbf{O}bservable \textbf{O}vergeneration \textbf{M}istakes in \textbf{Vision} language model\textbf{s}), which is hosted at the UncertaiNLP Workshop co-located with EMNLP 2026. Following the success of the 2024 and 2025 tasks, this time we aim to tackle hallucinations through a model-agnostic detection task focused on large vision-language models. Building on the recently introduced SHEEP dataset, designed for long-term evaluation across model generations, the task invites participants to detect and classify fine-grained hallucination spans in image-conditioned text generation (VQA, image captioning, etc.). The evaluation uses a five-class taxonomy of hallucinations spanning four languages: Chinese, English, French, and Italian. The shared task generated strong interest in the NLP community worldwide, with 27 teams contributing 600+ system submissions. The best systems achieve average scores of 0.58 in character-level correlation, 0.46 in label-conditioned correlation, and 0.51 in intersection-over-union (IoU) across four languages, outperforming the baselines by 30-40 points.

Related papers

• Findings of the Fifth Shared Task on Multilingual Coreference Resolution: Expanding Datasets for Long-Range Entities [1.5851688800400288]
  This paper describes the fifth edition of the Shared Task on Multilingual Coreference Resolution, held in conjunction with CODI-CRAC 2026.<n>The 2026 edition specifically emphasizes long-range entities, defined as coreferential chains spanning significant distances, across many words and sentences.<n>The task expanded its linguistic scope by incorporating five new datasets and two additional languages.
  arXiv  Detail & Related papers  (2026-05-20T16:35:09Z)
• DSC2025 -- ViHallu Challenge: Detecting Hallucination in Vietnamese LLMs [5.740643252319679]
  This paper introduces the DSC2025 ViHallu Challenge, the first large-scale shared task for detecting hallucinations in Vietnamese language models.<n>We present the ViHallu dataset, comprising 10,000 annotated triplets of (context, prompt, response) samples.<n>A total of 111 teams participated, with the best-performing system achieving a macro-F1 score of 84.80%, compared to a baseline encoder-only score of 32.83%.
  arXiv  Detail & Related papers  (2026-01-08T08:27:47Z)
• "AGI" team at SHROOM-CAP: Data-Centric Approach to Multilingual Hallucination Detection using XLM-RoBERTa [2.444311666637296]
  This paper describes our submission to the SHROOM-CAP 2025 shared task on scientific hallucination detection across 9 languages.<n>We unify and balance five existing datasets to create a comprehensive training corpus of 124,821 samples (50% correct, 50% hallucinated)<n>Our results demonstrate that systematic data curation can significantly outperform architectural innovations alone.
  arXiv  Detail & Related papers  (2025-11-23T05:48:27Z)
• SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Work [87.9341538630949]
  The first Sign Language Production Challenge was held as part of the third SLRTP Workshop at CVPR 2025.<n>The competition's aims are to evaluate architectures that translate from spoken language sentences to a sequence of skeleton poses.<n>This paper presents the challenge design and the winning methodologies.
  arXiv  Detail & Related papers  (2025-08-09T11:57:33Z)
• SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared Task on Hallucinations and Related Observable Overgeneration Mistakes [72.61348252096413]
  We present the Mu-SHROOM shared task which is focused on detecting hallucinations and other overgeneration mistakes in the output of instruction-tuned large language models (LLMs)<n>Mu-SHROOM addresses general-purpose LLMs in 14 languages, and frames the hallucination detection problem as a span-labeling task.<n>We received 2,618 submissions from 43 participating teams employing diverse methodologies.
  arXiv  Detail & Related papers  (2025-04-16T11:15:26Z)
• HausaNLP at SemEval-2025 Task 3: Towards a Fine-Grained Model-Aware Hallucination Detection [1.8230982862848586]
  We aim to provide a nuanced, model-aware understanding of hallucination occurrences and severity in English.<n>We used natural language inference and fine-tuned a ModernBERT model using a synthetic dataset of 400 samples.<n>Results indicate a moderately positive correlation between the model's confidence scores and the actual presence of hallucinations.
  arXiv  Detail & Related papers  (2025-03-25T13:40:22Z)
• SemEval-2024 Shared Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes [48.83290963506378]
  This paper presents the results of the SHROOM, a shared task focused on detecting hallucinations. We observe a number of key trends in how this approach was tackled. While a majority of the teams did outperform our proposed baseline system, the performances of top-scoring systems are still consistent with a random handling of the more challenging items.
  arXiv  Detail & Related papers  (2024-03-12T15:06:22Z)
• The All-Seeing Project V2: Towards General Relation Comprehension of the Open World [58.40101895719467]
  We present the All-Seeing Project V2, a new model and dataset designed for understanding object relations in images. We propose the All-Seeing Model V2 that integrates the formulation of text generation, object localization, and relation comprehension into a relation conversation task. Our model excels not only in perceiving and recognizing all objects within the image but also in grasping the intricate relation graph between them.
  arXiv  Detail & Related papers  (2024-02-29T18:59:17Z)
• mFACE: Multilingual Summarization with Factual Consistency Evaluation [79.60172087719356]
  Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent summaries. We leverage factual consistency evaluation models to improve multilingual summarization.
  arXiv  Detail & Related papers  (2022-12-20T19:52:41Z)
• e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks [52.918087305406296]
  We introduce e-ViL, a benchmark for evaluate explainable vision-language tasks. We also introduce e-SNLI-VE, the largest existing dataset with NLEs. We propose a new model that combines UNITER, which learns joint embeddings of images and text, and GPT-2, a pre-trained language model.
  arXiv  Detail & Related papers  (2021-05-08T18:46:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.