SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation

• URL: http://arxiv.org/abs/2606.03788v1
• Date: Tue, 02 Jun 2026 15:39:33 GMT
• Title: SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation
• Abstract summary: We shift the focus from Sign Language Translation (SLT) to Sign Language Understanding (SLU)<n>We evaluate systems based on their ability to correctly recover, from the input video, key semantic aspects of the original sentence.<n>We show the potential of SLU-2K by evaluating popular Multimodal Large Language Models (MLLMs) and two representative state-of-the-art systems, MMSTL and SpaMo.
• Score: 16.817821981397277
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sign Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directly measure whether a translation preserves the meaning of the source sign sequence. This is in contrast with the final objective of integrating SLT in assistive technology. In this work, we shift the focus from Sign Language Translation (SLT) to Sign Language Understanding (SLU), with particular emphasis on semantic understanding. Specifically, we evaluate systems based on their ability to correctly recover, from the input video, key semantic aspects of the original sentence, such as actions taking place and facts about people and objects. To enable this evaluation systematically, we propose SLU-2K, a dataset of 2,350 closed-ended video question-answer pairs based on the popular PHOENIX-2014T and CSL-Daily datasets. To obtain SLU-2K, we propose and extensively evaluate an automated data generation pipeline which produces questions across 7 categories, namely actions, locations, numbers, objects, people, time, and weather conditions. We show the potential of SLU-2K by evaluating popular Multimodal Large Language Models (MLLMs) and two representative state-of-the-art systems, MMSTL and SpaMo. Our results show that MLLMs reach near-random performance, highlighting the need for a more systematic integration of SLU in current AI systems. Furthermore, state-of-the-art translation systems carefully fine-tuned on in-domain data still exhibit a substantial semantic gap, with results ranging from 56.7% to 75.2%. These findings suggest that current SLT evaluation protocols overestimate true understanding and that future progress should be measured not only by fluency and n-gram overlap, but also by semantic correctness. Code, prompts, and benchmark files are available at https://github.com/ZenoTsT/SLU-2K

Related papers

• LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal [8.33491432828288]
  We study whether query-document relevance is linearly decodable from residual-stream activations in instruction-tuned language models.<n>Our results suggest that q-d relevance is encoded as a depth-dependent signal.<n>In several models, validation-selected probes match or outperform generated judgments and better preserve system rankings.
  arXiv  Detail & Related papers  (2026-07-17T01:54:45Z)
• In the LLM era, Word Sense Induction remains unsolved [5.5822932775420115]
  We propose an evaluation on a SemCor-derived dataset, respecting the original corpus polysemy and frequency distributions.<n>We assess pre-trained embeddings and clustering algorithms across parts of speech, and propose and evaluate an LLM-based WSI method for English.
  arXiv  Detail & Related papers  (2026-03-12T08:50:45Z)
• Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding [51.010563573083495]
  We introduce VRSLU, a novel dataset that integrates both Visual images and explicit Reasoning.<n>For over-idealized CA, we use GPT-4o and FLUX.1-dev to generate images reflecting users' environments and statuses, followed by human verification to ensure quality.<n>For reasoning, GPT-4o is employed to generate explanations for predicted labels, which are then refined by human annotators to ensure accuracy and coherence.
  arXiv  Detail & Related papers  (2025-11-24T11:32:24Z)
• Uni-Sign: Toward Unified Sign Language Understanding at Scale [90.76641997060513]
  We propose a unified pre-training framework that eliminates the gap between pre-training and downstream SLU tasks.<n>Uni-Sign achieves state-of-the-art performance across multiple downstream SLU tasks.
  arXiv  Detail & Related papers  (2025-01-25T11:51:23Z)
• Toward Informal Language Processing: Knowledge of Slang in Large Language Models [16.42982896928428]
  We construct a dataset that supports evaluation on a diverse set of tasks pertaining to automatic processing of slang. For both evaluation and finetuning, we show the effectiveness of our dataset on two core applications. We find that while LLMs such as GPT-4 achieve good performance in a zero-shot setting, smaller BERT-like models finetuned on our dataset achieve comparable performance.
  arXiv  Detail & Related papers  (2024-04-02T21:50:18Z)
• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding [56.756090143062536]
  We propose to use consistency regularization based on a hybrid data augmentation strategy. We conduct experiments on the MASSIVE dataset under both full-dataset and zero-shot settings. Our proposed method improves the performance on both intent detection and slot filling tasks.
  arXiv  Detail & Related papers  (2023-01-05T11:21:15Z)
• Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model [57.92200214957124]
  External language models (LMs) are used to improve the recognition performance of end-to-end (E2E) automatic speech recognition (ASR) systems. We propose a novel decoding algorithm where a word-level lattice is constructed on-the-fly to consider all possible word sequences. Our method consistently outperforms subword-level LMs, including N-gram LM and neural network LM.
  arXiv  Detail & Related papers  (2022-01-06T10:04:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.