CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language

• URL: http://arxiv.org/abs/2604.22367v1
• Date: Fri, 24 Apr 2026 08:59:33 GMT
• Title: CNSL-bench: Benchmarking the Sign Language Understanding Capabilities of MLLMs on Chinese National Sign Language
• Abstract summary: We introduce the first comprehensive Chinese emNational Sign Language benchmark designed for evaluating large language models (MLLMs) in sign language understanding.<n>The proposed CNSL-bench is characterized by: 1) Authoritative grounding, as it is anchored to the officially standardized textitNational Common Sign Language Dictionary.<n>Using CNSL-bench, we extensively evaluate 21 open-source and proprietary up-to-date MLLMs.
• Score: 29.349164636469187
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sign language research has achieved significant progress due to the advances in large language models (LLMs). However, the intrinsic ability of LLMs to understand sign language, especially in multimodal contexts, remains underexplored. To address this limitation, we introduce CNSL-bench, the first comprehensive Chinese em{National Sign Language benchmark designed for evaluating multimodal large language models (MLLMs) in sign language understanding. The proposed CNSL-bench is characterized by: 1) Authoritative grounding, as it is anchored to the officially standardized \textit{National Common Sign Language Dictionary, mitigating ambiguity from regional or non-canonical variants and ensuring consistent semantic definitions; 2) Multimodal coverage, providing aligned textual descriptions, illustrative images, and sign language videos; and 3) Articulatory diversity, supporting fine-grained analysis across key manual articulatory forms, including air-writing, finger-spelling, and the Chinese manual-alphabet. Using CNSL-bench, we extensively evaluate 21 open-source and proprietary up-to-date MLLMs. Our results reveal that, despite recent advances in multimodal modeling, current MLLMs remain substantially inferior to human performance, exhibiting systematic disparities across input modalities and manual articulatory forms. Additional diagnostic analyses suggest that several performance limitations persist beyond improvements in reasoning and that instruction-following robustness varies substantially across models.

Related papers

• Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding [88.11781604392606]
  This paper defines language representation as the linguistic and symbolic constructs used to map and model the real world.<n>We argue that shaping schemas through advanced language representation is the next frontier for expanding Large Language Models.
  arXiv  Detail & Related papers  (2026-05-10T02:42:29Z)
• LLM Probe: Evaluating LLMs for Low-Resource Languages [6.177998679139308]
  This paper presents a lexicon-based assessment framework designed to evaluate the linguistic skills of large language models (LLMs) in low-resource languages.<n>The framework analyzes models across four areas of language understanding: lexical alignment, part-of-speech recognition, morphosyntactic probing, and translation accuracy.
  arXiv  Detail & Related papers  (2026-03-31T10:03:38Z)
• Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios [5.599792629509229]
  AulSign is a novel method that leverages Large Language Models via dynamic prompting and in-context learning with sample selection and subsequent sign association.<n>We evaluate our method on both English and Italian languages using SignBank+, a recognized benchmark in the field, as well as the Italian LaCAM CNR-ISTC dataset.
  arXiv  Detail & Related papers  (2025-08-25T16:36:36Z)
• MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks [37.069856351074385]
  We introduce MCIF (Multimodal Crosslingual Instruction Following), the first multilingual human-annotated benchmark based on scientific talks.<n>It is designed to evaluate instruction-following in crosslingual, multimodal settings over both short- and long-form inputs.<n>It spans three core modalities -- speech, vision, and text -- and four diverse languages (English, German, Italian, and Chinese)
  arXiv  Detail & Related papers  (2025-07-25T19:00:51Z)
• How does a Multilingual LM Handle Multiple Languages? [0.0]
  This study critically examines capabilities in multilingual understanding, semantic representation, and cross-lingual knowledge transfer.<n>It assesses semantic similarity by analyzing multilingual word embeddings for consistency using cosine similarity.<n>It examines BLOOM-1.7B and Qwen2 through Named Entity Recognition and sentence similarity tasks to understand their linguistic structures.
  arXiv  Detail & Related papers  (2025-02-06T18:08:14Z)
• A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers [51.8203871494146]
  The rapid development of Large Language Models (LLMs) demonstrates remarkable multilingual capabilities in natural language processing.<n>Despite the breakthroughs of LLMs, the investigation into the multilingual scenario remains insufficient.<n>This survey aims to help the research community address multilingual problems and provide a comprehensive understanding of the core concepts, key techniques, and latest developments in multilingual natural language processing based on LLMs.
  arXiv  Detail & Related papers  (2024-05-17T17:47:39Z)
• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias [5.096332588720052]
  We present an overview of MLLMs, covering their evolutions, key techniques, and multilingual capacities.<n>Thirdly, we survey the state-of-the-art studies of multilingual representations and investigate whether the current MLLMs can learn a universal language representation.<n>Fourthly, we discuss bias on MLLMs, including its categories, evaluation metrics, and debiasing techniques.
  arXiv  Detail & Related papers  (2024-04-01T05:13:56Z)
• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models [117.20416338476856]
  Large language models (LLMs) demonstrate remarkable multilingual capabilities without being pre-trained on specially curated multilingual parallel corpora. We propose a novel detection method, language activation probability entropy (LAPE), to identify language-specific neurons within LLMs. Our findings indicate that LLMs' proficiency in processing a particular language is predominantly due to a small subset of neurons.
  arXiv  Detail & Related papers  (2024-02-26T09:36:05Z)
• A Multi-level Supervised Contrastive Learning Framework for Low-Resource Natural Language Inference [54.678516076366506]
  Natural Language Inference (NLI) is a growingly essential task in natural language understanding. Here we propose a multi-level supervised contrastive learning framework named MultiSCL for low-resource natural language inference.
  arXiv  Detail & Related papers  (2022-05-31T05:54:18Z)
• Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding [90.87454350016121]
  We develop novel code-switching schemes to generate hard negative examples for contrastive learning at all levels. We develop a label-aware joint model to leverage label semantics for cross-lingual knowledge transfer.
  arXiv  Detail & Related papers  (2022-05-07T13:44:28Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.