DeformAr: Rethinking NER Evaluation through Component Analysis and Visual Analytics

• URL: http://arxiv.org/abs/2512.00938v1
• Date: Sun, 30 Nov 2025 15:39:28 GMT
• Title: DeformAr: Rethinking NER Evaluation through Component Analysis and Visual Analytics
• Authors: Ahmed Mustafa Younes,
• Abstract summary: We introduce DeformAr, a framework to investigate and explain the performance discrepancy between Arabic and English NER systems.<n>DeformAr is the first Arabic-specific, component-based interpretability tool, offering a crucial resource for advancing model analysis in under-resourced languages.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer models have significantly advanced Natural Language Processing (NLP), demonstrating strong performance in English. However, their effectiveness in Arabic, particularly for Named Entity Recognition (NER), remains limited, even with larger pre-trained models. This performance gap stems from multiple factors, including tokenisation, dataset quality, and annotation inconsistencies. Existing studies often analyze these issues in isolation, failing to capture their joint effect on system behaviour and performance. We introduce DeformAr (Debugging and Evaluation Framework for Transformer-based NER Systems), a novel framework designed to investigate and explain the performance discrepancy between Arabic and English NER systems. DeformAr integrates a data extraction library and an interactive dashboard, supporting two modes of evaluation: cross-component analysis and behavioural analysis. The framework divides each language into dataset and model components to examine their interactions. The analysis proceeds in two stages. First, cross-component analysis provides systematic diagnostic measures across data and model subcomponents, addressing the "what," "how," and "why" behind observed discrepancies. The second stage applies behavioural analysis by combining interpretability techniques with token-level metrics, interactive visualisations, and representation space analysis. These stages enable a component-aware diagnostic process that detects model behaviours and explains them by linking them to underlying representational patterns and data factors. DeformAr is the first Arabic-specific, component-based interpretability tool, offering a crucial resource for advancing model analysis in under-resourced languages.

Related papers

• Simplifying Outcomes of Language Model Component Analyses with ELIA [2.498098282753967]
  ELIA (Explainable Language Interpretability Analysis) is an interactive web application that simplifies the outcomes of various language model component analyses for a broader audience.<n>The system integrates three key techniques -- Attribution Analysis, Function Vector Analysis, and Circuit Tracing -- and introduces a novel methodology: using a vision-language model to automatically generate natural language explanations.
  arXiv  Detail & Related papers  (2026-02-20T14:45:27Z)
• Boosting Accuracy and Interpretability in Multilingual Hate Speech Detection Through Layer Freezing and Explainable AI [0.0]
  This study examines the performance of three transformer-based models for sentiment analysis and hate speech detection.<n>The evaluation is conducted across five languages: English, Korean, Japanese, Chinese, and French.<n>We integrate the Local Interpretable Model-agnostic Explanations (LIME) framework to highlight the contribution of individual words to the models decisions.
  arXiv  Detail & Related papers  (2026-01-06T04:07:51Z)
• VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents [39.42078665719841]
  VIDEE is a system that supports entry-level data analysts to conduct advanced text analytics with intelligent agents.<n>We conduct two quantitative experiments to evaluate VIDEE's effectiveness and analyze common agent errors.
  arXiv  Detail & Related papers  (2025-06-17T05:24:58Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• A Novel Energy based Model Mechanism for Multi-modal Aspect-Based Sentiment Analysis [85.77557381023617]
  We propose a novel framework called DQPSA for multi-modal sentiment analysis. PDQ module uses the prompt as both a visual query and a language query to extract prompt-aware visual information. EPE module models the boundaries pairing of the analysis target from the perspective of an Energy-based Model.
  arXiv  Detail & Related papers  (2023-12-13T12:00:46Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• Sentiment Analysis on Brazilian Portuguese User Reviews [0.0]
  This work analyzes the predictive performance of a range of document embedding strategies, assuming the polarity as the system outcome. This analysis includes five sentiment analysis datasets in Brazilian Portuguese, unified in a single dataset, and a reference partitioning in training, testing, and validation sets, both made publicly available through a digital repository.
  arXiv  Detail & Related papers  (2021-12-10T11:18:26Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.