Mix of Experts Language Model for Named Entity Recognition

• URL: http://arxiv.org/abs/2404.19192v1
• Date: Tue, 30 Apr 2024 01:41:03 GMT
• Title: Mix of Experts Language Model for Named Entity Recognition
• Authors: Xinwei Chen, Kun Li, Tianyou Song, Jiangjian Guo,
• Abstract summary: We propose a robust NER model named BOND-MoE based on Mixture of Experts (MoE) Instead of relying on a single model for NER prediction, multiple models are trained and ensembled under the Expectation-Maximization framework. Experiments on real-world datasets show that the proposed method achieves state-of-the-art performance compared with other distantly supervised NER.
• Score: 4.120505838411977
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Named Entity Recognition (NER) is an essential steppingstone in the field of natural language processing. Although promising performance has been achieved by various distantly supervised models, we argue that distant supervision inevitably introduces incomplete and noisy annotations, which may mislead the model training process. To address this issue, we propose a robust NER model named BOND-MoE based on Mixture of Experts (MoE). Instead of relying on a single model for NER prediction, multiple models are trained and ensembled under the Expectation-Maximization (EM) framework, so that noisy supervision can be dramatically alleviated. In addition, we introduce a fair assignment module to balance the document-model assignment process. Extensive experiments on real-world datasets show that the proposed method achieves state-of-the-art performance compared with other distantly supervised NER.

Related papers

• Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning [78.72226641279863]
  Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. Our research explores task-specific model pruning to inform decisions about designing SMoE architectures. We introduce an adaptive task-aware pruning technique UNCURL to reduce the number of experts per MoE layer in an offline manner post-training.
  arXiv  Detail & Related papers  (2024-09-02T22:35:03Z)
• Robust Latent Representation Tuning for Image-text Classification [9.789498730131607]
  We propose a robust latent representation tuning method for large models. Our approach introduces a modality latent translation module to maximize the correlation between modalities, resulting in a robust representation. Within this framework, common semantics are refined during training, and robust performance is achieved even in the absence of one modality.
  arXiv  Detail & Related papers  (2024-06-10T06:29:00Z)
• A Large-Scale Evaluation of Speech Foundation Models [110.95827399522204]
  We establish the Speech processing Universal PERformance Benchmark (SUPERB) to study the effectiveness of the foundation model paradigm for speech. We propose a unified multi-tasking framework to address speech processing tasks in SUPERB using a frozen foundation model followed by task-specialized, lightweight prediction heads.
  arXiv  Detail & Related papers  (2024-04-15T00:03:16Z)
• SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities [10.193908215351497]
  We propose SCANNER, a model capable of effectively handling all three NER variants. SCANNER is a two-stage structure; we extract entity candidates in the first stage and use it as a query to get knowledge. To tackle the challenges arising from noisy annotations in NER datasets, we introduce a novel self-distillation method.
  arXiv  Detail & Related papers  (2024-04-02T13:05:41Z)
• Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer [59.43462055143123]
  The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning. In this study, we shed light on the homogeneous representation problem, wherein experts in the MoE fail to specialize and lack diversity. We propose an alternating training strategy that encourages each expert to update in a direction to the subspace spanned by other experts.
  arXiv  Detail & Related papers  (2023-10-15T07:20:28Z)
• Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
  We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
  arXiv  Detail & Related papers  (2023-03-13T16:00:31Z)
• MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective [57.19660234992812]
  NER model has achieved promising performance on standard NER benchmarks. Recent studies show that previous approaches may over-rely on entity mention information, resulting in poor performance on out-of-vocabulary (OOV) entity recognition. We propose MINER, a novel NER learning framework, to remedy this issue from an information-theoretic perspective.
  arXiv  Detail & Related papers  (2022-04-09T05:18:20Z)
• RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models [32.806292167848156]
  We propose RockNER to audit the robustness of named entity recognition models. We replace target entities with other entities of the same semantic class in Wikidata. At the context level, we use pre-trained language models to generate word substitutions.
  arXiv  Detail & Related papers  (2021-09-12T21:30:21Z)
• Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training [66.80558875393565]
  We study the problem of training named entity recognition (NER) models using only distantly-labeled data. We propose a noise-robust learning scheme comprised of a new loss function and a noisy label removal step. Our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.
  arXiv  Detail & Related papers  (2021-09-10T17:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.