LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts

• URL: http://arxiv.org/abs/2602.14060v1
• Date: Sun, 15 Feb 2026 09:18:22 GMT
• Title: LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts
• Abstract summary: LM-Lexicon is an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging.<n>Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.
• Score: 9.056157892858474
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.

Related papers

• Interpretable and Fair Generalized Additive Neural Networks via Multi-objective Learning [52.66717986145448]
  Interpretability and fairness are two of the most emphasized dimensions in trustworthy artificial intelligence (AI)<n>This paper focuses on neural network (NN)-based generalized additive models (GAMs), a class of self-interpretable models.
  arXiv  Detail & Related papers  (2026-09-05T07:29:42Z)
• Enhancing Building Semantics Preservation in AI Model Training with Large Language Model Encodings [1.2278711963663465]
  This study proposes a novel training approach that employs large language model (LLM) embeddings as encodings to preserve finer distinctions in building semantics.<n>We evaluated the proposed method by training GraphSAGE models to classify 42 building object subtypes across five high-rise residential building information models.<n> Experimental results demonstrated that LLM encodings outperformed the conventional one-hot baseline, with the llama-3 embedding achieving a weighted average F1-score of 0.8766, compared to 0.8475 for one-hot encoding.
  arXiv  Detail & Related papers  (2026-02-17T18:26:36Z)
• Expert Divergence Learning for MoE-based Language Models [21.857507958048696]
  We introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts.<n>Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora.<n>We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch.
  arXiv  Detail & Related papers  (2026-02-10T06:58:38Z)
• ERNIE 5.0 Technical Report [244.36480708815316]
  ERNIE 5.0 is a unified autoregressive foundation model for unified multimodal understanding and generation across text, image, video, and audio.<n>To address practical challenges in large-scale deployment under diverse resource constraints, ERNIE 5.0 adopts a novel elastic training paradigm.<n>We show that ERNIE 5.0 achieves strong and balanced performance across multiple modalities.
  arXiv  Detail & Related papers  (2026-02-04T16:18:15Z)
• Large Language Model enabled Mathematical Modeling [2.132096006921049]
  This research investigates the potential of Large Language Models (LLMs) to bridge the formulation gap using natural language understanding and code generation.<n>DeepSeek-R1 is a cost-efficient and high-performing model trained with reinforcement learning.<n>Our methodology includes baseline assessments, the development of a hallucination taxonomy, and the application of mitigation strategies.
  arXiv  Detail & Related papers  (2025-10-22T17:41:42Z)
• Continual Learning for Generative AI: From LLMs to MLLMs and Beyond [56.29231194002407]
  We present a comprehensive survey of continual learning methods for mainstream generative AI models.<n>We categorize these approaches into three paradigms: architecture-based, regularization-based, and replay-based.<n>We analyze continual learning setups for different generative models, including training objectives, benchmarks, and core backbones.
  arXiv  Detail & Related papers  (2025-06-16T02:27:25Z)
• Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models [5.211806751260724]
  We propose a hierarchical sparse dictionary learning (HSDL) method that uncovers the collaboration patterns among experts.<n>We also introduce the Contribution-Aware Expert Pruning (CAEP) algorithm, which effectively prunes low-contribution experts.
  arXiv  Detail & Related papers  (2025-04-16T04:06:15Z)
• Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them [9.952432291248954]
  We investigate the use of LLM-generated data for continual pretraining of encoder models in domains with limited data.<n>We compile a benchmark specifically designed for assessing embedding model performance in invasion biology.<n>Our results demonstrate that this approach achieves a fully automated pipeline for enhancing domain-specific understanding of small encoder models.
  arXiv  Detail & Related papers  (2025-03-27T21:51:24Z)
• A Survey of Model Architectures in Information Retrieval [59.61734783818073]
  The period from 2019 to the present has represented one of the biggest paradigm shifts in information retrieval (IR) and natural language processing (NLP)<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude with a forward-looking discussion of emerging challenges and future directions.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• AIDE: Agentically Improve Visual Language Model with Domain Experts [39.34183197101934]
  AIDE (Agentic Improvement through Domain Experts) is a novel framework that enables Visual Language Models to autonomously enhance their capabilities.<n>AIDE operates through a four-stage process: (1) identifying instances for refinement, (2) engaging domain experts for targeted analysis, (3) synthesizing expert outputs with existing data, and (4) integrating enhanced instances into the training pipeline.
  arXiv  Detail & Related papers  (2025-02-13T08:05:44Z)
• MERLOT: A Distilled LLM-based Mixture-of-Experts Framework for Scalable Encrypted Traffic Classification [19.476061046309052]
  We present a scalable mixture-of-expert (MoE) based refinement of distilled large language model optimized for encrypted traffic classification. Experiments on 10 datasets show superior or competitive performance over state-of-the-art models.
  arXiv  Detail & Related papers  (2024-11-20T03:01:41Z)
• Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts [75.85448576746373]
  We propose a method of grouping and pruning similar experts to improve the model's parameter efficiency.<n>We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures.<n>The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks.
  arXiv  Detail & Related papers  (2024-07-12T17:25:02Z)
• When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
  PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
  arXiv  Detail & Related papers  (2023-12-16T17:13:08Z)
• Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study [76.52997424694767]
  We present an in-depth empirical study of keyphrase extraction and keyphrase generation using pre-trained language models. We show that PLMs have competitive high-resource performance and state-of-the-art low-resource performance. Further results show that in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models.
  arXiv  Detail & Related papers  (2022-12-20T13:20:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.