SEKE: Specialised Experts for Keyword Extraction

• URL: http://arxiv.org/abs/2412.14087v1
• Date: Wed, 18 Dec 2024 17:34:32 GMT
• Title: SEKE: Specialised Experts for Keyword Extraction
• Abstract summary: Keywords extraction involves identifying the most descriptive words in a document.<n>We propose a novel supervised keyword extraction approach based on the mixture of experts (MoE) technique.<n>MoE uses a learnable routing sub-network to direct information to specialised experts, allowing them to specialize in distinct regions of the input space.
• Score: 5.8908163351315075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Keyword extraction involves identifying the most descriptive words in a document, allowing automatic categorisation and summarisation of large quantities of diverse textual data. Relying on the insight that real-world keyword detection often requires handling of diverse content, we propose a novel supervised keyword extraction approach based on the mixture of experts (MoE) technique. MoE uses a learnable routing sub-network to direct information to specialised experts, allowing them to specialize in distinct regions of the input space. SEKE, a mixture of Specialised Experts for supervised Keyword Extraction, uses DeBERTa as the backbone model and builds on the MoE framework, where experts attend to each token, by integrating it with a recurrent neural network (RNN), to allow successful extraction even on smaller corpora, where specialisation is harder due to lack of training data. The MoE framework also provides an insight into inner workings of individual experts, enhancing the explainability of the approach. We benchmark SEKE on multiple English datasets, achieving state-of-the-art performance compared to strong supervised and unsupervised baselines. Our analysis reveals that depending on data size and type, experts specialize in distinct syntactic and semantic components, such as punctuation, stopwords, parts-of-speech, or named entities. Code is available at: https://github.com/matejMartinc/SEKE_keyword_extraction

Related papers

• ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts [4.5311655360445515]
  Handwritten text recognition resources are often small and distributed across collections that differ in language, script, document structure, and annotation format.<n>We propose ExpertHTR, a unified vision-language framework that addresses this problem through complementary supervision and conditional model capacity.<n>Building on a jointly trained dense model, ExpertHTR introduces a sparse Mixture-of-Experts architecture with an always-active shared branch and conditionally routed full-MLP experts.
  arXiv  Detail & Related papers  (2026-09-11T10:59:32Z)
• ExpertLens: Visualizing Embedding Spaces for Post-Hoc Explainability in MoE Enhanced Retrievers [45.67070112128053]
  ExpertLens is a post-hoc explainability framework for Mixture-of-Experts (MoE)-enhanced dense retrievers.<n>We show that ExpertLens consistently improves embedding space structure and retrieval effectiveness.
  arXiv  Detail & Related papers  (2026-09-05T15:55:32Z)
• Named Entity Recognition of Historical Text via Large Language Model [1.5469452301122175]
  Named Entity Recognition (NER) plays a crucial role in extracting information from unstructured text.<n>Traditionally, NER is addressed using supervised machine learning approaches, which require large amounts of annotated training data.<n>In this study, we explore the feasibility of applying LLMs to NER in historical documents using zero-shot and few-shot prompting strategies.<n>Our experiments, conducted on the HIPE-2022 dataset, show that LLMs can achieve reasonably strong performance on NER tasks in this setting.
  arXiv  Detail & Related papers  (2025-08-25T14:52:11Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)
• Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy [52.426623750562335]
  We introduce ToTER (Topical taxonomy Enhanced Retrieval) framework. ToTER identifies the central topics of queries and documents with the guidance of the taxonomy, and exploits their topical relatedness to supplement missing contexts. As a plug-and-play framework, ToTER can be flexibly employed to enhance various PLM-based retrievers.
  arXiv  Detail & Related papers  (2024-03-07T02:34:54Z)
• OpenSD: Unified Open-Vocabulary Segmentation and Detection [24.08879095731279]
  We present a universal transformer-based framework, abbreviated as OpenSD, to handle open-vocabulary segmentation and detection tasks. To better leverage CLIP for end-to-end segmentation and detection, we propose dual classifiers to handle the in-vocabulary domain and out-of-vocabulary domain. The results demonstrate that OpenSD outperforms state-of-the-art open-vocabulary segmentation and detection methods in both closed- and open-vocabulary settings.
  arXiv  Detail & Related papers  (2023-12-10T08:51:34Z)
• Open-Vocabulary Camouflaged Object Segmentation [66.94945066779988]
  We introduce a new task, open-vocabulary camouflaged object segmentation (OVCOS) We construct a large-scale complex scene dataset (textbfOVCamo) containing 11,483 hand-selected images with fine annotations and corresponding object classes. By integrating the guidance of class semantic knowledge and the supplement of visual structure cues from the edge and depth information, the proposed method can efficiently capture camouflaged objects.
  arXiv  Detail & Related papers  (2023-11-19T06:00:39Z)
• Instruct and Extract: Instruction Tuning for On-Demand Information Extraction [86.29491354355356]
  On-Demand Information Extraction aims to fulfill the personalized demands of real-world users. We present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE.
  arXiv  Detail & Related papers  (2023-10-24T17:54:25Z)
• Expert-sourcing Domain-specific Knowledge: The Case of Synonym Validation [14.51095331294056]
  We illustrate tool support that we adopted and extended to source domain-specific knowledge from experts. We provide insight in design decisions that aim at motivating experts to dedicate their time at performing the labelling task. We foresee that the approach of expert-sourcing is applicable to any data labelling task in software engineering.
  arXiv  Detail & Related papers  (2023-09-28T19:02:33Z)
• DiscoverPath: A Knowledge Refinement and Retrieval System for Interdisciplinarity on Biomedical Research [96.10765714077208]
  Traditional keyword-based search engines fall short in assisting users who may not be familiar with specific terminologies. We present a knowledge graph-based paper search engine for biomedical research to enhance the user experience. The system, dubbed DiscoverPath, employs Named Entity Recognition (NER) and part-of-speech (POS) tagging to extract terminologies and relationships from article abstracts to create a KG.
  arXiv  Detail & Related papers  (2023-09-04T20:52:33Z)
• A Simple Framework for Open-Vocabulary Segmentation and Detection [85.21641508535679]
  We present OpenSeeD, a simple Open-vocabulary and Detection framework that jointly learns from different segmentation and detection datasets. We first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. After pre-training, our model exhibits competitive or stronger zero-shot transferability for both segmentation and detection.
  arXiv  Detail & Related papers  (2023-03-14T17:58:34Z)
• Enriching Relation Extraction with OpenIE [70.52564277675056]
  Relation extraction (RE) is a sub-discipline of information extraction (IE) In this work, we explore how recent approaches for open information extraction (OpenIE) may help to improve the task of RE. Our experiments over two annotated corpora, KnowledgeNet and FewRel, demonstrate the improved accuracy of our enriched models.
  arXiv  Detail & Related papers  (2022-12-19T11:26:23Z)
• Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability [3.021134753248103]
  Sparsely-gated Mixture of Expert (MoE) layers have been successfully applied for scaling large transformers. In this work, we apply sparse MoE layers to CNNs for computer vision tasks and analyze the resulting effect on model interpretability.
  arXiv  Detail & Related papers  (2022-04-22T09:40:23Z)
• Providing Insights for Open-Response Surveys via End-to-End Context-Aware Clustering [2.6094411360258185]
  In this work, we present a novel end-to-end context-aware framework that extracts, aggregates, and abbreviates embedded semantic patterns in open-response survey data. Our framework relies on a pre-trained natural language model in order to encode the textual data into semantic vectors. Our framework reduces the costs at-scale by automating the process of extracting the most insightful information pieces from survey data.
  arXiv  Detail & Related papers  (2022-03-02T18:24:10Z)
• Multi-Round Parsing-based Multiword Rules for Scientific OpenIE [18.163915930906693]
  OpenIE identifies a relational phrase to describe the relationship between a subject and an object. In this work, we present a set of rules for extracting structured information based on dependency parsing. Results on novel datasets show the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2021-08-04T14:17:48Z)
• TNT-KID: Transformer-based Neural Tagger for Keyword Identification [7.91883337742071]
  We present a novel algorithm for keyword identification called Transformer-based Neural Tagger for Keyword IDentification (TNT-KID) By adapting the transformer architecture for a specific task at hand and leveraging language model pretraining on a domain specific corpus, the model is capable of overcoming deficiencies of both supervised and unsupervised state-of-the-art approaches to keyword extraction.
  arXiv  Detail & Related papers  (2020-03-20T09:55:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.