Contrastive Bi-Encoder Models for Multi-Label Skill Extraction: Enhancing ESCO Ontology Matching with BERT and Attention Mechanisms

• URL: http://arxiv.org/abs/2601.09119v1
• Date: Wed, 14 Jan 2026 03:43:45 GMT
• Title: Contrastive Bi-Encoder Models for Multi-Label Skill Extraction: Enhancing ESCO Ontology Matching with BERT and Attention Mechanisms
• Authors: Yongming Sun,
• Abstract summary: Fine-dense labor market analysis increasingly relies on mapping unstructured job advertisements to standardized skill such as ESCO.<n>We propose a zero-shot skill extraction framework that eliminates the need for manually labeled job-ad training data.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-grained labor market analysis increasingly relies on mapping unstructured job advertisements to standardized skill taxonomies such as ESCO. This mapping is naturally formulated as an Extreme Multi-Label Classification (XMLC) problem, but supervised solutions are constrained by the scarcity and cost of large-scale, taxonomy-aligned annotations--especially in non-English settings where job-ad language diverges substantially from formal skill definitions. We propose a zero-shot skill extraction framework that eliminates the need for manually labeled job-ad training data. The framework uses a Large Language Model (LLM) to synthesize training instances from ESCO definitions, and introduces hierarchically constrained multi-skill generation based on ESCO Level-2 categories to improve semantic coherence in multi-label contexts. On top of the synthetic corpus, we train a contrastive bi-encoder that aligns job-ad sentences with ESCO skill descriptions in a shared embedding space; the encoder augments a BERT backbone with BiLSTM and attention pooling to better model long, information-dense requirement statements. An upstream RoBERTa-based binary filter removes non-skill sentences to improve end-to-end precision. Experiments show that (i) hierarchy-conditioned generation improves both fluency and discriminability relative to unconstrained pairing, and (ii) the resulting multi-label model transfers effectively to real-world Chinese job advertisements, achieving strong zero-shot retrieval performance (F1@5 = 0.72) and outperforming TF--IDF and standard BERT baselines. Overall, the proposed pipeline provides a scalable, data-efficient pathway for automated skill coding in labor economics and workforce analytics.

Related papers

• Compass-Embedding v4: Robust Contrastive Learning for Multilingual E-commerce Embeddings [12.049937870582113]
  We present a high-efficiency multilingual embedding framework specifically optimized for Southeast Asian (SEA) e-commerce scenarios.<n> Compass-Embedding v4 addresses three core challenges.<n>We construct a diversified training corpus through context-grounded synthetic data generation, cross-lingual translation, and structured e-commerce data construction.
  arXiv  Detail & Related papers  (2025-12-25T13:41:53Z)
• DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning [54.35107462768146]
  Cold-start active learning (CSAL) selects valuable instances from an unlabeled dataset for manual annotation.<n>Existing CSAL methods overlook weak classes and hard representative examples, resulting in biased learning.<n>This paper proposes a novel dual-diversity enhancing and uncertainty-aware framework for CSAL.
  arXiv  Detail & Related papers  (2025-02-01T04:00:03Z)
• Learnable Item Tokenization for Generative Recommendation [113.80559032128065]
  We propose LETTER (a LEarnable Tokenizer for generaTivE Recommendation), which integrates hierarchical semantics, collaborative signals, and code assignment diversity.<n> LETTER incorporates Residual Quantized VAE for semantic regularization, a contrastive alignment loss for collaborative regularization, and a diversity loss to mitigate code assignment bias.
  arXiv  Detail & Related papers  (2024-05-12T15:49:38Z)
• TnT-LLM: Text Mining at Scale with Large Language Models [24.731544646232962]
  Large Language Models (LLMs) automate the process of end-to-end label generation and assignment with minimal human effort. We show that TnT-LLM generates more accurate and relevant label when compared against state-of-the-art baselines. We also share our practical experiences and insights on the challenges and opportunities of using LLMs for large-scale text mining in real-world applications.
  arXiv  Detail & Related papers  (2024-03-18T18:45:28Z)
• Hierarchical Classification of Transversal Skills in Job Ads Based on Sentence Embeddings [0.0]
  This paper aims to identify correlations between job ad requirements and skill sets using a deep learning model. The approach involves data collection, preprocessing, and labeling using ESCO (European Skills, Competences, and Occupations) taxonomy.
  arXiv  Detail & Related papers  (2024-01-10T11:07:32Z)
• Prompt Tuned Embedding Classification for Multi-Label Industry Sector Allocation [2.024620791810963]
  This study benchmarks the performance of Prompt Tuning and baselines for multi-label text classification. It is applied to classifying companies into an investment firm's proprietary industry taxonomy. We confirm that the model's performance is consistent across both well-known and less-known companies.
  arXiv  Detail & Related papers  (2023-09-21T13:45:32Z)
• Transfer-Free Data-Efficient Multilingual Slot Labeling [82.02076369811402]
  Slot labeling is a core component of task-oriented dialogue (ToD) systems. To mitigate the inherent data scarcity issue, current research on multilingual ToD assumes that sufficient English-language annotated data are always available. We propose a two-stage slot labeling approach (termed TWOSL) which transforms standard multilingual sentence encoders into effective slot labelers.
  arXiv  Detail & Related papers  (2023-05-22T22:47:32Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders [111.65322283420805]
  We present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bi-directional EncodeR) AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model.
  arXiv  Detail & Related papers  (2020-10-15T18:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.