Efficient Text Encoders for Labor Market Analysis

• URL: http://arxiv.org/abs/2505.24640v1
• Date: Fri, 30 May 2025 14:27:25 GMT
• Title: Efficient Text Encoders for Labor Market Analysis
• Authors: Jens-Joris Decorte, Jeroen Van Hautte, Chris Develder, Thomas Demeester,
• Abstract summary: We present textbfConT-match, a novel contrastive learning approach with token-level attention for the extreme multi-label classification task of skill classification.<n>We also present textbfSkill-XL, a new benchmark with exhaustive, sentence-level skill annotations.<n>Our models are efficient, accurate, and scalable, making them ideal for large-scale, real-time labor market analysis.
• Score: 11.083396379885478
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Labor market analysis relies on extracting insights from job advertisements, which provide valuable yet unstructured information on job titles and corresponding skill requirements. While state-of-the-art methods for skill extraction achieve strong performance, they depend on large language models (LLMs), which are computationally expensive and slow. In this paper, we propose \textbf{ConTeXT-match}, a novel contrastive learning approach with token-level attention that is well-suited for the extreme multi-label classification task of skill classification. \textbf{ConTeXT-match} significantly improves skill extraction efficiency and performance, achieving state-of-the-art results with a lightweight bi-encoder model. To support robust evaluation, we introduce \textbf{Skill-XL}, a new benchmark with exhaustive, sentence-level skill annotations that explicitly address the redundancy in the large label space. Finally, we present \textbf{JobBERT V2}, an improved job title normalization model that leverages extracted skills to produce high-quality job title representations. Experiments demonstrate that our models are efficient, accurate, and scalable, making them ideal for large-scale, real-time labor market analysis.

Related papers

• RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs [76.3459242819381]
  RefineX is a novel framework for large-scale, surgical refinement of pre-training data through programmatic editing tasks.<n>The core strength of RefineX lies in distilling high-quality, expert-guided end-to-end refinement results into minimal edit-based deletion programs.<n>We evaluate RefineX across from-scratch pre-training at multiple model scales and find that it consistently outperforms models trained on raw, filtered, or alternatively refined data.
  arXiv  Detail & Related papers  (2025-07-04T02:19:58Z)
• Learning Task Representations from In-Context Learning [73.72066284711462]
  Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning.<n>We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads.<n>We show that our method's effectiveness stems from aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model.
  arXiv  Detail & Related papers  (2025-02-08T00:16:44Z)
• Enhancing Talent Employment Insights Through Feature Extraction with LLM Finetuning [0.0]
  We develop a robust pipeline to identify variables such as remote work availability, remuneration structures, educational requirements, and work experience preferences.<n>Our methodology combines semantic chunking, retrieval-augmented generation (RAG), and fine-tuning DistilBERT models to overcome the limitations of traditional parsing tools.<n>We present a comprehensive evaluation of our fine-tuned models and analyze their strengths, limitations, and potential for scaling.
  arXiv  Detail & Related papers  (2025-01-13T19:49:49Z)
• An Energy-based Model for Word-level AutoCompletion in Computer-aided Translation [97.3797716862478]
  Word-level AutoCompletion (WLAC) is a rewarding yet challenging task in Computer-aided Translation. Existing work addresses this task through a classification model based on a neural network that maps the hidden vector of the input context into its corresponding label. This work proposes an energy-based model for WLAC, which enables the context hidden vector to capture crucial information from the source sentence.
  arXiv  Detail & Related papers  (2024-07-29T15:07:19Z)
• Computational Job Market Analysis with Natural Language Processing [5.117211717291377]
  This thesis investigates Natural Language Processing (NLP) technology for extracting relevant information from job descriptions. We frame the problem, obtaining annotated data, and introducing extraction methodologies. Our contributions include job description datasets, a de-identification dataset, and a novel active learning algorithm for efficient model training.
  arXiv  Detail & Related papers  (2024-04-29T14:52:38Z)
• Rethinking Skill Extraction in the Job Market Domain using Large Language Models [20.256353240384133]
  Skill Extraction involves identifying skills and qualifications mentioned in documents such as job postings and resumes. The reliance on manually annotated data limits the generalizability of such approaches. In this paper, we explore the use of in-context learning to overcome these challenges.
  arXiv  Detail & Related papers  (2024-02-06T09:23:26Z)
• NNOSE: Nearest Neighbor Occupational Skill Extraction [55.22292957778972]
  We tackle the complexity in occupational skill datasets. We employ an external datastore for retrieving similar skills in a dataset-unifying manner. We observe a performance gain in predicting infrequent patterns, with substantial gains of up to 30% span-F1 in cross-dataset settings.
  arXiv  Detail & Related papers  (2024-01-30T15:18:29Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• Extreme Multi-Label Skill Extraction Training using Large Language Models [19.095612333241288]
  We describe a cost-effective approach to generate an accurate, fully synthetic labeled dataset for skill extraction. Our results show a consistent increase of between 15 to 25 percentage points in textitR-Precision@5 compared to previously published results.
  arXiv  Detail & Related papers  (2023-07-20T11:29:15Z)
• Composite Learning for Robust and Effective Dense Predictions [81.2055761433725]
  Multi-task learning promises better model generalization on a target task by jointly optimizing it with an auxiliary task. We find that jointly training a dense prediction (target) task with a self-supervised (auxiliary) task can consistently improve the performance of the target task, while eliminating the need for labeling auxiliary tasks.
  arXiv  Detail & Related papers  (2022-10-13T17:59:16Z)
• Design of Negative Sampling Strategies for Distantly Supervised Skill Extraction [19.43668931500507]
  We propose an end-to-end system for skill extraction, based on distant supervision through literal matching. We observe that using the ESCO taxonomy to select negative examples from related skills yields the biggest improvements. We release the benchmark dataset for research purposes to stimulate further research on the task.
  arXiv  Detail & Related papers  (2022-09-13T13:37:06Z)
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning [120.38381203153159]
  Reinforcement learning can train policies that effectively perform complex tasks. For long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and composing lower-level skills. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill.
  arXiv  Detail & Related papers  (2021-11-04T22:46:16Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.