Enhancing PLM Performance on Labour Market Tasks via Instruction-based Finetuning and Prompt-tuning with Rules

• URL: http://arxiv.org/abs/2308.16770v1
• Date: Thu, 31 Aug 2023 14:47:00 GMT
• Title: Enhancing PLM Performance on Labour Market Tasks via Instruction-based Finetuning and Prompt-tuning with Rules
• Authors: Jarno Vrolijk and David Graus
• Abstract summary: We show the effectiveness of prompt-based tuning of pre-trained language models (PLM) in labour market specific applications. Our results indicate that cost-efficient methods such as PTR and instruction tuning without exemplars can significantly increase the performance of PLMs.
• Score: 0.9790236766474201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increased digitization of the labour market has given researchers, educators, and companies the means to analyze and better understand the labour market. However, labour market resources, although available in high volumes, tend to be unstructured, and as such, research towards methodologies for the identification, linking, and extraction of entities becomes more and more important. Against the backdrop of this quest for better labour market representations, resource constraints and the unavailability of large-scale annotated data cause a reliance on human domain experts. We demonstrate the effectiveness of prompt-based tuning of pre-trained language models (PLM) in labour market specific applications. Our results indicate that cost-efficient methods such as PTR and instruction tuning without exemplars can significantly increase the performance of PLMs on downstream labour market applications without introducing additional model layers, manual annotations, and data augmentation.

Related papers

• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process. We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Comparative Analysis of Encoder-Based NER and Large Language Models for Skill Extraction from Russian Job Vacancies [0.0]
  This study compares Named Entity Recognition methods based on encoders with Large Language Models (LLMs) for extracting skills from Russian job vacancies. Results indicate that traditional NER models, especially DeepPavlov RuBERT NER tuned, outperform LLMs across various metrics including accuracy, precision, recall, and inference time. This research contributes to the field of natural language processing (NLP) and its application in the labor market, particularly in non-English contexts.
  arXiv  Detail & Related papers  (2024-07-29T09:08:40Z)
• LLM-Empowered State Representation for Reinforcement Learning [64.3351150030341]
  State representations in reinforcement learning often omit critical task-related details. We propose LLM-Empowered State Representation (LESR), a novel approach that utilizes LLM to autonomously generate task-related state representation codes. LESR exhibits high sample efficiency and outperforms state-of-the-art baselines by an average of 29% in accumulated reward in Mujoco tasks and 30% in success rates in Gym-Robotics tasks.
  arXiv  Detail & Related papers  (2024-07-18T07:47:51Z)
• What Teaches Robots to Walk, Teaches Them to Trade too -- Regime Adaptive Execution using Informed Data and LLMs [1.6317061277457001]
  We introduce an innovative approach that leverages world knowledge of pretrained LLMs (aka. 'privileged information' in robotics) and dynamically adapts them using intrinsic, natural market rewards. Strong empirical results demonstrate the efficacy of our method in adapting to regime shifts in financial markets. The proposed algorithmic framework outperforms best-performing SOTA LLM models on the existing (FLARE) benchmark stock-movement (SM) tasks by more than 15% improved accuracy.
  arXiv  Detail & Related papers  (2024-06-20T00:17:28Z)
• Chain-of-Thought Prompting for Demographic Inference with Large Multimodal Models [58.58594658683919]
  Large multimodal models (LMMs) have shown transformative potential across various research tasks. Our findings indicate LMMs possess advantages in zero-shot learning, interpretability, and handling uncurated 'in-the-wild' inputs. We propose a Chain-of-Thought augmented prompting approach, which effectively mitigates the off-target prediction issue.
  arXiv  Detail & Related papers  (2024-05-24T16:26:56Z)
• Computational Job Market Analysis with Natural Language Processing [5.117211717291377]
  This thesis investigates Natural Language Processing (NLP) technology for extracting relevant information from job descriptions. We frame the problem, obtaining annotated data, and introducing extraction methodologies. Our contributions include job description datasets, a de-identification dataset, and a novel active learning algorithm for efficient model training.
  arXiv  Detail & Related papers  (2024-04-29T14:52:38Z)
• LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition [67.96794382040547]
  $LLM-DA$ is a novel data augmentation technique based on large language models (LLMs) for the few-shot NER task. Our approach involves employing 14 contextual rewriting strategies, designing entity replacements of the same type, and incorporating noise injection to enhance robustness.
  arXiv  Detail & Related papers  (2024-02-22T14:19:56Z)
• Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression [64.07696663255155]
  Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. We introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT) which effectively transfers the knowledge of LLMs to extremely small-scale models.
  arXiv  Detail & Related papers  (2023-10-24T07:58:20Z)
• Large Process Models: Business Process Management in the Age of Generative AI [4.249492423406116]
  Large Process Model (LPM) combines correlation power of Large Language Models with analytical precision and reliability of knowledge-based systems and automated reasoning approaches. LPM would allow organizations to receive context-specific (tailored) process and other business models, analytical deep-dives, and improvement recommendations.
  arXiv  Detail & Related papers  (2023-09-02T10:32:53Z)
• What do LLMs Know about Financial Markets? A Case Study on Reddit Market Sentiment Analysis [15.195505464654493]
  Market sentiment analysis on social media content requires knowledge of both financial markets and social media jargon. Our pipeline generates weak financial sentiment labels for Reddit posts with a large language model (LLM) With only a handful of prompts, the final model performs on par with existing supervised models.
  arXiv  Detail & Related papers  (2022-12-21T19:11:19Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.