Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines

• URL: http://arxiv.org/abs/2505.07857v1
• Date: Thu, 08 May 2025 08:38:40 GMT
• Title: Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines
• Authors: Faiza Hassan, Summra Saleem, Kashif Javed, Muhammad Nabeel Asim, Abdur Rehman, Andreas Dengel,
• Abstract summary: This paper introduces a unique contrastive learning approach that leverages unlabeled Urdu data to re-train pre-trained language models.<n>It reaps the combined potential of pre-trained LLMs and the prototype-informed attention mechanism to create an end-to-end intent detection pipeline.<n>Under the paradigm of proposed predictive pipeline, it explores the potential of 6 distinct language models and 13 distinct similarity computation methods.
• Score: 5.191443390565865
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multifarious intent detection predictors are developed for different languages, including English, Chinese and French, however, the field remains underdeveloped for Urdu, the 10th most spoken language. In the realm of well-known languages, intent detection predictors utilize the strategy of few-shot learning and prediction of unseen classes based on the model training on seen classes. However, Urdu language lacks few-shot strategy based intent detection predictors and traditional predictors are focused on prediction of the same classes which models have seen in the train set. To empower Urdu language specific intent detection, this introduces a unique contrastive learning approach that leverages unlabeled Urdu data to re-train pre-trained language models. This re-training empowers LLMs representation learning for the downstream intent detection task. Finally, it reaps the combined potential of pre-trained LLMs and the prototype-informed attention mechanism to create a comprehensive end-to-end LLMPIA intent detection pipeline. Under the paradigm of proposed predictive pipeline, it explores the potential of 6 distinct language models and 13 distinct similarity computation methods. The proposed framework is evaluated on 2 public benchmark datasets, namely ATIS encompassing 5836 samples and Web Queries having 8519 samples. Across ATIS dataset under 4-way 1 shot and 4-way 5 shot experimental settings LLMPIA achieved 83.28% and 98.25% F1-Score and on Web Queries dataset produced 76.23% and 84.42% F1-Score, respectively. In an additional case study on the Web Queries dataset under same classes train and test set settings, LLMPIA outperformed state-of-the-art predictor by 53.55% F1-Score.

Related papers

• Trajectory Prediction Meets Large Language Models: A Survey [55.70506060739684]
  Recent advances in large language models (LLMs) have sparked growing interest in integrating language-driven techniques into trajectory prediction.<n>This survey provides a comprehensive overview of this emerging field, categorizing recent work into five directions.
  arXiv  Detail & Related papers  (2025-06-03T21:36:56Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection.<n>We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks [0.9786690381850356]
  This study presents in-depth examination of 7 prominent Large Language Models (LLMs) across 17 tasks using 22 datasets, 13.8 hours of speech, in a zero-shot setting, and their performance against state-of-the-art (SOTA) models.<n>Our results emphasize that models with fewer parameters but richer language-specific data, like Llama 3.1-8B, often outperform larger models with lower language diversity, such as GPT-3.5, in several tasks.
  arXiv  Detail & Related papers  (2024-05-24T11:30:37Z)
• Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks [12.7259425362286]
  We investigate how multilingual models might leverage key-value memories. For autoregressive models trained on two or more languages, do all neurons (across layers) respond equally to all languages? Our findings reveal that the layers closest to the network's input or output tend to exhibit more language-specific behaviour compared to the layers in the middle.
  arXiv  Detail & Related papers  (2023-10-24T06:45:00Z)
• Tokenizer Choice For LLM Training: Negligible or Crucial? [30.33170936148845]
  We study the influence of tokenizer choice on Large Language Models (LLMs) downstream performance by training 24 mono- and multilingual LLMs. We find that the tokenizer choice can significantly impact the model's downstream performance and training costs. We show that multilingual tokenizers trained on the five most frequent European languages require vocabulary size increases of factor three in comparison to English.
  arXiv  Detail & Related papers  (2023-10-12T22:44:19Z)
• Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding [34.81777967854573]
  We propose a unified method that integrates multilingual pretrained speech and text models and performs E2E-SLU on six datasets in four languages. We investigate how the proposed method can be improved by pretraining on widely available speech recognition data using several training objectives.
  arXiv  Detail & Related papers  (2023-10-09T19:22:51Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension [51.953428342923885]
  We develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning mechanism is developed to learn the fine difference between the accurate answer and other candidates.
  arXiv  Detail & Related papers  (2021-12-09T07:31:15Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages [75.08199398141744]
  We present AmericasNLI, an extension of XNLI (Conneau et al.), to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%.
  arXiv  Detail & Related papers  (2021-04-18T05:32:28Z)
• An Attention Ensemble Approach for Efficient Text Classification of Indian Languages [0.0]
  This paper focuses on the coarse-grained technical domain identification of short text documents in Marathi, a Devanagari script-based Indian language. A hybrid CNN-BiLSTM attention ensemble model is proposed that competently combines the intermediate sentence representations generated by the convolutional neural network and the bidirectional long short-term memory, leading to efficient text classification. Experimental results show that the proposed model outperforms various baseline machine learning and deep learning models in the given task, giving the best validation accuracy of 89.57% and f1-score of 0.8875.
  arXiv  Detail & Related papers  (2021-02-20T07:31:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.