Far Out: Evaluating Language Models on Slang in Australian and Indian English

• URL: http://arxiv.org/abs/2602.15373v2
• Date: Wed, 18 Feb 2026 02:11:26 GMT
• Title: Far Out: Evaluating Language Models on Slang in Australian and Indian English
• Abstract summary: Language models exhibit systematic performance gaps when processing text in non-standard language varieties.<n>We present a comprehensive evaluation of slang awareness in Indian English (en-IN) and Australian English (en-AU) across seven state-of-the-art language models.
• Score: 7.858383859663038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models exhibit systematic performance gaps when processing text in non-standard language varieties, yet their ability to comprehend variety-specific slang remains underexplored for several languages. We present a comprehensive evaluation of slang awareness in Indian English (en-IN) and Australian English (en-AU) across seven state-of-the-art language models. We construct two complementary datasets: WEB, containing 377 web-sourced usage examples from Urban Dictionary, and GEN, featuring 1,492 synthetically generated usages of these slang terms, across diverse scenarios. We assess language models on three tasks: target word prediction (TWP), guided target word prediction (TWP$^*$) and target word selection (TWS). Our results reveal four key findings: (1) Higher average model performance TWS versus TWP and TWP$^*$, with average accuracy score increasing from 0.03 to 0.49 respectively (2) Stronger average model performance on WEB versus GEN datasets, with average similarity score increasing by 0.03 and 0.05 across TWP and TWP$^*$ tasks respectively (3) en-IN tasks outperform en-AU when averaged across all models and datasets, with TWS demonstrating the largest disparity, increasing average accuracy from 0.44 to 0.54. These findings underscore fundamental asymmetries between generative and discriminative competencies for variety-specific language, particularly in the context of slang expressions despite being in a technologically rich language such as English.

Related papers

• FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognition [12.125413756152833]
  We introduce FiNERweb, a dataset-creation pipeline that scales the teacher-student paradigm to 91 languages and 25 scripts.<n>Building on FineWeb-Edu, our approach trains regression models to identify NER-relevant passages and annotate them with multilingual LLMs.<n>Our experiments show that the regression model achieves more than 84 F1, and that models trained on FiNERweb obtain comparable or improved performance in zero shot transfer settings.
  arXiv  Detail & Related papers  (2025-12-15T20:36:39Z)
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought [23.847410628315544]
  We introduce **Language-Mixed CoT**, a reasoning schema that switches between English and a target language.<n>We train ninve models (4B-35B) across six families (Qwen2.5, Llama-3.1, Gemma-3, etc)<n>Our best model, **KO-REAson-35B**, achieves state-of-the-art performance, with the highest overall average score (64.0 pm 25)
  arXiv  Detail & Related papers  (2025-10-05T14:39:41Z)
• A Comparative Evaluation of Large Language Models for Persian Sentiment Analysis and Emotion Detection in Social Media Texts [2.820011731460364]
  This study presents a comparative evaluation of four large language models (LLMs) for sentiment analysis and emotion detection in Persian social media texts.<n>The results show that all models reach an acceptable level of performance, and a statistical comparison of the best three models indicates no significant differences among them.<n>The findings indicate that the emotion detection task is more challenging for all models compared to the sentiment analysis task, and the misclassification patterns can represent some challenges in Persian language texts.
  arXiv  Detail & Related papers  (2025-09-18T12:59:07Z)
• Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settings [1.1556013985948772]
  We evaluate transferability of pre-trained language models to low-resource Indonesian local languages.<n>We group the target languages into three categories: seen, partially seen, and unseen.<n> Multilingual models perform best on seen languages, moderately on partially seen ones, and poorly on unseen languages.<n>We find that MAD-X significantly improves performance, especially for seen and partially seen languages, without requiring labeled data in the target language.
  arXiv  Detail & Related papers  (2025-07-02T12:17:55Z)
• BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English [8.823927892310238]
  BESSTIE is a benchmark for sentiment and sarcasm classification for three varieties of English: Australian (en-AU), Indian (en-IN), and British (en-UK)<n>We collect datasets for these language varieties using two methods: location-based for Google Places reviews, and topic-based filtering for Reddit comments.<n>Native speakers of the language varieties manually annotate the datasets with sentiment and sarcasm labels.
  arXiv  Detail & Related papers  (2024-12-06T02:34:40Z)
• A Text-to-Text Model for Multilingual Offensive Language Identification [19.23565690468299]
  This study presents the first pre-trained model with encoder-decoder architecture for offensive language identification with text-to-text transformers (T5) Our pre-trained T5 model outperforms other transformer-based models fine-tuned for offensive language detection, such as fBERT and HateBERT, in multiple English benchmarks. Following a similar approach, we also train the first multilingual pre-trained model for offensive language identification using mT5.
  arXiv  Detail & Related papers  (2023-12-06T09:37:27Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Retrieval-based Disentangled Representation Learning with Natural Language Supervision [61.75109410513864]
  We present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employ a bi-encoder model to represent both data and natural language in a vocabulary space, enabling the model to distinguish intrinsic dimensions that capture characteristics within data through its natural language counterpart, thus disentanglement.
  arXiv  Detail & Related papers  (2022-12-15T10:20:42Z)
• ABINet++: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Spotting [121.11880210592497]
  We argue that the limited capacity of language models comes from 1) implicit language modeling; 2) unidirectional feature representation; and 3) language model with noise input. We propose an autonomous, bidirectional and iterative ABINet++ for scene text spotting.
  arXiv  Detail & Related papers  (2022-11-19T03:50:33Z)
• Elaboration-Generating Commonsense Question Answering at Scale [77.96137534751445]
  In question answering requiring common sense, language models (e.g., GPT-3) have been used to generate text expressing background knowledge. We finetune smaller language models to generate useful intermediate context, referred to here as elaborations. Our framework alternates between updating two language models -- an elaboration generator and an answer predictor -- allowing each to influence the other.
  arXiv  Detail & Related papers  (2022-09-02T18:32:09Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)
• NEMO: Frequentist Inference Approach to Constrained Linguistic Typology Feature Prediction in SIGTYP 2020 Shared Task [83.43738174234053]
  We employ frequentist inference to represent correlations between typological features and use this representation to train simple multi-class estimators that predict individual features. Our best configuration achieved the micro-averaged accuracy score of 0.66 on 149 test languages.
  arXiv  Detail & Related papers  (2020-10-12T19:25:43Z)
• Language Models are Few-Shot Learners [61.36677350504291]
  We show that scaling up language models greatly improves task-agnostic, few-shot performance. We train GPT-3, an autoregressive language model with 175 billion parameters, and test its performance in the few-shot setting. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
  arXiv  Detail & Related papers  (2020-05-28T17:29:03Z)
• Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages [112.65994041398481]
  We propose a Bayesian generative model for the space of neural parameters. We infer the posteriors over such latent variables based on data from seen task-language combinations. Our model yields comparable or better results than state-of-the-art, zero-shot cross-lingual transfer methods.
  arXiv  Detail & Related papers  (2020-01-30T16:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.