Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning

• URL: http://arxiv.org/abs/2606.00334v1
• Date: Fri, 29 May 2026 20:19:49 GMT
• Title: Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning
• Abstract summary: We introduce the Triangulated Preference Shift score, a metric that triangulates between human gold standards, base models, and variants.<n>We provide data across six model families, anchor the results in the literature, and illustrate the general approach's utility.
• Score: 0.33463386076921503
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Various language domains have undergone remarkable changes in recent years; these shifts are largely attributed to the advent of Large Language Models and their misalignment with natural language usage. These misalignments are thought to partly originate in the preference-learning stage, e.g. Reinforcement Learning from Human Feedback, which generally makes models more useful but simultaneously may introduce systematic lexical bias. In terms of lexical behavior, this is visible in a model's preference for certain formats or the overuse of words (delve, furthermore), even when such patterns are not present in base model outputs. Research on lexical misalignment induced during preference training is constrained by reliance on manual curation. We address this, by introducing the Triangulated Preference Shift score, a metric that triangulates between human gold standards, base models, and instruct variants to isolate shifts induced specifically by preference learning, without manual curation. We provide data across six model families, anchor the results in the literature, and illustrate the general approach's utility by analyzing whether preference learning shifts models toward what could be interpreted as a "language of prestige". The metric provides an initial automated method to quantify behavioral shifts attributable to preference tuning, and thus, may help inform model alignment and development of trustworthy AI.

Related papers

• Language Models Generalize to Human-like Word Order Preferences [1.9371675344367494]
  We investigate whether language models exhibit the same bias under similar conditions.<n>We find that they consistently prefer scope-homomorphic orders despite never observing them during training.<n>While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences.
  arXiv  Detail & Related papers  (2026-08-05T16:33:46Z)
• Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models [0.0]
  The language used by digital chat assistants such as ChatGPT can diverge from human expectations (misalignment)<n>This paper introduces two curation-free, assumption-light evaluation metrics: the Lexical Alignment Score and the Triangulated Preference Shift.<n>The procedure identifies, without manual intervention, overused items such as'suggest', 'additionally', and'strategy', and estimates their link to preference learning.
  arXiv  Detail & Related papers  (2026-06-02T05:23:45Z)
• Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward [9.099589602551573]
  Language models are vulnerable to short adversarial suffixes that can reliably alter predictions.<n>In this paper, a reinforcement learning framework is used where the suffix is treated as a policy and trained with Proximal Policy Optimization.<n>Results show that RL-trained suffixes consistently degrade accuracy and transfer more effectively across tasks and models than previous adversarial triggers of similar genres.
  arXiv  Detail & Related papers  (2025-12-09T00:18:06Z)
• Mitigating Biases in Language Models via Bias Unlearning [27.565946855618368]
  We propose BiasUnlearn, a novel model debiasing framework which achieves targeted debiasing via dual-pathway unlearning mechanisms.<n>The results show that BiasUnlearn outperforms existing methods in mitigating bias in language models while retaining language modeling capabilities.
  arXiv  Detail & Related papers  (2025-09-30T02:15:12Z)
• Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding [53.63482987410292]
  We present a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models.<n>We evaluate our method on four natural language understanding (NLU) datasets covering both binary and multi-class classification tasks.
  arXiv  Detail & Related papers  (2025-07-13T19:36:17Z)
• A Distributional Perspective on Word Learning in Neural Language Models [57.41607944290822]
  There are no widely agreed-upon metrics for word learning in language models.<n>We argue that distributional signatures studied in prior work fail to capture key distributional information.<n>We obtain learning trajectories for a selection of small language models we train from scratch.
  arXiv  Detail & Related papers  (2025-02-09T13:15:59Z)
• Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora [4.008122785948581]
  We introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them.<n>Our method enhances preference learning by automatically detecting and selecting consistent annotations.
  arXiv  Detail & Related papers  (2024-08-23T02:27:14Z)
• RecExplainer: Aligning Large Language Models for Explaining Recommendation Models [50.74181089742969]
  Large language models (LLMs) have demonstrated remarkable intelligence in understanding, reasoning, and instruction following. This paper presents the initial exploration of using LLMs as surrogate models to explain black-box recommender models. To facilitate an effective alignment, we introduce three methods: behavior alignment, intention alignment, and hybrid alignment.
  arXiv  Detail & Related papers  (2023-11-18T03:05:43Z)
• Training Language Models with Language Feedback at Scale [50.70091340506957]
  We introduce learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback.
  arXiv  Detail & Related papers  (2023-03-28T17:04:15Z)
• Debiasing Vision-Language Models via Biased Prompts [79.04467131711775]
  We propose a general approach for debiasing vision-language foundation models by projecting out biased directions in the text embedding. We show that debiasing only the text embedding with a calibrated projection matrix suffices to yield robust classifiers and fair generative models.
  arXiv  Detail & Related papers  (2023-01-31T20:09:33Z)
• A General Language Assistant as a Laboratory for Alignment [3.3598752405752106]
  We study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. We study a preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.
  arXiv  Detail & Related papers  (2021-12-01T22:24:34Z)
• Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
  Human-readability is an important and desirable standard for machine-learned model interpretability. We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code. We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
  arXiv  Detail & Related papers  (2021-01-21T01:46:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.