Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback

• URL: http://arxiv.org/abs/2508.01930v1
• Date: Sun, 03 Aug 2025 21:45:37 GMT
• Title: Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback
• Authors: Tom S. Juzek, Zina B. Ward,
• Abstract summary: Large Language Models (LLMs) are known to overuse certain terms like "delve" and "intricate"<n>This study investigates the contribution of Learning from Human Feedback (LHF)<n>We more conclusively link LHF to lexical overuse by experimentally emulating the LHF procedure.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) are known to overuse certain terms like "delve" and "intricate." The exact reasons for these lexical choices, however, have been unclear. Using Meta's Llama model, this study investigates the contribution of Learning from Human Feedback (LHF), under which we subsume Reinforcement Learning from Human Feedback and Direct Preference Optimization. We present a straightforward procedure for detecting the lexical preferences of LLMs that are potentially LHF-induced. Next, we more conclusively link LHF to lexical overuse by experimentally emulating the LHF procedure and demonstrating that participants systematically prefer text variants that include certain words. This lexical overuse can be seen as a sort of misalignment, though our study highlights the potential divergence between the lexical expectations of different populations -- namely LHF workers versus LLM users. Our work contributes to the growing body of research on explainable artificial intelligence and emphasizes the importance of both data and procedural transparency in alignment research.

Related papers

• Using AI to replicate human experimental results: a motion study [0.11838866556981258]
  This paper explores the potential of large language models (LLMs) as reliable analytical tools in linguistic research.<n>It focuses on the emergence of affective meanings in temporal expressions involving manner-of-motion verbs.
  arXiv  Detail & Related papers  (2025-07-14T14:47:01Z)
• Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models [0.0]
  It is widely assumed that scientists' use of large language models (LLMs) is responsible for language change.<n>We develop a formal, transferable method to characterize these linguistic changes.<n>We find 21 focal words whose increased occurrence in scientific abstracts is likely the result of LLM usage.<n>We assess whether reinforcement learning from human feedback contributes to the overuse of focal words.
  arXiv  Detail & Related papers  (2024-12-16T02:27:59Z)
• Iter-AHMCL: Alleviate Hallucination for Large Language Model via Iterative Model-level Contrastive Learning [16.883679810267342]
  Iterative Model-level Contrastive Learning (Iter-AHMCL) to address hallucination. This paper introduces a novel approach called Iterative Model-level Contrastive Learning (Iter-AHMCL) to address hallucination.
  arXiv  Detail & Related papers  (2024-10-16T00:15:40Z)
• Uncovering Factor Level Preferences to Improve Human-Model Alignment [58.50191593880829]
  We introduce PROFILE, a framework that uncovers and quantifies the influence of specific factors driving preferences. ProFILE's factor level analysis explains the 'why' behind human-model alignment and misalignment. We demonstrate how leveraging factor level insights, including addressing misaligned factors, can improve alignment with human preferences.
  arXiv  Detail & Related papers  (2024-10-09T15:02:34Z)
• A Survey on Human Preference Learning for Large Language Models [81.41868485811625]
  The recent surge of versatile large language models (LLMs) largely depends on aligning increasingly capable foundation models with human intentions by preference learning. This survey covers the sources and formats of preference feedback, the modeling and usage of preference signals, as well as the evaluation of the aligned LLMs.
  arXiv  Detail & Related papers  (2024-06-17T03:52:51Z)
• RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs [49.386699863989335]
  Training large language models (LLMs) to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals.
  arXiv  Detail & Related papers  (2024-04-12T15:54:15Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• Probing Large Language Models from A Human Behavioral Perspective [24.109080140701188]
  Large Language Models (LLMs) have emerged as dominant foundational models in modern NLP. The understanding of their prediction processes and internal mechanisms, such as feed-forward networks (FFN) and multi-head self-attention (MHSA) remains largely unexplored.
  arXiv  Detail & Related papers  (2023-10-08T16:16:21Z)
• Aligning Large Language Models with Human: A Survey [53.6014921995006]
  Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect information. This survey presents a comprehensive overview of these alignment technologies, including the following aspects.
  arXiv  Detail & Related papers  (2023-07-24T17:44:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.