From Distributional to Overton Pluralism: Investigating Large Language Model Alignment

• URL: http://arxiv.org/abs/2406.17692v1
• Date: Tue, 25 Jun 2024 16:32:33 GMT
• Title: From Distributional to Overton Pluralism: Investigating Large Language Model Alignment
• Authors: Thom Lake, Eunsol Choi, Greg Durrett,
• Abstract summary: We re-examine previously reported reductions in response diversity post-alignment. Our analysis suggests that an apparent drop in the diversity of responses is largely explained by quality control and information aggregation. Findings indicate that current alignment techniques capture but do not extend the useful subset of assistant-like base LLM behavior.
• Score: 82.99849359892112
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The alignment process changes several properties of a large language model's (LLM's) output distribution. We analyze two aspects of post-alignment distributional shift of LLM responses. First, we re-examine previously reported reductions in response diversity post-alignment. Our analysis suggests that an apparent drop in the diversity of responses is largely explained by quality control and information aggregation. Alignment suppresses irrelevant and unhelpful content while shifting the output distribution toward longer responses that cover information spanning several responses from the base LLM, essentially presenting diverse information in a single response. Finding little evidence that alignment suppresses useful information, it is natural to ask the opposite question: do aligned models surface information that cannot be recovered from base models? Our second investigation shows this is not the case and the behavior of aligned models is recoverable from base models without fine-tuning. A combination of in-context examples and lower-resolution semantic hints about response content can elicit responses from base LLMs that are as similar to alignment-tuned LLM responses as alignment-tuned LLM responses are to each other. Taken together, these results indicate that current alignment techniques capture but do not extend the useful subset of assistant-like base LLM behavior, providing further evidence for the Superficial Alignment Hypothesis. They also show that in-context alignment can go surprisingly far as a strategy for imitating aligned LLMs without fine-tuning. Our code and data is available at https://github.com/thomlake/investigating-alignment.

Related papers

• Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuning [68.57166425493283]
  Refusal-Aware Instruction Tuning (RAIT) enables Large Language Models (LLMs) to refuse to answer unknown questions. RAIT modifies training samples based on the correctness of the initial LLM's response. This crude approach can cause LLMs to excessively refuse answering questions they could have correctly answered.
  arXiv  Detail & Related papers  (2024-10-09T14:12:51Z)
• Understanding Alignment in Multimodal LLMs: A Comprehensive Study [46.33812471516309]
  We analyze each aspect of preference alignment in Multimodal Large Language Models (MLLMs) We show that combining offline and online methods can improve the performance of the model in certain scenarios. We introduce a novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS)
  arXiv  Detail & Related papers  (2024-07-02T17:55:03Z)
• CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation [76.31621715032558]
  Grounded generation aims to equip language models (LMs) with the ability to produce more credible and accountable responses. We introduce CaLM, a novel verification framework. Our framework empowers smaller LMs, which rely less on parametric memory, to validate the output of larger LMs.
  arXiv  Detail & Related papers  (2024-06-08T06:04:55Z)
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning [61.68787689234622]
  A recent study, LIMA, shows that using merely 1K examples for alignment tuning can achieve significant alignment performance as well. This raises questions about how exactly the alignment tuning transforms a base LLM. We show that the gap between tuning-free and tuning-based alignment methods can be significantly reduced through strategic prompting.
  arXiv  Detail & Related papers  (2023-12-04T00:46:11Z)
• Effective Large Language Model Adaptation for Improved Grounding and Citation Generation [48.07830615309543]
  This paper focuses on improving large language models (LLMs) by grounding their responses in retrieved passages and by providing citations. We propose a new framework, AGREE, that improves the grounding from a holistic perspective. Our framework tunes LLMs to selfground the claims in their responses and provide accurate citations to retrieved documents.
  arXiv  Detail & Related papers  (2023-11-16T03:22:25Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)
• Enhancing In-Context Learning with Answer Feedback for Multi-Span Question Answering [9.158919909909146]
  In this paper, we propose a novel way of employing labeled data such as it informs LLM of some undesired output. Experiments on three multi-span question answering datasets and a keyphrase extraction dataset show that our new prompting strategy consistently improves LLM's in-context learning performance.
  arXiv  Detail & Related papers  (2023-06-07T15:20:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.