SPRI: Aligning Large Language Models with Context-Situated Principles

• URL: http://arxiv.org/abs/2502.03397v1
• Date: Wed, 05 Feb 2025 17:32:29 GMT
• Title: SPRI: Aligning Large Language Models with Context-Situated Principles
• Authors: Hongli Zhan, Muneeza Azmat, Raya Horesh, Junyi Jessy Li, Mikhail Yurochkin,
• Abstract summary: Situated-PRInciples (SPRI) is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones.
• Score: 53.07731637246485
• License:
• Abstract: Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https://github.com/honglizhan/SPRI-public.

Related papers

• Optimizing Large Language Models for Dynamic Constraints through Human-in-the-Loop Discriminators [0.0]
  Large Language Models (LLMs) have recently demonstrated impressive capabilities across various real-world applications. We propose a flexible framework that enables LLMs to interact with system interfaces, summarize constraint concepts, and continually optimize performance metrics. Our framework achieved a $7.78%$ pass rate with the human discriminator and a $6.11%$ pass rate with the LLM-based discriminator.
  arXiv  Detail & Related papers  (2024-10-19T17:27:38Z)
• P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task [94.08478298711789]
  Embodied Everyday Task is a popular task in the embodied AI community. Natural language instructions often lack explicit task planning. Extensive training is required to equip models with knowledge of the task environment.
  arXiv  Detail & Related papers  (2024-09-17T15:29:34Z)
• Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Values [47.779186412943076]
  We propose a novel basic value alignment paradigm and a value space spanned by basic value dimensions. Inspired by basic values in humanity and social science across cultures, this work proposes a novel basic value alignment paradigm and a value space spanned by basic value dimensions. To foster future research, we apply the representative Schwartz's Theory of Basic Values as an example and construct FULCRA, a dataset consisting of 5k (LLM output, value vector) pairs.
  arXiv  Detail & Related papers  (2023-11-15T10:29:28Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)
• Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision [84.31474052176343]
  Recent AI-assistant agents, such as ChatGPT, rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback to align the output with human intentions. This dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision. We propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision.
  arXiv  Detail & Related papers  (2023-05-04T17:59:28Z)
• IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases [53.054598423181844]
  We present a novel NER cascade approach comprising three steps. We empirically demonstrate the significance of external knowledge bases in accurately classifying fine-grained and emerging entities. Our system exhibits robust performance in the MultiCoNER2 shared task, even in the low-resource language setting.
  arXiv  Detail & Related papers  (2023-04-20T20:30:34Z)
• Machine Learning Approaches for Principle Prediction in Naturally Occurring Stories [9.652610879417326]
  We explore the use of machine learning models for the task of normative principle prediction on naturally occurring story data. We show that while individual principles can be classified, the ambiguity of what "moral principles" represent poses a challenge for both human participants and autonomous systems.
  arXiv  Detail & Related papers  (2022-11-19T03:14:23Z)
• LMPriors: Pre-Trained Language Models as Task-Specific Priors [78.97143833642971]
  We develop principled techniques for augmenting our models with suitable priors. This is to encourage them to learn in ways that are compatible with our understanding of the world. We draw inspiration from the recent successes of large-scale language models (LMs) to construct task-specific priors distilled from the rich knowledge of LMs.
  arXiv  Detail & Related papers  (2022-10-22T19:09:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.