Group Preference Optimization: Few-Shot Alignment of Large Language
Models
- URL: http://arxiv.org/abs/2310.11523v1
- Date: Tue, 17 Oct 2023 18:41:57 GMT
- Title: Group Preference Optimization: Few-Shot Alignment of Large Language
Models
- Authors: Siyan Zhao, John Dang, Aditya Grover
- Abstract summary: Group Preference Optimization steers language models to preferences of individual groups in a few-shot manner.
We empirically validate the efficacy of GPO through rigorous evaluations using large language models with varied sizes.
Our results demonstrate that GPO not only aligns models more accurately but also requires fewer group-specific preferences, and less training and inference computing resources.
- Score: 31.991620847943036
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Many applications of large language models (LLMs), ranging from chatbots to
creative writing, require nuanced subjective judgments that can differ
significantly across different groups. Existing alignment algorithms can be
expensive to align for each group, requiring prohibitive amounts of
group-specific preference data and computation for real-world use cases. We
introduce Group Preference Optimization (GPO), an alignment framework that
steers language models to preferences of individual groups in a few-shot
manner. In GPO, we augment the base LLM with an independent transformer module
trained to predict the preferences of a group for the LLM generations. For
few-shot learning, we parameterize this module as an in-context autoregressive
transformer and train it via meta-learning on several groups. We empirically
validate the efficacy of GPO through rigorous evaluations using LLMs with
varied sizes on three human opinion adaptation tasks. These tasks involve
adapting to the preferences of US demographic groups, global countries, and
individual users. Our results demonstrate that GPO not only aligns models more
accurately but also requires fewer group-specific preferences, and less
training and inference computing resources, outperforming existing strategies
such as in-context steering and fine-tuning methods.
Related papers
- SoupLM: Model Integration in Large Language and Multi-Modal Models [51.12227693121004]
Training large language models (LLMs) requires significant computing resources.
Existing publicly available LLMs are typically pre-trained on diverse, privately curated datasets spanning various tasks.
arXiv Detail & Related papers (2024-07-11T05:38:15Z) - Pareto-Optimal Learning from Preferences with Hidden Context [18.340302968130683]
We propose POPL, which frames discrepant group preferences as objectives with potential trade-offs.
Our empirical evaluations demonstrate that POPL surpasses baseline methods in learning sets of reward functions.
POPL can serve as a foundation for techniques optimizing specific notions of group fairness.
arXiv Detail & Related papers (2024-06-21T18:57:38Z) - Group Robust Preference Optimization in Reward-free RLHF [23.622835830345725]
We propose a novel Group Robust Preference Optimization (GRPO) method to align large language models to individual groups' preferences robustly.
To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss.
We significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies.
arXiv Detail & Related papers (2024-05-30T17:50:04Z) - Self-Exploring Language Models: Active Preference Elicitation for Online Alignment [90.4820014819937]
We propose a bilevel objective optimistically biased towards potentially high-reward responses to actively explore out-of-distribution regions.
Our experimental results demonstrate that when finetuned on Zephyr-7B-SFT and Llama-3-8B-Instruct models, SELM significantly boosts the performance on instruction-following benchmarks.
arXiv Detail & Related papers (2024-05-29T17:59:07Z) - Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
We introduce a novel closed-form formulation for direct preference optimization using multiple reference models.
The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models.
Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
arXiv Detail & Related papers (2024-05-26T00:29:04Z) - Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts [95.09994361995389]
Relative Preference Optimization (RPO) is designed to discern between more and less preferred responses derived from both identical and related prompts.
RPO has demonstrated a superior ability to align large language models with user preferences and to improve their adaptability during the training process.
arXiv Detail & Related papers (2024-02-12T22:47:57Z) - Do Membership Inference Attacks Work on Large Language Models? [145.90022632726883]
Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data.
We perform a large-scale evaluation of MIAs over a suite of language models trained on the Pile, ranging from 160M to 12B parameters.
We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains.
arXiv Detail & Related papers (2024-02-12T17:52:05Z) - Active Preference Inference using Language Models and Probabilistic Reasoning [13.523369679010685]
We introduce an inference-time algorithm that helps large language models infer user preferences.
Our algorithm uses a probabilistic model whose conditional distributions are defined by prompting an LLM.
Results in a simplified interactive web shopping setting with real product items show that an LLM equipped with our entropy reduction algorithm outperforms baselines.
arXiv Detail & Related papers (2023-12-19T09:58:54Z) - Overcoming Data Sparsity in Group Recommendation [52.00998276970403]
Group recommender systems should be able to accurately learn not only users' personal preferences but also preference aggregation strategy.
In this paper, we take Bipartite Graphding Model (BGEM), the self-attention mechanism and Graph Convolutional Networks (GCNs) as basic building blocks to learn group and user representations in a unified way.
arXiv Detail & Related papers (2020-10-02T07:11:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.