Bayesian Preference Elicitation with Language Models
- URL: http://arxiv.org/abs/2403.05534v1
- Date: Fri, 8 Mar 2024 18:57:52 GMT
- Title: Bayesian Preference Elicitation with Language Models
- Authors: Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas,
Alex Tamkin, Belinda Z. Li
- Abstract summary: We introduce OPEN, a framework that uses BOED to guide the choice of informative questions and an LM to extract features.
In user studies, we find that OPEN outperforms existing LM- and BOED-based methods for preference elicitation.
- Score: 82.58230273253939
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Aligning AI systems to users' interests requires understanding and
incorporating humans' complex values and preferences. Recently, language models
(LMs) have been used to gather information about the preferences of human
users. This preference data can be used to fine-tune or guide other LMs and/or
AI systems. However, LMs have been shown to struggle with crucial aspects of
preference learning: quantifying uncertainty, modeling human mental states, and
asking informative questions. These challenges have been addressed in other
areas of machine learning, such as Bayesian Optimal Experimental Design (BOED),
which focus on designing informative queries within a well-defined feature
space. But these methods, in turn, are difficult to scale and apply to
real-world problems where simply identifying the relevant features can be
difficult. We introduce OPEN (Optimal Preference Elicitation with Natural
language) a framework that uses BOED to guide the choice of informative
questions and an LM to extract features and translate abstract BOED queries
into natural language questions. By combining the flexibility of LMs with the
rigor of BOED, OPEN can optimize the informativity of queries while remaining
adaptable to real-world domains. In user studies, we find that OPEN outperforms
existing LM- and BOED-based methods for preference elicitation.
Related papers
- LLM-assisted Explicit and Implicit Multi-interest Learning Framework for Sequential Recommendation [50.98046887582194]
We propose an explicit and implicit multi-interest learning framework to model user interests on two levels: behavior and semantics.
The proposed EIMF framework effectively and efficiently combines small models with LLM to improve the accuracy of multi-interest modeling.
arXiv Detail & Related papers (2024-11-14T13:00:23Z) - Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models [33.488331159912136]
Instruction tuning plays a critical role in aligning large language models (LLMs) with human preference.
Data assessment and selection methods have been proposed in the fields of natural language processing (NLP) and deep learning.
We present a comprehensive review on existing literature of data assessment and selection especially for instruction tuning of LLMs.
arXiv Detail & Related papers (2024-08-04T16:50:07Z) - Language Representations Can be What Recommenders Need: Findings and Potentials [57.90679739598295]
We show that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance.
This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation.
Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.
arXiv Detail & Related papers (2024-07-07T17:05:24Z) - Aligning Large Language Models from Self-Reference AI Feedback with one General Principle [61.105703857868775]
We propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback.
Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference.
Finally, we determine which answer better fits human preferences according to the criticism.
arXiv Detail & Related papers (2024-06-17T03:51:46Z) - Crafting Interpretable Embeddings by Asking LLMs Questions [89.49960984640363]
Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks.
We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM.
We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli.
arXiv Detail & Related papers (2024-05-26T22:30:29Z) - Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation [18.550311424902358]
Large language models (LLMs) enable fully natural language (NL) PE dialogues.
We propose a novel NL-PE algorithm, PEBOL, which uses Natural Language Inference (NLI) between user preference utterances and NL item descriptions.
We numerically evaluate our methods in controlled simulations, finding that PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline's MRR@10 of 0.17.
arXiv Detail & Related papers (2024-05-02T03:35:21Z) - Preference-Conditioned Language-Guided Abstraction [24.626805570296064]
We observe that how humans behave reveals how they see the world.
In this work, we propose using language models (LMs) to query for those preferences directly given knowledge that a change in behavior has occurred.
We demonstrate our framework's ability to construct effective preference-conditioned abstractions in simulated experiments, a user study, and on a real Spot robot performing mobile manipulation tasks.
arXiv Detail & Related papers (2024-02-05T15:12:15Z) - Active Preference Inference using Language Models and Probabilistic Reasoning [13.523369679010685]
We introduce an inference-time algorithm that helps large language models infer user preferences.
Our algorithm uses a probabilistic model whose conditional distributions are defined by prompting an LLM.
Results in a simplified interactive web shopping setting with real product items show that an LLM equipped with our entropy reduction algorithm outperforms baselines.
arXiv Detail & Related papers (2023-12-19T09:58:54Z) - Adapting LLMs for Efficient, Personalized Information Retrieval: Methods
and Implications [0.7832189413179361]
Large Language Models (LLMs) excel in comprehending and generating human-like text.
This paper explores strategies for integrating Language Models (LLMs) with Information Retrieval (IR) systems.
arXiv Detail & Related papers (2023-11-21T02:01:01Z) - Eliciting Human Preferences with Language Models [56.68637202313052]
Language models (LMs) can be directed to perform target tasks by using labeled examples or natural language prompts.
We propose to use *LMs themselves* to guide the task specification process.
We study GATE in three domains: email validation, content recommendation, and moral reasoning.
arXiv Detail & Related papers (2023-10-17T21:11:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.