Bayesian Preference Elicitation with Language Models

• URL: http://arxiv.org/abs/2403.05534v1
• Date: Fri, 8 Mar 2024 18:57:52 GMT
• Title: Bayesian Preference Elicitation with Language Models
• Authors: Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, Belinda Z. Li
• Abstract summary: We introduce OPEN, a framework that uses BOED to guide the choice of informative questions and an LM to extract features. In user studies, we find that OPEN outperforms existing LM- and BOED-based methods for preference elicitation.
• Score: 82.58230273253939
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aligning AI systems to users' interests requires understanding and incorporating humans' complex values and preferences. Recently, language models (LMs) have been used to gather information about the preferences of human users. This preference data can be used to fine-tune or guide other LMs and/or AI systems. However, LMs have been shown to struggle with crucial aspects of preference learning: quantifying uncertainty, modeling human mental states, and asking informative questions. These challenges have been addressed in other areas of machine learning, such as Bayesian Optimal Experimental Design (BOED), which focus on designing informative queries within a well-defined feature space. But these methods, in turn, are difficult to scale and apply to real-world problems where simply identifying the relevant features can be difficult. We introduce OPEN (Optimal Preference Elicitation with Natural language) a framework that uses BOED to guide the choice of informative questions and an LM to extract features and translate abstract BOED queries into natural language questions. By combining the flexibility of LMs with the rigor of BOED, OPEN can optimize the informativity of queries while remaining adaptable to real-world domains. In user studies, we find that OPEN outperforms existing LM- and BOED-based methods for preference elicitation.

Related papers

• Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models [33.488331159912136]
  Instruction tuning plays a critical role in aligning large language models (LLMs) with human preference. Data assessment and selection methods have been proposed in the fields of natural language processing (NLP) and deep learning. We present a comprehensive review on existing literature of data assessment and selection especially for instruction tuning of LLMs.
  arXiv  Detail & Related papers  (2024-08-04T16:50:07Z)
• Language Representations Can be What Recommenders Need: Findings and Potentials [57.90679739598295]
  We show that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance. This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation. Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.
  arXiv  Detail & Related papers  (2024-07-07T17:05:24Z)
• Aligning Large Language Models from Self-Reference AI Feedback with one General Principle [61.105703857868775]
  We propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference. Finally, we determine which answer better fits human preferences according to the criticism.
  arXiv  Detail & Related papers  (2024-06-17T03:51:46Z)
• Crafting Interpretable Embeddings by Asking LLMs Questions [89.49960984640363]
  Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks. We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM. We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli.
  arXiv  Detail & Related papers  (2024-05-26T22:30:29Z)
• Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation [18.550311424902358]
  Large language models (LLMs) enable fully natural language (NL) PE dialogues. We propose a novel NL-PE algorithm, PEBOL, which uses Natural Language Inference (NLI) between user preference utterances and NL item descriptions. We numerically evaluate our methods in controlled simulations, finding that PEBOL can achieve an MRR@10 of up to 0.27 compared to the best monolithic LLM baseline's MRR@10 of 0.17.
  arXiv  Detail & Related papers  (2024-05-02T03:35:21Z)
• Preference-Conditioned Language-Guided Abstraction [24.626805570296064]
  We observe that how humans behave reveals how they see the world. In this work, we propose using language models (LMs) to query for those preferences directly given knowledge that a change in behavior has occurred. We demonstrate our framework's ability to construct effective preference-conditioned abstractions in simulated experiments, a user study, and on a real Spot robot performing mobile manipulation tasks.
  arXiv  Detail & Related papers  (2024-02-05T15:12:15Z)
• Active Preference Inference using Language Models and Probabilistic Reasoning [13.523369679010685]
  We introduce an inference-time algorithm that helps large language models infer user preferences. Our algorithm uses a probabilistic model whose conditional distributions are defined by prompting an LLM. Results in a simplified interactive web shopping setting with real product items show that an LLM equipped with our entropy reduction algorithm outperforms baselines.
  arXiv  Detail & Related papers  (2023-12-19T09:58:54Z)
• Adapting LLMs for Efficient, Personalized Information Retrieval: Methods and Implications [0.7832189413179361]
  Large Language Models (LLMs) excel in comprehending and generating human-like text. This paper explores strategies for integrating Language Models (LLMs) with Information Retrieval (IR) systems.
  arXiv  Detail & Related papers  (2023-11-21T02:01:01Z)
• Eliciting Human Preferences with Language Models [56.68637202313052]
  Language models (LMs) can be directed to perform target tasks by using labeled examples or natural language prompts. We propose to use *LMs themselves* to guide the task specification process. We study GATE in three domains: email validation, content recommendation, and moral reasoning.
  arXiv  Detail & Related papers  (2023-10-17T21:11:21Z)
• Offline RL for Natural Language Generation with Implicit Language Q Learning [87.76695816348027]
  Large language models can be inconsistent when it comes to completing user specified tasks. We propose a novel RL method, that combines both the flexible utility framework of RL with the ability of supervised learning. In addition to empirically validating ILQL, we present a detailed empirical analysis situations where offline RL can be useful in natural language generation settings.
  arXiv  Detail & Related papers  (2022-06-05T18:38:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.