RPS: Information Elicitation with Reinforcement Prompt Selection

• URL: http://arxiv.org/abs/2604.13817v1
• Date: Wed, 15 Apr 2026 12:51:01 GMT
• Title: RPS: Information Elicitation with Reinforcement Prompt Selection
• Abstract summary: In many interactive AI applications, users often withhold sensitive or uncertain information due to privacy concerns, ambiguity, or social hesitation.<n>We propose Reinforcement Prompt Selection (RPS), a lightweight reinforcement learning framework that formulates prompt selection as a sequential decision-making problem.<n>RPS learns a policy over a pool of prompts to adaptively elicit concealed or incompletely expressed information from users through dialogue.
• Score: 16.448966784570313
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have shown remarkable capabilities in dialogue generation and reasoning, yet their effectiveness in eliciting user-known but concealed information in open-ended conversations remains limited. In many interactive AI applications, such as personal assistants, tutoring systems, and legal or clinical support, users often withhold sensitive or uncertain information due to privacy concerns, ambiguity, or social hesitation. This makes it challenging for LLMs to gather complete and contextually relevant inputs. In this work, we define the problem of information elicitation in open-ended dialogue settings and propose Reinforcement Prompt Selection (RPS), a lightweight reinforcement learning framework that formulates prompt selection as a sequential decision-making problem. To analyze this problem in a controlled setting, we design a synthetic experiment, where a reinforcement learning agent outperforms a random query baseline, illustrating the potential of policy-based approaches for adaptive information elicitation. Building on this insight, RPS learns a policy over a pool of prompts to adaptively elicit concealed or incompletely expressed information from users through dialogue. We also introduce IELegal, a new benchmark dataset constructed from real legal case documents, which simulates dialogue-based information elicitation tasks aimed at uncovering case-relevant facts. In this setting, RPS outperforms static prompt baselines, demonstrating the effectiveness of adaptive prompt selection for eliciting critical information in LLM-driven dialogue systems.

Related papers

• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• PrivacyPAD: A Reinforcement Learning Framework for Dynamic Privacy-Aware Delegation [33.37227619820212]
  We introduce a novel reinforcement learning framework called PrivacyPAD to solve this problem.<n>Our framework trains an agent to dynamically route text chunks, learning a policy that optimally balances the trade-off between privacy leakage and task performance.<n>Our framework achieves a new state-of-the-art on the privacy-utility frontier.
  arXiv  Detail & Related papers  (2025-10-16T19:38:36Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• Contextual Integrity in LLMs via Reasoning and Reinforcement Learning [41.795843170879046]
  We develop a reinforcement learning framework that instills in models the reasoning necessary to achieve contextual integrity.<n>We show that our method substantially reduces inappropriate information disclosure while maintaining task performance.
  arXiv  Detail & Related papers  (2025-05-29T21:26:21Z)
• From Reviews to Dialogues: Active Synthesis for Zero-Shot LLM-based Conversational Recommender System [49.57258257916805]
  Large Language Models (LLMs) demonstrate strong zero-shot recommendation capabilities.<n>Practical applications often favor smaller, internally managed recommender models due to scalability, interpretability, and data privacy constraints.<n>We propose an active data augmentation framework that synthesizes conversational training data by leveraging black-box LLMs guided by active learning techniques.
  arXiv  Detail & Related papers  (2025-04-21T23:05:47Z)
• Few-shot Policy (de)composition in Conversational Question Answering [54.259440408606515]
  We propose a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting.<n>We show that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies.<n>We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning.
  arXiv  Detail & Related papers  (2025-01-20T08:40:15Z)
• Selective Prompting Tuning for Personalized Conversations with LLMs [31.28284591597932]
  We propose textbfSelective textbfPrompt textbfTuning (SPT), which softly prompts large language models (LLMs) for personalized conversations in a selective way. SPT significantly enhances response diversity by up to 90%, along with improvements in other critical performance indicators.
  arXiv  Detail & Related papers  (2024-06-26T09:03:52Z)
• Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Training [33.57497419019826]
  Action-Based Contrastive Self-Training enables data-efficient dialogue policy learning in multi-turn conversation modeling.<n>We demonstrate ACT's efficacy under in data-efficient tuning scenarios, even when there is no action label available.<n>We also propose evaluating LLMs' ability to function as conversational agents by examining whether they can implicitly recognize and reason about ambiguity in conversation.
  arXiv  Detail & Related papers  (2024-05-31T22:44:48Z)
• Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents [121.46051697742608]
  We introduce a new dialogue policy planning paradigm to strategize dialogue problems with a tunable language model plug-in named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data. PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications.
  arXiv  Detail & Related papers  (2023-11-01T03:20:16Z)
• Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration [72.04629217161656]
  This work focuses on three aspects of proactive dialogue systems: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thought prompting scheme.
  arXiv  Detail & Related papers  (2023-05-23T02:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.