YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents

• URL: http://arxiv.org/abs/2604.10968v1
• Date: Mon, 13 Apr 2026 04:12:58 GMT
• Title: YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents
• Abstract summary: We introduce Information Elicitation Agents (IEAs) in which the agent's goal is to elicit information from users to support the agent's institutional or task-oriented objectives.<n>To enable systematic research on this setting, we present YIELD, a 26M-token dataset of 2,281 ethically sourced, human-to-human dialogues.<n>We formalize information elicitation as a finite-horizon POMDP and propose novel metrics tailored to IEAs.
• Score: 3.909651701500542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academic interviewing, judicial proceedings, and journalistic investigations, involve broader institutional decision-making processes and require agents that can elicit information from users. In this paper, we introduce Information Elicitation Agents (IEAs) in which the agent's goal is to elicit information from users to support the agent's institutional or task-oriented objectives. To enable systematic research on this setting, we present YIELD, a 26M-token dataset of 2,281 ethically sourced, human-to-human dialogues. Moreover, we formalize information elicitation as a finite-horizon POMDP and propose novel metrics tailored to IEAs. Pilot experiments on multiple foundation LLMs show that training on YIELD improves their alignment with real elicitation behavior and findings are corroborated by human evaluation. We release YIELD under CC BY 4.0. The dataset, project code, evaluation tools, and fine-tuned model adapters are available at: https://github.com/infosenselab/yield.

Related papers

• Insight Agents: An LLM-Based Multi-Agent System for Data Insights [6.3353477880949045]
  We aim to develop Insight Agents (IA), a conversational multi-agent Data Insight system, to provide E-commerce sellers with personalized data and business insights.<n>IA has been launched for Amazon sellers in US, which has achieved high accuracy of 90% based on human evaluation, with latency of P90 below 15s.
  arXiv  Detail & Related papers  (2026-01-27T20:51:01Z)
• Data-Dependent Goal Modeling for ML-Enabled Law Enforcement Systems [2.8686050325188153]
  This paper reports our experience applying the GORE framework KAOS to designing an ML-enabled system for identifying suspects in online child sexual abuse.<n>A key finding is the central role of data elicitation: data requirements constrain refinement choices and candidate agents while influencing how goals are linked, operationalized, and satisfied.<n>We contribute a reference model for integrating GORE with data-driven system development, and identify gaps in KAOS, particularly the need for explicit support for data elicitation and quality management.
  arXiv  Detail & Related papers  (2026-01-09T15:22:02Z)
• How can we assess human-agent interactions? Case studies in software agent design [52.953425368394306]
  We make two major steps towards the rigorous assessment of human-agent interactions.<n>We propose PULSE, a framework for more efficient human-centric evaluation of agent designs.<n>We deploy the framework on a large-scale web platform built around the open-source software agent OpenHands.
  arXiv  Detail & Related papers  (2025-10-10T19:04:28Z)
• LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback [121.78866929908871]
  Large Action Models (LAMs) for AI Agents offer incredible potential but face challenges due to the need for high-quality training data.<n>We present LAM SIMULATOR, a comprehensive framework designed for online exploration of agentic tasks with high-quality feedback.<n>Our framework features a dynamic task query generator, an extensive collection of tools, and an interactive environment where Large Language Model (LLM) Agents can call tools and receive real-time feedback.
  arXiv  Detail & Related papers  (2025-06-02T22:36:02Z)
• Contextual Integrity in LLMs via Reasoning and Reinforcement Learning [41.795843170879046]
  We develop a reinforcement learning framework that instills in models the reasoning necessary to achieve contextual integrity.<n>We show that our method substantially reduces inappropriate information disclosure while maintaining task performance.
  arXiv  Detail & Related papers  (2025-05-29T21:26:21Z)
• InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generation [63.55258191625131]
  InfoDeepSeek is a new benchmark for assessing agentic information seeking in real-world, dynamic web environments.<n>We propose a systematic methodology for constructing challenging queries satisfying the criteria of determinacy, difficulty, and diversity.<n>We develop the first evaluation framework tailored to dynamic agentic information seeking, including fine-grained metrics about the accuracy, utility, and compactness of information seeking outcomes.
  arXiv  Detail & Related papers  (2025-05-21T14:44:40Z)
• R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science [70.1638335489284]
  High-level machine learning engineering tasks remain labor-intensive and iterative.<n>We introduce R&D-Agent, a comprehensive, decoupled, and framework that formalizes the machine learning process.<n>R&D-Agent defines the MLE into two phases and six components, turning agent design for MLE into a principled, testable process.
  arXiv  Detail & Related papers  (2025-05-20T06:07:00Z)
• AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents [66.29263282311258]
  We introduce a new benchmark AgentDAM that measures if AI web-navigation agents follow the privacy principle of data minimization''<n>Our benchmark simulates realistic web interaction scenarios end-to-end and is adaptable to all existing web navigation agents.
  arXiv  Detail & Related papers  (2025-03-12T19:30:31Z)
• Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems [6.8738526619759535]
  offline datasets have been used to evaluate task-oriented dialogue (TOD) models. User-agents, which are context-aware, can simulate the variability and unpredictability of human conversations.
  arXiv  Detail & Related papers  (2024-11-15T06:05:45Z)
• Ask-before-Plan: Proactive Language Agents for Real-World Planning [68.08024918064503]
  Proactive Agent Planning requires language agents to predict clarification needs based on user-agent conversation and agent-environment interaction. We propose a novel multi-agent framework, Clarification-Execution-Planning (textttCEP), which consists of three agents specialized in clarification, execution, and planning.
  arXiv  Detail & Related papers  (2024-06-18T14:07:28Z)
• Towards a RAG-based Summarization Agent for the Electron-Ion Collider [0.5504260452953508]
  A Retrieval Augmented Generation (RAG)--based Summarization AI for EIC (RAGS4EIC) is under development. This AI-Agent not only condenses information but also effectively references relevant responses, offering substantial advantages for collaborators. Our project involves a two-step approach: first, querying a comprehensive vector database containing all pertinent experiment information; second, utilizing a Large Language Model (LLM) to generate concise summaries enriched with citations based on user queries and retrieved data.
  arXiv  Detail & Related papers  (2024-03-23T05:32:46Z)
• INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions [47.90088587508672]
  InSCIt is a dataset for Information-Seeking Conversations with mixed-initiative Interactions. It contains 4.7K user-agent turns from 805 human-human conversations. We report results of two systems based on state-of-the-art models of conversational knowledge identification and open-domain question answering.
  arXiv  Detail & Related papers  (2022-07-02T06:18:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.