Relevance is not enough: A Communication-Oriented Retrieval System for Consequential Scientific Question Answering

• URL: http://arxiv.org/abs/2609.06222v1
• Date: Sat, 05 Sep 2026 18:41:09 GMT
• Title: Relevance is not enough: A Communication-Oriented Retrieval System for Consequential Scientific Question Answering
• Abstract summary: We focus on consequential scientific questions whose results directly shape people's lives and study them through a public water-quality communication system.<n>Our system first classifies each question by reasoning type, then generates follow-up questions to identify missing evidence.<n>One component clearly distinguishes between what is known and what is uncertain, while another rewrites scientific details into accessible language.
• Score: 0.22499166814992436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems increasingly answer scientific questions about health, safety, and the environment. But most retrieval-augmented generation systems are tuned to provide factually correct, on-topic answers rather than to help non-experts understand what those answers mean for their lives and decisions. We focus on consequential scientific questions whose results directly shape people's lives and study them through a public water-quality communication system, where residents and community leaders interpret the findings and choose actions. Their experiences show that on-topic answers can still be insufficient without explanation and context and that the emotional weight of risk information cannot be ignored. Our system first classifies each question by reasoning type (for example, causal versus policy-based), then generates follow-up questions to identify missing evidence and retrieve it. One component clearly distinguishes between what is known and what is uncertain, while another rewrites scientific details into accessible language, using persona-based styles, such as a caring neighbor or an administrative official, to adapt tone and readability. Ablations on over $160$ questions show that the system uses $\textit{an order of magnitude less context}$ and, in several configurations, improves human-rated completeness. A completeness metric co-designed with community members and a fine-tuned learned judge reveal that standard relevance scores explain about $1\%$ of variation in human completeness ratings, and even the tuned judge only moderately aligns with humans, indicating that completeness is a distinct human-centered objective that current metrics do not reliably capture.

Related papers

• A dataset of rated conceptual arguments [49.54098427599201]
  We study large language models' ability to reason about conceptual questions.<n>Most philosophical questions are central components of questions in AI safety, decision theory, and social choice.<n>We introduce a dataset of 951 argumentative critiques of 442 position texts.
  arXiv  Detail & Related papers  (2026-07-29T22:30:59Z)
• Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance [73.19687314438133]
  We study how reliance is affected by contextual features of an interaction. We find that contextual characteristics significantly affect human reliance behavior. Our results show that calibration and language quality alone are insufficient in evaluating the risks of human-LM interactions.
  arXiv  Detail & Related papers  (2024-07-10T18:00:05Z)
• Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness [106.52630978891054]
  We present a taxonomy of uncertainty specific to vision-language AI systems. We also introduce a new metric confidence-weighted accuracy, that is well correlated with both accuracy and calibration error.
  arXiv  Detail & Related papers  (2024-07-02T04:23:54Z)
• ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales? [7.307538454513983]
  This study explores the alignment between ChatGPT and human assessments across multiple scales. We sample 300 data instances from three NLE datasets and collect 900 human annotations for both informativeness and clarity scores. Our results show that ChatGPT aligns better with humans in more coarse-grained scales.
  arXiv  Detail & Related papers  (2024-03-26T04:07:08Z)
• SADAS: A Dialogue Assistant System Towards Remediating Norm Violations in Bilingual Socio-Cultural Conversations [56.31816995795216]
  Socially-Aware Dialogue Assistant System (SADAS) is designed to ensure that conversations unfold with respect and understanding. Our system's novel architecture includes: (1) identifying the categories of norms present in the dialogue, (2) detecting potential norm violations, (3) evaluating the severity of these violations, and (4) implementing targeted remedies to rectify the breaches.
  arXiv  Detail & Related papers  (2024-01-29T08:54:21Z)
• Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach [98.13835740351932]
  This research aims to create a dataset and computational framework for systems that discuss and refine their predictions through dialogue. We show that the proposed system can have beneficial discussions with humans improving the accuracy by up to 25 points in the natural language inference task.
  arXiv  Detail & Related papers  (2023-05-19T16:24:50Z)
• Overinformative Question Answering by Humans and Machines [26.31070412632125]
  We show that overinformativeness in human answering is driven by considerations of relevance to the questioner's goals. We show that GPT-3 is highly sensitive to the form of the prompt and only human-like answer patterns when guided by an example and cognitively-motivated explanation.
  arXiv  Detail & Related papers  (2023-05-11T21:41:41Z)
• World Knowledge in Multiple Choice Reading Comprehension [0.0]
  Multiple choice reading comprehension systems are able to answer questions significantly better than random on average without access to the contextual passage. This paper examines the possibility of exploiting this observation as a tool for test designers to ensure that the use of "world knowledge" is acceptable for a particular set of questions. Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question.
  arXiv  Detail & Related papers  (2022-11-13T23:02:09Z)
• Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision [53.692793122749414]
  We introduce a medical question understanding and answering system with knowledge grounding and semantic self-supervision. Our system is a pipeline that first summarizes a long, medical, user-written question, using a supervised summarization loss. The system first matches the summarized user question with an FAQ from a trusted medical knowledge base, and then retrieves a fixed number of relevant sentences from the corresponding answer document.
  arXiv  Detail & Related papers  (2022-09-30T08:20:32Z)
• Interactive Question Answering Systems: Literature Review [17.033640293433397]
  Interactive question answering is a recently proposed and increasingly popular solution that resides at the intersection of question answering and dialogue systems. By permitting the user to ask more questions, interactive question answering enables users to dynamically interact with the system and receive more precise results. This survey offers a detailed overview of the interactive question-answering methods that are prevalent in current literature.
  arXiv  Detail & Related papers  (2022-09-04T13:46:54Z)
• Revealing Persona Biases in Dialogue Systems [64.96908171646808]
  We present the first large-scale study on persona biases in dialogue systems. We conduct analyses on personas of different social classes, sexual orientations, races, and genders. In our studies of the Blender and DialoGPT dialogue systems, we show that the choice of personas can affect the degree of harms in generated responses.
  arXiv  Detail & Related papers  (2021-04-18T05:44:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.