YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discovery

• URL: http://arxiv.org/abs/2509.19182v1
• Date: Tue, 23 Sep 2025 15:57:42 GMT
• Title: YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discovery
• Authors: Devin Lange, Shanghua Gao, Pengwei Sui, Austen Money, Priya Misner, Marinka Zitnik, Nils Gehlenborg,
• Abstract summary: We bridge the gap between natural language and interactive visualizations by generating structured declarative output with a multi-agent system.<n>We include widgets, which allow users to adjust the values of that structured output through user interface elements.
• Score: 27.577426841656788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Incorporating natural language input has the potential to improve the capabilities of biomedical data discovery interfaces. However, user interface elements and visualizations are still powerful tools for interacting with data, even in the new world of generative AI. In our prototype system, YAC, Yet Another Chatbot, we bridge the gap between natural language and interactive visualizations by generating structured declarative output with a multi-agent system and interpreting that output to render linked interactive visualizations and apply data filters. Furthermore, we include widgets, which allow users to adjust the values of that structured output through user interface elements. We reflect on the capabilities and design of this system with an analysis of its technical dimensions and illustrate the capabilities through four usage scenarios.

Related papers

• A Generative AI System for Biomedical Data Discovery with Grammar-Based Visualizations [27.577426841656788]
  We explore the potential for combining generative AI with grammar-based visualizations for biomedical data discovery.<n>In our prototype, we use a multi-agent system to generate visualization specifications and apply filters.<n>These visualizations are linked together, resulting in an interactive dashboard that is progressively constructed.
  arXiv  Detail & Related papers  (2025-09-19T22:20:24Z)
• Generative Interfaces for Language Models [70.25765232527762]
  We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with humans preferring them in over 70% of cases.
  arXiv  Detail & Related papers  (2025-08-26T17:43:20Z)
• InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions [22.007942964950217]
  We develop InterChat, a generative visual analytics system that combines direct manipulation of visual elements with natural language inputs.<n>This integration enables precise intent communication and supports progressive, visually driven exploratory data analyses.
  arXiv  Detail & Related papers  (2025-03-06T05:35:19Z)
• LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models [50.259006481656094]
  We present a novel interactive application aimed towards understanding the internal mechanisms of large vision-language models. Our interface is designed to enhance the interpretability of the image patches, which are instrumental in generating an answer. We present a case study of how our application can aid in understanding failure mechanisms in a popular large multi-modal model: LLaVA.
  arXiv  Detail & Related papers  (2024-04-03T23:57:34Z)
• Large Language User Interfaces: Voice Interactive User Interfaces powered by LLMs [5.06113628525842]
  We present a framework that can serve as an intermediary between a user and their user interface (UI) We employ a system that stands upon textual semantic mappings of UI components, in the form of annotations. Our engine can classify the most appropriate application, extract relevant parameters, and subsequently execute precise predictions of the user's expected actions.
  arXiv  Detail & Related papers  (2024-02-07T21:08:49Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• SHINE: Syntax-augmented Hierarchical Interactive Encoder for Zero-shot Cross-lingual Information Extraction [47.88887327545667]
  In this study, a syntax-augmented hierarchical interactive encoder (SHINE) is proposed to transfer cross-lingual IE knowledge. SHINE is capable of interactively capturing complementary information between features and contextual information. Experiments across seven languages on three IE tasks and four benchmarks verify the effectiveness and generalization ability of the proposed method.
  arXiv  Detail & Related papers  (2023-05-21T08:02:06Z)
• NL2INTERFACE: Interactive Visualization Interface Generation from Natural Language Queries [19.355412315639462]
  NL2INTERFACE generates interactive multi-visualization interfaces from natural language queries. Users can interact with the interfaces to easily transform the data and quickly see the results in the visualizations.
  arXiv  Detail & Related papers  (2022-09-19T08:31:50Z)
• GenNI: Human-AI Collaboration for Data-Backed Text Generation [102.08127062293111]
  Table2Text systems generate textual output based on structured data utilizing machine learning. GenNI (Generation Negotiation Interface) is an interactive visual system for high-level human-AI collaboration in producing descriptive text.
  arXiv  Detail & Related papers  (2021-10-19T18:07:07Z)
• INVIGORATE: Interactive Visual Grounding and Grasping in Clutter [56.00554240240515]
  INVIGORATE is a robot system that interacts with human through natural language and grasps a specified object in clutter. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. We build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules.
  arXiv  Detail & Related papers  (2021-08-25T07:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.