Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasets

• URL: http://arxiv.org/abs/2303.12898v1
• Date: Wed, 22 Mar 2023 20:26:30 GMT
• Title: Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasets
• Authors: Richard Tarbell, Kim-Kwang Raymond Choo, Glenn Dietrich and Anthony Rios
• Abstract summary: We show that there is still a long way to go before solving text-to-generation in the medical domain. We evaluate state-of-the-art language models showing substantial drops in performance with accuracy dropping from up to 92% to 28%. We introduce a novel data augmentation approach to improve the generalizability of relational language models.
• Score: 46.12592636378064
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Electronic medical records (EMRs) are stored in relational databases. It can be challenging to access the required information if the user is unfamiliar with the database schema or general database fundamentals. Hence, researchers have explored text-to-SQL generation methods that provide healthcare professionals direct access to EMR data without needing a database expert. However, currently available datasets have been essentially "solved" with state-of-the-art models achieving accuracy greater than or near 90%. In this paper, we show that there is still a long way to go before solving text-to-SQL generation in the medical domain. To show this, we create new splits of the existing medical text-to-SQL dataset MIMICSQL that better measure the generalizability of the resulting models. We evaluate state-of-the-art language models on our new split showing substantial drops in performance with accuracy dropping from up to 92% to 28%, thus showing substantial room for improvement. Moreover, we introduce a novel data augmentation approach to improve the generalizability of the language models. Overall, this paper is the first step towards developing more robust text-to-SQL models in the medical domain.\footnote{The dataset and code will be released upon acceptance.

Related papers

• LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs [58.59113843970975]
  Text-to-answer models are pivotal for making Electronic Health Records accessible to healthcare professionals without knowledge. We present a self-training strategy using pseudo-labeled un-answerable questions to enhance the reliability of text-to-answer models for EHRs.
  arXiv  Detail & Related papers  (2024-05-18T03:25:44Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• Retrieval augmented text-to-SQL generation for epidemiological question answering using electronic health records [0.6138671548064356]
  We introduce an end-to-end methodology that combines text-to-generation with retrieval augmented generation (RAG) to answer epidemiological questions. RAG offers a promising direction for improving their capabilities, as shown in a realistic industry setting.
  arXiv  Detail & Related papers  (2024-03-14T09:45:05Z)
• Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries [4.141402725050671]
  This paper is the first in-depth evaluation of the data model robustness of Text-to-- systems in practice. It is based on a real-world deployment of FootballDB, a system that was deployed over a 9 month period in the context of the FIFA World Cup 2022. All of our data is based on real user questions that were asked live to the system. We manually labeled and translated a subset of these questions for three different data models.
  arXiv  Detail & Related papers  (2024-02-13T10:28:57Z)
• Using text embedding models and vector databases as text classifiers with the example of medical data [0.0]
  We explore the use of vector databases and embedding models as a means of encoding, and classifying text with the example and application in the field of medicine. We show the robustness of these tools depends heavily on the sparsity of the data presented, and even with low amounts of data in the vector database itself, the vector database does a good job at classifying data.
  arXiv  Detail & Related papers  (2024-02-07T22:15:15Z)
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs [89.68522473384522]
  We present Bird, a big benchmark for large-scale database grounded in text-to-efficient tasks. Our emphasis on database values highlights the new challenges of dirty database contents. Even the most effective text-to-efficient models, i.e. ChatGPT, achieves only 40.08% in execution accuracy.
  arXiv  Detail & Related papers  (2023-05-04T19:02:29Z)
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training [86.91380874390778]
  We present Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. Based on experimental results, neural semantics that leverage GAP MODEL obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-generative benchmarks.
  arXiv  Detail & Related papers  (2020-12-18T15:53:50Z)
• IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation [61.09660709356527]
  We propose a database schema interaction graph encoder to utilize historicalal information of database schema items. We evaluate our model on the benchmark SParC and Co datasets.
  arXiv  Detail & Related papers  (2020-11-11T12:56:21Z)
• Data Agnostic RoBERTa-based Natural Language to SQL Query Generation [0.0]
  The NL2 task aims at finding deep learning approaches to solve the problem converting by natural language questions into valid queries. We have presented an approach with data privacy at its core. Although we have not achieved state of the art results, we have eliminated the need for the table right from the training of the model.
  arXiv  Detail & Related papers  (2020-10-11T13:18:46Z)
• Predicting Unplanned Readmissions with Highly Unstructured Data [0.0]
  Deep learning techniques have been successfully applied to predict unplanned readmissions of patients in medical centers. Most of the models proposed so far are tailored to English text data and assume that electronic medical records follow standards common in developed countries. We propose a deep learning architecture for predicting unplanned readmissions that consumes data that is significantly less structured compared with previous models in the literature.
  arXiv  Detail & Related papers  (2020-03-19T23:21:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.