LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs

• URL: http://arxiv.org/abs/2405.11162v1
• Date: Sat, 18 May 2024 03:25:44 GMT
• Title: LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs
• Authors: Yongrae Jo, Seongyun Lee, Minju Seo, Sung Ju Hwang, Moontae Lee,
• Abstract summary: Text-to-answer models are pivotal for making Electronic Health Records accessible to healthcare professionals without knowledge. We present a self-training strategy using pseudo-labeled un-answerable questions to enhance the reliability of text-to-answer models for EHRs.
• Score: 58.59113843970975
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-to-SQL models are pivotal for making Electronic Health Records (EHRs) accessible to healthcare professionals without SQL knowledge. With the advancements in large language models, these systems have become more adept at translating complex questions into SQL queries. Nonetheless, the critical need for reliability in healthcare necessitates these models to accurately identify unanswerable questions or uncertain predictions, preventing misinformation. To address this problem, we present a self-training strategy using pseudo-labeled unanswerable questions to enhance the reliability of text-to-SQL models for EHRs. This approach includes a two-stage training process followed by a filtering method based on the token entropy and query execution. Our methodology's effectiveness is validated by our top performance in the EHRSQL 2024 shared task, showcasing the potential to improve healthcare decision-making through more reliable text-to-SQL systems.

Related papers

• Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL [83.99974309930072]
  Knowledge distillation (KD) is a common approach, which aims to distill the larger teacher model into a smaller student model. We propose to improve the KD with Imperfect Data, namely KID, which effectively boosts the performance without introducing much training budget. KID can not only achieve consistent and significant performance gains across all model types and sizes, but also effectively improve the training efficiency.
  arXiv  Detail & Related papers  (2024-10-15T07:51:00Z)
• PromptMind Team at EHRSQL-2024: Improving Reliability of SQL Generation using Ensemble LLMs [0.0]
  We propose two approaches that leverage large language models for prompting and fine-tuning to generate EHR queries. In both techniques, we concentrate on bridging the gap between the real-world knowledge on which LLMs are trained and the domain specific knowledge required for the task. We show that an approach further enhances generation reliability by reducing errors.
  arXiv  Detail & Related papers  (2024-05-14T07:16:56Z)
• Overview of the EHRSQL 2024 Shared Task on Reliable Text-to-SQL Modeling on Electronic Health Records [12.692089512684955]
  One strategy is to build a question-answering system, possibly leveraging text-to- relational models. The EHR 2024 shared task aims to advance and promote research in developing a question-answering system for EHRs. Among more than 100 participants who applied to the shared task, eight teams were formed and completed the entire shared task requirement.
  arXiv  Detail & Related papers  (2024-05-04T04:12:18Z)
• ProbGate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling [0.0]
  We introduce an entropy-based method to identify and filter out unanswerable results. We experimentally verified that our method can filter unanswerable questions, which can be widely utilized.
  arXiv  Detail & Related papers  (2024-04-25T14:55:07Z)
• Retrieval augmented text-to-SQL generation for epidemiological question answering using electronic health records [0.6138671548064356]
  We introduce an end-to-end methodology that combines text-to-generation with retrieval augmented generation (RAG) to answer epidemiological questions. RAG offers a promising direction for improving their capabilities, as shown in a realistic industry setting.
  arXiv  Detail & Related papers  (2024-03-14T09:45:05Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasets [46.12592636378064]
  We show that there is still a long way to go before solving text-to-generation in the medical domain. We evaluate state-of-the-art language models showing substantial drops in performance with accuracy dropping from up to 92% to 28%. We introduce a novel data augmentation approach to improve the generalizability of relational language models.
  arXiv  Detail & Related papers  (2023-03-22T20:26:30Z)
• Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness [115.66421993459663]
  Recent studies reveal that text-to- models are vulnerable to task-specific perturbations. We propose a comprehensive robustness benchmark based on Spider to diagnose the model. We conduct a diagnostic study of the state-of-the-art models on the set.
  arXiv  Detail & Related papers  (2023-01-21T03:57:18Z)
• EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records [36.213730355895805]
  The utterances were collected from 222 hospital staff members, including physicians, nurses, and insurance review and health records teams. We manually linked these questions to two open-source EHR databases, MIMIC-III and eICU, and included various time expressions and held-out unanswerable questions in the dataset.
  arXiv  Detail & Related papers  (2023-01-16T05:10:20Z)
• A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions [102.8606542189429]
  The goal of text-to-corpora parsing is to convert a natural language (NL) question to its corresponding structured query language () based on the evidences provided by databases. Deep neural networks have significantly advanced this task by neural generation models, which automatically learn a mapping function from an input NL question to an output query.
  arXiv  Detail & Related papers  (2022-08-29T14:24:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.