Evaluating Entity Retrieval in Electronic Health Records: a Semantic Gap Perspective

• URL: http://arxiv.org/abs/2502.06252v1
• Date: Mon, 10 Feb 2025 08:33:47 GMT
• Title: Evaluating Entity Retrieval in Electronic Health Records: a Semantic Gap Perspective
• Authors: Zhengyun Zhao, Hongyi Yuan, Jingjing Liu, Haichao Chen, Huaiyuan Ying, Songchi Zhou, Sheng Yu,
• Abstract summary: We propose the development and release of a novel benchmark for evaluating entity retrieval in EHRs. We use discharge summaries from the MIMIC-III dataset, generate 1,246 queries, and provide over 77,000 relevance annotations. To offer the first assessment of the semantic gap, we introduce a novel classification system for relevance matches.
• Score: 11.786980537459405
• License:
• Abstract: Entity retrieval plays a crucial role in the utilization of Electronic Health Records (EHRs) and is applied across a wide range of clinical practices. However, a comprehensive evaluation of this task is lacking due to the absence of a public benchmark. In this paper, we propose the development and release of a novel benchmark for evaluating entity retrieval in EHRs, with a particular focus on the semantic gap issue. Using discharge summaries from the MIMIC-III dataset, we incorporate ICD codes and prescription labels associated with the notes as queries, and annotate relevance judgments using GPT-4. In total, we use 1,000 patient notes, generate 1,246 queries, and provide over 77,000 relevance annotations. To offer the first assessment of the semantic gap, we introduce a novel classification system for relevance matches. Leveraging GPT-4, we categorize each relevant pair into one of five categories: string, synonym, abbreviation, hyponym, and implication. Using the proposed benchmark, we evaluate several retrieval methods, including BM25, query expansion, and state-of-the-art dense retrievers. Our findings show that BM25 provides a strong baseline but struggles with semantic matches. Query expansion significantly improves performance, though it slightly reduces string match capabilities. Dense retrievers outperform traditional methods, particularly for semantic matches, and general-domain dense retrievers often surpass those trained specifically in the biomedical domain.

Related papers

• ECLIPSE: Contrastive Dimension Importance Estimation with Pseudo-Irrelevance Feedback for Dense Retrieval [14.72046677914345]
  Recent advances in Information Retrieval have leveraged high-dimensional embedding spaces to improve the retrieval of relevant documents. Despite these high-dimensional representations, documents relevant to a query reside on a lower-dimensional, query-dependent manifold. We propose a novel methodology that addresses these limitations by leveraging information from both relevant and non-relevant documents.
  arXiv  Detail & Related papers  (2024-12-19T15:45:06Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  Many complex real-world queries require in-depth reasoning to identify relevant documents. We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy [66.95501113584541]
  Utility and topical relevance are critical measures in information retrieval. We propose an Iterative utiliTy judgmEnt fraMework to promote each step of the cycle of Retrieval-Augmented Generation.
  arXiv  Detail & Related papers  (2024-06-17T07:52:42Z)
• Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy [52.426623750562335]
  We introduce ToTER (Topical taxonomy Enhanced Retrieval) framework. ToTER identifies the central topics of queries and documents with the guidance of the taxonomy, and exploits their topical relatedness to supplement missing contexts. As a plug-and-play framework, ToTER can be flexibly employed to enhance various PLM-based retrievers.
  arXiv  Detail & Related papers  (2024-03-07T02:34:54Z)
• Augmented Embeddings for Custom Retrievals [13.773007276544913]
  We introduce Adapted Dense Retrieval, a mechanism to transform embeddings to enable improved task-specific, heterogeneous and strict retrieval. Dense Retrieval works by learning a low-rank residual adaptation of the pretrained black-box embedding.
  arXiv  Detail & Related papers  (2023-10-09T03:29:35Z)
• Parametric Classification for Generalized Category Discovery: A Baseline Study [70.73212959385387]
  Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples. We investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem. We propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers.
  arXiv  Detail & Related papers  (2022-11-21T18:47:11Z)
• Query Expansion Using Contextual Clue Sampling with Language Models [69.51976926838232]
  We propose a combination of an effective filtering strategy and fusion of the retrieved documents based on the generation probability of each context. Our lexical matching based approach achieves a similar top-5/top-20 retrieval accuracy and higher top-100 accuracy compared with the well-established dense retrieval model DPR. For end-to-end QA, the reader model also benefits from our method and achieves the highest Exact-Match score against several competitive baselines.
  arXiv  Detail & Related papers  (2022-10-13T15:18:04Z)
• Mirror Matching: Document Matching Approach in Seed-driven Document Ranking for Medical Systematic Reviews [31.3220495275256]
  Document ranking is an approach for assisting researchers by providing document rankings where relevant documents are ranked higher than irrelevant ones. We propose a document matching measure named Mirror Matching, which calculates matching scores between medical abstract texts by incorporating common writing patterns.
  arXiv  Detail & Related papers  (2021-12-28T22:27:52Z)
• Impact of detecting clinical trial elements in exploration of COVID-19 literature [29.027162080682643]
  We compare the results retrieved by a standard search engine with those filtered using clinically-relevant concepts and their relations. We find that the relational concept selection filters the original retrieved collection in a way that decreases the proportion of unjudged documents.
  arXiv  Detail & Related papers  (2021-05-25T23:41:24Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Exemplar Auditing for Multi-Label Biomedical Text Classification [0.4873362301533824]
  We generalize a recently proposed zero-shot sequence labeling method, "supervised labeling via a convolutional decomposition" The approach yields classification with "introspection", relating the fine-grained features of an inference-time prediction to their nearest neighbors. Our proposed approach yields both a competitively effective classification model and an interrogation mechanism to aid healthcare workers in understanding the salient features that drive the model's predictions.
  arXiv  Detail & Related papers  (2020-04-07T02:54:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.