Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track

• URL: http://arxiv.org/abs/2507.14096v2
• Date: Mon, 21 Jul 2025 18:01:44 GMT
• Title: Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track
• Authors: Brian Ondov, William Xia, Kush Attal, Ishita Unde, Jerry He, Dina Demner-Fushman,
• Abstract summary: We hosted the Plain Language Adaptation of Biomedical Abstracts track at the 2023 and 2024 Text Retrieval Conference.<n>Tasks included complete, sentence-level, rewriting of abstracts and identifying and replacing difficult terms.<n>Twelve teams spanning twelve countries participated in the track, with models from multilayer perceptrons to large pretrained transformers.
• Score: 14.011972819032328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Objective: Recent advances in language models have shown potential to adapt professional-facing biomedical literature to plain language, making it accessible to patients and caregivers. However, their unpredictability, combined with the high potential for harm in this domain, means rigorous evaluation is necessary. Our goals with this track were to stimulate research and to provide high-quality evaluation of the most promising systems. Methods: We hosted the Plain Language Adaptation of Biomedical Abstracts (PLABA) track at the 2023 and 2024 Text Retrieval Conferences. Tasks included complete, sentence-level, rewriting of abstracts (Task 1) as well as identifying and replacing difficult terms (Task 2). For automatic evaluation of Task 1, we developed a four-fold set of professionally-written references. Submissions for both Tasks 1 and 2 were provided extensive manual evaluation from biomedical experts. Results: Twelve teams spanning twelve countries participated in the track, with models from multilayer perceptrons to large pretrained transformers. In manual judgments of Task 1, top-performing models rivaled human levels of factual accuracy and completeness, but not simplicity or brevity. Automatic, reference-based metrics generally did not correlate well with manual judgments. In Task 2, systems struggled with identifying difficult terms and classifying how to replace them. When generating replacements, however, LLM-based systems did well in manually judged accuracy, completeness, and simplicity, though not in brevity. Conclusion: The PLABA track showed promise for using Large Language Models to adapt biomedical literature for the general public, while also highlighting their deficiencies and the need for improved automatic benchmarking tools.

Related papers

• Towards Domain Specification of Embedding Models in Medicine [1.0713888959520208]
  We propose a comprehensive benchmark suite of 51 tasks spanning classification, clustering, pair classification, and retrieval modeled on the Massive Text Embedding Benchmark (MTEB)<n>Our results demonstrate that this combined approach establishes a robust evaluation framework and yields embeddings that consistently outperform state of the art alternatives in different tasks.
  arXiv  Detail & Related papers  (2025-07-25T16:15:00Z)
• Zero-Shot Document-Level Biomedical Relation Extraction via Scenario-based Prompt Design in Two-Stage with LLM [7.808231572590279]
  We propose a novel approach to achieve the same results from unannotated full documents using general large language models (LLMs) with lower hardware and labor costs.<n>Our approach combines two major stages: named entity recognition (NER) and relation extraction (RE)<n>To enhance the effectiveness of prompt, we propose a five-part template structure and a scenario-based prompt design principles.
  arXiv  Detail & Related papers  (2025-05-02T07:33:20Z)
• Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Framework [56.858564736806414]
  This paper proposes a novel framework leveraging large language models (LLMs) to identify interventions in scientific literature.<n>Our approach successfully identified 2,421 interventions from a corpus of 64,177 research articles in the speech-language pathology domain.
  arXiv  Detail & Related papers  (2024-08-20T16:42:23Z)
• Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs [54.223394825528665]
  We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models. We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
  arXiv  Detail & Related papers  (2023-12-21T14:26:57Z)
• Taiyi: A Bilingual Fine-Tuned Large Language Model for Diverse Biomedical Tasks [19.091278630792615]
  Most existing biomedical large language models (LLMs) focus on enhancing performance in monolingual biomedical question answering and conversation tasks. We present Taiyi, a bilingual fine-tuned LLM for diverse biomedical tasks.
  arXiv  Detail & Related papers  (2023-11-20T08:51:30Z)
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)
• Benchmarking large language models for biomedical natural language processing applications and recommendations [22.668383945059762]
  Large Language Models (LLMs) have shown promise in general domains.<n>We compare their zero-shot, few-shot, and fine-tuning performance with traditional fine-tuning of BERT or BART models.<n>We find issues like missing information and hallucinations in LLM outputs.
  arXiv  Detail & Related papers  (2023-05-10T13:40:06Z)
• Readability Controllable Biomedical Document Summarization [17.166794984161964]
  We introduce a new task of readability controllable summarization for biomedical documents. It aims to recognise users' readability demands and generate summaries that better suit their needs.
  arXiv  Detail & Related papers  (2022-10-10T14:03:20Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Self-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer [63.72621204057025]
  Expert-layman text style transfer technologies have the potential to improve communication between scientific communities and the general public. High-quality information produced by experts is often filled with difficult jargon laypeople struggle to understand. This is a particularly notable issue in the medical domain, where layman are often confused by medical text online.
  arXiv  Detail & Related papers  (2021-10-06T17:57:22Z)
• CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark [51.38557174322772]
  We present the first Chinese Biomedical Language Understanding Evaluation benchmark. It is a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification. We report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.
  arXiv  Detail & Related papers  (2021-06-15T12:25:30Z)
• Automated Lay Language Summarization of Biomedical Scientific Reviews [16.01452242066412]
  Health literacy has emerged as a crucial factor in making appropriate health decisions and ensuring treatment outcomes. Medical jargon and the complex structure of professional language in this domain make health information especially hard to interpret. This paper introduces the novel task of automated generation of lay language summaries of biomedical scientific reviews.
  arXiv  Detail & Related papers  (2020-12-23T10:01:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.