Relation Extraction Capabilities of LLMs on Clinical Text: A Bilingual Evaluation for English and Turkish

• URL: http://arxiv.org/abs/2601.09367v1
• Date: Wed, 14 Jan 2026 10:49:46 GMT
• Title: Relation Extraction Capabilities of LLMs on Clinical Text: A Bilingual Evaluation for English and Turkish
• Authors: Aidana Aidynkyzy, Oğuz Dikenelli, Oylum Alatlı, Şebnem Bora,
• Abstract summary: We present the first comprehensive bilingual evaluation of large language models (LLM) for the clinical Relation Extraction task in both English and Turkish.<n>We assess a diverse set of prompting strategies, including multiple in-context learning (ICL) and Chain-of-Thought (CoT) approaches.<n>Our results show that prompting-based LLM approaches consistently outperform traditional fine-tuned models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The scarcity of annotated datasets for clinical information extraction in non-English languages hinders the evaluation of large language model (LLM)-based methods developed primarily in English. In this study, we present the first comprehensive bilingual evaluation of LLMs for the clinical Relation Extraction (RE) task in both English and Turkish. To facilitate this evaluation, we introduce the first English-Turkish parallel clinical RE dataset, derived and carefully curated from the 2010 i2b2/VA relation classification corpus. We systematically assess a diverse set of prompting strategies, including multiple in-context learning (ICL) and Chain-of-Thought (CoT) approaches, and compare their performance to fine-tuned baselines such as PURE. Furthermore, we propose Relation-Aware Retrieval (RAR), a novel in-context example selection method based on contrastive learning, that is specifically designed to capture both sentence-level and relation-level semantics. Our results show that prompting-based LLM approaches consistently outperform traditional fine-tuned models. Moreover, evaluations for English performed better than their Turkish counterparts across all evaluated LLMs and prompting techniques. Among ICL methods, RAR achieves the highest performance, with Gemini 1.5 Flash reaching a micro-F1 score of 0.906 in English and 0.888 in Turkish. Performance further improves to 0.918 F1 in English when RAR is combined with a structured reasoning prompt using the DeepSeek-V3 model. These findings highlight the importance of high-quality demonstration retrieval and underscore the potential of advanced retrieval and prompting techniques to bridge resource gaps in clinical natural language processing.

Related papers

• Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation Extraction [22.152211131922943]
  We propose HYDRE -- HYbrid Distantly Supervised Relation Extraction framework.<n>It first uses a trained DSRE model to identify the top-k candidate relations for a given test sentence.<n>It then uses a novel dynamic exemplar retrieval strategy that extracts reliable, sentence-level exemplars from training data.
  arXiv  Detail & Related papers  (2025-10-21T06:55:19Z)
• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs [73.27182315028021]
  LANPO is a framework that cleanly separates the roles of feedback: language guides exploration, while numerical rewards drive optimization.<n>Our work provides a robust method for integrating historical experiences into the LLM RL loop, creating more effective and data-efficient learning agents.
  arXiv  Detail & Related papers  (2025-10-18T15:51:19Z)
• Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models [64.54005959758733]
  We introduce code-switching in-context learning (CSICL) as a principled and robust approach for overcoming the translation barrier during inference.<n>We conduct extensive experiments across 4 LLMs, 6 datasets, and 10 languages, spanning both knowledge-intensive and reasoning-oriented domains.<n>Our results demonstrate CSICL consistently outperforms X-ICL baselines, achieving gains of 3.1%p and 1.9%p in both target and unseen languages.
  arXiv  Detail & Related papers  (2025-10-07T08:35:42Z)
• Efficient extraction of medication information from clinical notes: an evaluation in two languages [2.5610226051536578]
  We propose an original transformer-based architecture for the extraction of entities and their relations pertaining to patients' medication regimen.<n>We used this approach to train and evaluate a model on French clinical notes, using a newly annotated corpus from Hopitaux Universitaires de Strasbourg.
  arXiv  Detail & Related papers  (2025-02-05T15:13:08Z)
• ICLEval: Evaluating In-Context Learning Ability of Large Language Models [68.7494310749199]
  In-Context Learning (ICL) is a critical capability of Large Language Models (LLMs) as it empowers them to comprehend and reason across interconnected inputs.<n>Existing evaluation frameworks primarily focus on language abilities and knowledge, often overlooking the assessment of ICL ability.<n>We introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which encompasses two key sub-abilities: exact copying and rule learning.
  arXiv  Detail & Related papers  (2024-06-21T08:06:10Z)
• Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking [1.3716808114696444]
  Large Language Models (LLMs) are becoming crucial across various fields, emphasizing the urgency for high-quality models in underrepresented languages. This study explores the unique challenges faced by low-resource languages, such as data scarcity, model selection, evaluation, and computational limitations.
  arXiv  Detail & Related papers  (2024-05-07T21:58:45Z)
• Adapting Mental Health Prediction Tasks for Cross-lingual Learning via Meta-Training and In-context Learning with Large Language Model [3.3590922002216193]
  We use model-agnostic meta-learning and leveraging large language models (LLMs) to address this gap. We first apply a meta-learning model with self-supervision, which results in improved model initialisation for rapid adaptation and cross-lingual transfer. In parallel, we use LLMs' in-context learning capabilities to assess their performance accuracy across the Swahili mental health prediction tasks.
  arXiv  Detail & Related papers  (2024-04-13T17:11:35Z)
• Using Natural Language Explanations to Improve Robustness of In-context Learning [35.18010811754959]
  Large language models (LLMs) can excel in many tasks via in-context learning (ICL) We investigate whether augmenting ICL with natural language explanations (NLEs) improves the robustness of LLMs on adversarial datasets.
  arXiv  Detail & Related papers  (2023-11-13T18:49:13Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.