MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports

• URL: http://arxiv.org/abs/2605.03103v1
• Date: Mon, 04 May 2026 19:37:21 GMT
• Title: MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• Abstract summary: We present MedStruct-S, a benchmark specifically designed to evaluate tasks under unknown keys and OCR noise.<n>Our results show that encoder-only models achieve the best performance for non-null-value key-conditioned QA.<n>These findings show that the benchmark provides a reliable and practical basis for selecting and comparing models across different semi-structured IE settings.
• Score: 2.289837306672451
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Semi-structured information extraction (IE) from OCR-derived clinical reports is crucial for efficiently reconstructing patients' longitudinal medical histories. In practice, this scenario commonly involves three tasks: (i) field-header (key) discovery, (ii) key-conditioned question answering (QA), and (iii) end-to-end key-value pair extraction. However, existing evaluations often under-model two factors: heterogeneous and incompletely known key representations, and OCR-induced noise. This makes it difficult to assess model robustness in real-world settings. We present MedStruct-S, a benchmark specifically designed to evaluate these tasks under unknown keys and OCR noise. MedStruct-S contains 3,582 annotated real-world clinical report pages. Using MedStruct-S, we benchmark two representative paradigms: encoder-only sequence labeling with post-processing and decoder-only structured generation, covering four encoder-only and five decoder-only models spanning 0.11B to 103B parameters. Our results show that encoder-only models achieve the best performance for non-null-value key-conditioned QA despite being substantially smaller than decoder-only models. When comparing models of similar order of magnitude, encoder-only models still perform better overall. Without controlling for model scale, fine-tuned decoder-only models deliver the strongest overall results. These findings show that the benchmark provides a reliable and practical basis for selecting and comparing models across different semi-structured IE settings.

Related papers

• PathReportEval: A Systematic Benchmark for Pathology Report Generation [5.733136272690028]
  We present a standardized benchmark and evaluation framework for pathology report generation.<n>The framework standardizes preprocessing, feature extraction, training, decoding, and evaluation.<n>A central contribution is the Clinical Report Quality Score (CRQS), a clinically grounded metric for evaluating factual correctness.
  arXiv  Detail & Related papers  (2026-07-20T18:59:47Z)
• Key Coverage Matters: Semi-Structured Extraction of OCR Clinical Reports [5.2232051743738905]
  Clinical reports are often fragmented across healthcare institutions because privacy regulations and data silos limit direct information sharing.<n>We formulate this problem as canonical key-conditioned extractive question answering over OCR-derived clinical reports.<n>Using a 0.2B BERT-based model, experiments on real-world reports from more than 20 hospitals show performance improves monotonically with key coverage.
  arXiv  Detail & Related papers  (2026-05-10T09:29:33Z)
• VesSAM: Efficient Multi-Prompting for Segmenting Complex Vessel [68.24765319399286]
  We present VesSAM, a powerful and efficient framework tailored for 2D vessel segmentation.<n>VesSAM integrates (1) a convolutional adapter to enhance local texture features, (2) a multi-prompt encoder that fuses anatomical prompts, and (3) a lightweight mask decoder to reduce jagged artifacts.<n>VesSAM consistently outperforms state-of-the-art PEFT-based SAM variants by over 10% Dice and 13% IoU.
  arXiv  Detail & Related papers  (2025-11-02T15:47:05Z)
• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Hybrid-Segmentor: A Hybrid Approach to Automated Fine-Grained Crack Segmentation in Civil Infrastructure [52.2025114590481]
  We introduce Hybrid-Segmentor, an encoder-decoder based approach that is capable of extracting both fine-grained local and global crack features. This allows the model to improve its generalization capabilities in distinguish various type of shapes, surfaces and sizes of cracks. The proposed model outperforms existing benchmark models across 5 quantitative metrics (accuracy 0.971, precision 0.804, recall 0.744, F1-score 0.770, and IoU score 0.630), achieving state-of-the-art status.
  arXiv  Detail & Related papers  (2024-09-04T16:47:16Z)
• TF4CTR: Twin Focus Framework for CTR Prediction via Adaptive Sample Differentiation [14.047096669510369]
  This paper introduces a novel CTR prediction framework by integrating the plug-and-play Twin Focus (TF) Loss, Sample Selection Embedding Module (SSEM), and Dynamic Fusion Module (DFM)<n>Experiments on five real-world datasets confirm the effectiveness and compatibility of the framework.
  arXiv  Detail & Related papers  (2024-05-06T05:22:40Z)
• DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics [36.057925881268226]
  DictLLM is an innovative framework designed to improve the modeling of key-value structured data, like medical laboratory reports, for generating medical diagnoses. We carry out experiments using various LLM models on a comprehensive real-world medical laboratory report dataset for automatic diagnosis generation.
  arXiv  Detail & Related papers  (2024-02-18T07:10:02Z)
• AMR Parsing with Causal Hierarchical Attention and Pointers [54.382865897298046]
  We introduce new target forms of AMR parsing and a novel model, CHAP, which is equipped with causal hierarchical attention and the pointer mechanism. Experiments show that our model outperforms baseline models on four out of five benchmarks in the setting of no additional data.
  arXiv  Detail & Related papers  (2023-10-18T13:44:26Z)
• A quality assurance framework for real-time monitoring of deep learning segmentation models in radiotherapy [3.5752677591512487]
  This work uses cardiac substructure segmentation as an example task to establish a quality assurance framework. A benchmark dataset consisting of Computed Tomography (CT) images along with manual cardiac delineations of 241 patients was collected. An image domain shift detector was developed by utilizing a trained Denoising autoencoder (DAE) and two hand-engineered features. A regression model was trained to predict the per-patient segmentation accuracy, measured by Dice similarity coefficient (DSC)
  arXiv  Detail & Related papers  (2023-05-19T14:51:05Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Learning Disentangled Latent Factors from Paired Data in Cross-Modal Retrieval: An Implicit Identifiable VAE Approach [33.61751393224223]
  We deal with the problem of learning the underlying disentangled latent factors that are shared between the paired bi-modal data in cross-modal retrieval. We propose a novel idea of the implicit decoder, which completely removes the ambient data decoding module from a latent variable model. Our model is shown to identify the factors accurately, significantly outperforming conventional encoder-decoder latent variable models.
  arXiv  Detail & Related papers  (2020-12-01T17:47:50Z)
• End-to-End Object Detection with Transformers [88.06357745922716]
  We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss.
  arXiv  Detail & Related papers  (2020-05-26T17:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.