Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays

• URL: http://arxiv.org/abs/2606.20852v1
• Date: Thu, 18 Jun 2026 18:36:13 GMT
• Title: Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays
• Abstract summary: Contrastive Activation Addition (CAA) can improve pneumonia classification in chest radiograph VLMs without updating model weights.<n>Three frozen chest radiograph VLMs were evaluated on the public Kermany pneumonia test set.<n>CAA substantially altered prediction score distributions and operating characteristics without fine-tuning.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference-time engineering can alter model behavior without fine-tuning. However, its utility for improving diagnostic performance in medical vision-language models (VLMs) remains unclear. We aim to evaluate whether Contrastive Activation Addition (CAA) can improve pneumonia classification in chest radiograph VLMs without updating model weights. Three frozen chest radiograph VLMs (MedGemma-4B-IT, NV-Reason-CXR-3B, and CheXOne-3B) were evaluated on the public Kermany pneumonia test set. Classification was based on the logits of the tokens Yes and No under a binary prompt. Steering vectors included a 30-pair answer-bias control, a 30-pair pneumonia text contrast, and an image-conditioned contrast derived from 30 pneumonia and 30 normal development images. A deterministic 200-image development set was used for layer and scale selection (100 images) and threshold calibration (100 images). Performance was assessed using ROC-AUC, PR-AUC, F1 score, threshold analyses, reverse-vector controls, random-vector controls, and conditional bootstrap confidence intervals. Fixed-threshold F1 improvements were frequently observed but did not consistently indicate improved diagnostic performance. For MedGemma-4B-IT. NV-Reason-CXR-3B showed the strongest benefit: calibrated F1 improved from 0.7692 in the zero-shot setting to 0.8619 with pneumonia-text steering and to 0.8727 with image-conditioned steering. For CheXOne-3B, pneumonia-text steering increased calibrated F1 from 0.8528 to 0.8666, although the confidence interval crossed zero. On this public pneumonia benchmark, CAA substantially altered prediction score distributions and operating characteristics without fine-tuning. Meaningful performance gains were observed in one of three evaluated VLMs, suggesting that activation steering may serve as a lightweight approach for adapting medical VLM behavior.

Related papers

• A Leakage-Aware Comparative Benchmark of Machine Learning, Deep Learning, and Transformer Models for Reliable Leukemia Detection [0.0]
  Automated classification of acute lymphoblastic leukemia (ALL) from peripheral blood smear images has often reported near-perfect performance on the C-NMC 2019 dataset.<n>We show that such results can be inflated by patient-level data leakage caused by random image-level partitioning.<n>We establish a leakage-aware benchmark under a strict subject-disjoint protocol.
  arXiv  Detail & Related papers  (2026-06-23T04:53:49Z)
• Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering [94.37535002230504]
  We develop a training-free, inference-time control framework termed Semantically Decoupled Latent Steering.<n>Our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition.<n>We show that our approach significantly reduces the probability of historical hallucinations.
  arXiv  Detail & Related papers  (2026-02-27T04:49:01Z)
• Explainable Deep Learning for Pediatric Pneumonia Detection in Chest X-Ray Images [0.0]
  Pneumonia remains a leading cause of morbidity and mortality among children worldwide.<n>This study compares two state-of-the-art convolutional neural network (CNN) architectures for automated pediatric pneumonia detection.
  arXiv  Detail & Related papers  (2026-01-14T19:21:32Z)
• Weakly Supervised Pneumonia Localization from Chest X-Rays Using Deep Neural Network and Grad-CAM Explanations [0.0]
  This study proposes a weakly supervised deep learning framework for pneumonia classification and localization from chest X-rays.<n>Instead of costly pixel-level annotations, our approach utilizes image-level labels to generate clinically meaningful heatmaps.
  arXiv  Detail & Related papers  (2025-11-01T08:44:24Z)
• LightPneumoNet: Lightweight Pneumonia Classifier [0.0]
  This study introduces LightPneumoNet, an efficient, lightweight convolutional neural network (CNN) built from scratch.<n>Our model was trained on a public dataset of 5,856 chest X-ray images.<n>On an independent test set, our model delivered exceptional performance, achieving an overall accuracy of 0.942, precision of 0.92, and an F1-Score of 0.96.
  arXiv  Detail & Related papers  (2025-10-13T10:14:17Z)
• Does Bigger Mean Better? Comparitive Analysis of CNNs and Biomedical Vision Language Modles in Medical Diagnosis [7.41395379449452]
  This paper presents a comparative analysis between a supervised lightweight Convolutional Neural Network (CNN) and a zero-shot medical Vision-Language Model (VLM)<n>Our experiments show that supervised CNNs serve as highly competitive baselines in both cases.<n>By optimizing the classification threshold on a validation set, the performance of BiomedCLIP is significantly boosted across both datasets.
  arXiv  Detail & Related papers  (2025-10-01T01:46:09Z)
• MOZART: Ensembling Approach for COVID-19 Detection using Chest X-Ray Imagery [0.0]
  COVID-19, has led to a global pandemic that strained the healthcare systems. Traditional convolutional neural networks (CNNs) achieve impressive accuracy. We introduce the MOZART framework, an ensemble learning approach that enhances the virus detection.
  arXiv  Detail & Related papers  (2024-10-11T21:02:58Z)
• Brain Tumor Classification on MRI in Light of Molecular Markers [56.99710477905796]
  Co-deletion of the 1p/19q gene is associated with clinical outcomes in low-grade gliomas.<n>This study aims to utilize a specially MRI-based convolutional neural network for brain cancer detection.
  arXiv  Detail & Related papers  (2024-09-29T07:04:26Z)
• A Federated Learning Framework for Stenosis Detection [70.27581181445329]
  This study explores the use of Federated Learning (FL) for stenosis detection in coronary angiography images (CA) Two heterogeneous datasets from two institutions were considered: dataset 1 includes 1219 images from 200 patients, which we acquired at the Ospedale Riuniti of Ancona (Italy) dataset 2 includes 7492 sequential images from 90 patients from a previous study available in the literature.
  arXiv  Detail & Related papers  (2023-10-30T11:13:40Z)
• Self-Supervised Pretraining Improves Performance and Inference Efficiency in Multiple Lung Ultrasound Interpretation Tasks [65.23740556896654]
  We investigated whether self-supervised pretraining could produce a neural network feature extractor applicable to multiple classification tasks in lung ultrasound analysis. When fine-tuning on three lung ultrasound tasks, pretrained models resulted in an improvement of the average across-task area under the receiver operating curve (AUC) by 0.032 and 0.061 on local and external test sets respectively.
  arXiv  Detail & Related papers  (2023-09-05T21:36:42Z)
• Attention-based Saliency Maps Improve Interpretability of Pneumothorax Classification [52.77024349608834]
  To investigate chest radiograph (CXR) classification performance of vision transformers (ViT) and interpretability of attention-based saliency. ViTs were fine-tuned for lung disease classification using four public data sets: CheXpert, Chest X-Ray 14, MIMIC CXR, and VinBigData. ViTs had comparable CXR classification AUCs compared with state-of-the-art CNNs.
  arXiv  Detail & Related papers  (2023-03-03T12:05:41Z)
• Predicting COVID-19 Pneumonia Severity on Chest X-ray with Deep Learning [57.00601760750389]
  We present a severity score prediction model for COVID-19 pneumonia for frontal chest X-ray images. Such a tool can gauge severity of COVID-19 lung infections that can be used for escalation or de-escalation of care.
  arXiv  Detail & Related papers  (2020-05-24T23:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.