Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography

• URL: http://arxiv.org/abs/2209.01676v1
• Date: Sun, 4 Sep 2022 19:08:44 GMT
• Title: Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography
• Authors: Thomas Z. Li, Kaiwen Xu, Riqiang Gao, Yucheng Tang, Thomas A. Lasko, Fabien Maldonado, Kim Sandler, Bennett A. Landman
• Abstract summary: longitudinal medical imaging presents the unique challenge of sparse, irregular time intervals in data acquisition. We propose two interpretations of a time-distance vision transformer (ViT) by using vector embeddings of continuous time and a temporal emphasis model to scale self-attention weights. This work represents the first self-attention-based framework for classifying longitudinal medical images.
• Score: 4.924544172166966
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Features learned from single radiologic images are unable to provide information about whether and how much a lesion may be changing over time. Time-dependent features computed from repeated images can capture those changes and help identify malignant lesions by their temporal behavior. However, longitudinal medical imaging presents the unique challenge of sparse, irregular time intervals in data acquisition. While self-attention has been shown to be a versatile and efficient learning mechanism for time series and natural images, its potential for interpreting temporal distance between sparse, irregularly sampled spatial features has not been explored. In this work, we propose two interpretations of a time-distance vision transformer (ViT) by using (1) vector embeddings of continuous time and (2) a temporal emphasis model to scale self-attention weights. The two algorithms are evaluated based on benign versus malignant lung cancer discrimination of synthetic pulmonary nodules and lung screening computed tomography studies from the National Lung Screening Trial (NLST). Experiments evaluating the time-distance ViTs on synthetic nodules show a fundamental improvement in classifying irregularly sampled longitudinal images when compared to standard ViTs. In cross-validation on screening chest CTs from the NLST, our methods (0.785 and 0.786 AUC respectively) significantly outperform a cross-sectional approach (0.734 AUC) and match the discriminative performance of the leading longitudinal medical imaging algorithm (0.779 AUC) on benign versus malignant classification. This work represents the first self-attention-based framework for classifying longitudinal medical images. Our code is available at https://github.com/tom1193/time-distance-transformer.

Related papers

• st-DTPM: Spatial-Temporal Guided Diffusion Transformer Probabilistic Model for Delayed Scan PET Image Prediction [6.812140684449128]
  Several studies have indicated that dual-time PET imaging holds promise in distinguishing between malignant and benign tumor processes. We propose a novel spatial-temporal guided diffusion transformer probabilistic model (st-DTPM) to solve dual-time PET imaging prediction problem.
  arXiv  Detail & Related papers  (2024-10-30T06:37:55Z)
• Longitudinal Segmentation of MS Lesions via Temporal Difference Weighting [2.0168790328644697]
  We introduce a novel approach that explicitly incorporates temporal differences between baseline and follow-up scans through a unique architectural inductive bias called Difference Weighting Block. We achieve superior scores in lesion segmentation as well as lesion detection as compared to state-of-the-art longitudinal and single timepoint models across two datasets.
  arXiv  Detail & Related papers  (2024-09-20T11:30:54Z)
• Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling [49.52787013516891]
  Our proposed Longitudinal Transformer for Survival Analysis (LTSA) enables dynamic disease prognosis from longitudinal medical imaging. A temporal attention analysis also suggested that, while the most recent image is typically the most influential, prior imaging still provides additional prognostic value.
  arXiv  Detail & Related papers  (2024-05-14T17:15:28Z)
• Swin-Tempo: Temporal-Aware Lung Nodule Detection in CT Scans as Video Sequences Using Swin Transformer-Enhanced UNet [2.7547288571938795]
  We present an innovative model that harnesses the strengths of both convolutional neural networks and vision transformers. Inspired by object detection in videos, we treat each 3D CT image as a video, individual slices as frames, and lung nodules as objects, enabling a time-series application.
  arXiv  Detail & Related papers  (2023-10-05T07:48:55Z)
• Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures From Routine EHRs for Pulmonary Nodule Classification [4.002181247287472]
  We propose a transformer-based multimodal strategy to integrate repeat imaging with longitudinal clinical signatures from EHRs for solitary pulmonary nodule (SPN) classification. We perform unsupervised disentanglement of latent clinical signatures and leverage time-distance scaled self-attention to jointly learn from clinical signatures expressions and chest computed tomography (CT) scans.
  arXiv  Detail & Related papers  (2023-04-06T03:03:07Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)
• Stain-invariant self supervised learning for histopathology image analysis [74.98663573628743]
  We present a self-supervised algorithm for several classification tasks within hematoxylin and eosin stained images of breast cancer. Our method achieves the state-of-the-art performance on several publicly available breast cancer datasets.
  arXiv  Detail & Related papers  (2022-11-14T18:16:36Z)
• Harmonizing Pathological and Normal Pixels for Pseudo-healthy Synthesis [68.5287824124996]
  We present a new type of discriminator, the segmentor, to accurately locate the lesions and improve the visual quality of pseudo-healthy images. We apply the generated images into medical image enhancement and utilize the enhanced results to cope with the low contrast problem. Comprehensive experiments on the T2 modality of BraTS demonstrate that the proposed method substantially outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-29T08:41:17Z)
• Single volume lung biomechanics from chest computed tomography using a mode preserving generative adversarial network [10.406580531987418]
  We propose a generative adversarial learning approach for estimating local tissue expansion directly from a single CT scan. The proposed framework was trained and evaluated on 2500 subjects from the SPIROMICS cohort. Our model achieved an overall PSNR of 18.95 decibels, SSIM of 0.840, and Spearman's correlation of 0.61 at a high spatial resolution of 1 mm3.
  arXiv  Detail & Related papers  (2021-10-15T06:17:52Z)
• CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation [56.622832383316215]
  We propose a novel approach to translate unpaired contrast computed tomography (CT) scans to non-contrast CT scans. Our approach is based on cycle-consistent generative adversarial convolutional transformers, for short, CyTran. Our empirical results show that CyTran outperforms all competing methods.
  arXiv  Detail & Related papers  (2021-10-12T23:25:03Z)
• M3Lung-Sys: A Deep Learning System for Multi-Class Lung Pneumonia Screening from CT Imaging [85.00066186644466]
  We propose a Multi-task Multi-slice Deep Learning System (M3Lung-Sys) for multi-class lung pneumonia screening from CT imaging. In addition to distinguish COVID-19 from Healthy, H1N1, and CAP cases, our M 3 Lung-Sys also be able to locate the areas of relevant lesions.
  arXiv  Detail & Related papers  (2020-10-07T06:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.