TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography

• URL: http://arxiv.org/abs/2608.18166v1
• Date: Sun, 16 Aug 2026 16:14:48 GMT
• Title: TractoGraphVLM: A Unified Vision-Language Framework for White Matter Tractography
• Abstract summary: TractoGraphVLM is a unified framework for four tasks, bundle classification, text-to-tract, anatomical, and visual question answering.<n>GPS transformer produces bundle embeddings aligned with a frozen BiomedBERT text encoder via contrastive learning.<n>BioGPT decoder with visual captions generates tokens and answers.
• Score: 0.30586855806896046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision language models have transformed 2D medical imaging, yet extending them to 3D white matter tractography remains challenging due to the complex topology of fiber bundles. We introduce TractoGraphVLM, a unified framework for four tasks, bundle classification, text-to-tract retrieval, anatomical captioning, and visual question answering, built on a shared GPS architecture, training procedure, and read-out design. Fiber bundles are represented as streamline graphs whose nodes encode 3D position and tangent orientation. A General, Powerful, Scalable (GPS) graph transformer produces bundle embeddings aligned with a frozen BiomedBERT text encoder via contrastive learning, while a BioGPT decoder with visual prefix tokens generates captions and answers. A single shared encoder and decoder is trained jointly across all four tasks and evaluated from one checkpoint. Trained on HCP Young Adult subjects, TractoGraphVLM achieves 91.8% bundle classification accuracy, 84.7% retrieval R@1, BLEU-4=20.1, ROUGE-L=66.8, and 66.4% VQA accuracy on a held-out test set. The same checkpoints transfer zero-shot to HCP Aging subjects, with a modest drop on discriminative tasks and a larger drop on generative tasks, showing robustness to age and acquisition shift. Language supervision yields richer representations than label-only training, recovering structure like hemisphere and fiber family, carried by captions but never given as a label. Swapping only the visual encoder, graphs preserving fiber orientation outperform volumetric baselines, with GPS giving the best balance. Generative metrics measure consistency with a structured knowledge base rather than independent clinical text; even so, TractoGraphVLM shows that classifying, retrieving, describing, and answering questions about a white matter bundle can be served by one jointly trained model that learns transferable neuroanatomy from language alone.

Related papers

• Tractogram foundation model [46.43770468662999]
  We introduce TractFM, a tractogram foundation model that learns reusable representations directly from whole-brain streamline sets.<n> TractFM combines a local streamline encoder with a permutation-equivariant tractogram encoder, allowing all streamlines from a subject to be contextualized jointly.<n>Its frozen representations achieve accurate tract parcellation and predict age and sex across independent datasets.
  arXiv  Detail & Related papers  (2026-06-03T19:45:23Z)
• GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations [1.2679579476597642]
  We present GLINT (Gated Language-Image alignmeNT), a framework that explicitly models this sparse correspondence.<n> GLINT enables zero-shot classification, grounding, and segmentation from free-text queries.<n>It is the first to demonstrate zero-shot segmentation on 3D CT volumes without mask supervision.
  arXiv  Detail & Related papers  (2026-06-02T05:40:51Z)
• Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression [0.0]
  We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes.<n>A phased training curriculum adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder.<n>Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision.
  arXiv  Detail & Related papers  (2026-03-24T15:13:30Z)
• A Graph-Based Framework for Interpretable Whole Slide Image Analysis [86.37618055724441]
  We develop a framework that transforms whole-slide images into biologically-informed graph representations.<n>Our approach builds graph nodes from tissue regions that respect natural structures, not arbitrary grids.<n>We demonstrate strong performance on challenging cancer staging and survival prediction tasks.
  arXiv  Detail & Related papers  (2025-03-14T20:15:04Z)
• A self-supervised framework for learning whole slide representations [52.774822784847565]
  We present Slide Pre-trained Transformers (SPT) for gigapixel-scale self-supervision of whole slide images. We benchmark SPT visual representations on five diagnostic tasks across three biomedical microscopy datasets.
  arXiv  Detail & Related papers  (2024-02-09T05:05:28Z)
• G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training [10.786853837756686]
  We propose a new medical vision-language pre-training framework, named textbfGlobal to textbfDense level representation learning (G2D) G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation.
  arXiv  Detail & Related papers  (2023-12-03T22:44:04Z)
• Transforming Visual Scene Graphs to Image Captions [69.13204024990672]
  We propose to transform Scene Graphs (TSG) into more descriptive captions. In TSG, we apply multi-head attention (MHA) to design the Graph Neural Network (GNN) for embedding scene graphs. In TSG, each expert is built on MHA, for discriminating the graph embeddings to generate different kinds of words.
  arXiv  Detail & Related papers  (2023-05-03T15:18:37Z)
• Triple-View Feature Learning for Medical Image Segmentation [9.992387025633805]
  TriSegNet is a semi-supervised semantic segmentation framework. It uses triple-view feature learning on a limited amount of labelled data and a large amount of unlabeled data.
  arXiv  Detail & Related papers  (2022-08-12T14:41:40Z)
• Two-Stream Graph Convolutional Network for Intra-oral Scanner Image Segmentation [133.02190910009384]
  We propose a two-stream graph convolutional network (i.e., TSGCN) to handle inter-view confusion between different raw attributes. Our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation.
  arXiv  Detail & Related papers  (2022-04-19T10:41:09Z)
• Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion [53.31911669146451]
  Human-curated knowledge graphs provide critical supportive information to various natural language processing tasks. These graphs are usually incomplete, urging auto-completion of them. graph embedding approaches, e.g., TransE, learn structured knowledge via representing graph elements into dense embeddings. textual encoding approaches, e.g., KG-BERT, resort to graph triple's text and triple-level contextualized representations.
  arXiv  Detail & Related papers  (2020-04-30T13:50:34Z)
• Structured Landmark Detection via Topology-Adapting Deep Graph Learning [75.20602712947016]
  We present a new topology-adapting deep graph learning approach for accurate anatomical facial and medical landmark detection. The proposed method constructs graph signals leveraging both local image features and global shape features. Experiments are conducted on three public facial image datasets (WFLW, 300W, and COFW-68) as well as three real-world X-ray medical datasets (Cephalometric (public), Hand and Pelvis)
  arXiv  Detail & Related papers  (2020-04-17T11:55:03Z)
• 3D medical image segmentation with labeled and unlabeled data using autoencoders at the example of liver segmentation in CT images [58.720142291102135]
  This work investigates the potential of autoencoder-extracted features to improve segmentation with a convolutional neural network. A convolutional autoencoder was used to extract features from unlabeled data and a multi-scale, fully convolutional CNN was used to perform the target task of 3D liver segmentation in CT images.
  arXiv  Detail & Related papers  (2020-03-17T20:20:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.