Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

• URL: http://arxiv.org/abs/2607.06309v1
• Date: Tue, 07 Jul 2026 14:11:54 GMT
• Title: Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification
• Abstract summary: Multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention.<n>We propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone.<n>Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines.
• Score: 0.8029049649310211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities. However, existing multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention, which can entangle view-specific and shared representations and restrict interaction to limited network depths. To address these limitations, we propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone. The framework reformulates inter-view interaction as structured token-level communication, where dedicated fusion tokens explicitly encode bidirectional information exchange between CC and MLO views via cross-attention, serving as intermediate carriers of cross-view dependencies rather than relying on direct feature fusion. Unlike conventional methods that apply fusion at a single layer, fusion modules are inserted at multiple transformer depths, enabling progressive and repeated interaction across the encoder hierarchy. Fusion tokens are reintegrated into the token sequence and refined by subsequent transformer layers, facilitating hierarchical propagation of complementary information while preserving view-specific structure. Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines. On the VinDr-Mammo BI-RADS classification task, the framework achieves 50.40% F1-score and 0.8090 AUC, including a 0.10 AUC improvement over a dual-view fusion baseline in the binary setting. Ablation studies further validate the effectiveness of token-based fusion and multi-depth interaction design.

Related papers

• CtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image Fusion [51.060328159429154]
  Infrared and visible image fusion generates all-weather perception-capable images by combining complementary modalities.<n>We propose CtrlFuse, a controllable image fusion framework that enables interactive dynamic fusion guided by mask prompts.<n> Experiments demonstrate state-of-the-art results in both fusion controllability and segmentation accuracy, with the adapted task branch even outperforming the original segmentation model.
  arXiv  Detail & Related papers  (2026-01-12T13:36:48Z)
• A Multi-scale Fused Graph Neural Network with Inter-view Contrastive Learning for Spatial Transcriptomics Data Clustering [7.214595408714774]
  stMFG is proposed, a multi-scale interactive fusion graph network that introduces layer-wise cross-view attention to dynamically integrate spatial and gene features after each convolution.<n>It outperforms state-of-the-art methods, achieving up to 14% ARI improvement on certain slices.
  arXiv  Detail & Related papers  (2025-12-18T05:13:55Z)
• Hierarchical Identity Learning for Unsupervised Visible-Infrared Person Re-Identification [81.3063589622217]
  Unsupervised visible-infrared person re-identification (USVI-ReID) aims to learn modality-invariant image features from unlabeled cross-modal person datasets.
  arXiv  Detail & Related papers  (2025-09-15T05:10:43Z)
• DART: Dual Adaptive Refinement Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  Open-Vocabulary Multi-Label Recognition (OV-MLR) aims to identify multiple seen and unseen object categories within an image.<n> Vision-Language Pre-training models offer a strong open-vocabulary foundation, but struggle with fine-grained localization under weak supervision.<n>We propose the Dual Adaptive Refinement Transfer (DART) framework to overcome these limitations.
  arXiv  Detail & Related papers  (2025-08-07T17:22:33Z)
• Unleashing Vision Foundation Models for Coronary Artery Segmentation: Parallel ViT-CNN Encoding and Variational Fusion [12.839049648094893]
  coronary artery segmentation is critical for computeraided diagnosis of coronary artery disease (CAD)<n>We propose a novel framework that leverages the power of vision foundation models (VFMs) through a parallel encoding architecture.<n>The proposed framework significantly outperforms state-of-the-art methods, achieving superior performance in accurate coronary artery segmentation.
  arXiv  Detail & Related papers  (2025-07-17T09:25:00Z)
• Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection [0.0]
  Multi-modal learning has emerged as a crucial research direction.<n>Existing approaches often suffer from insufficient cross-modal interactions and rigid fusion strategies.<n>We propose Co-AttenDWG, co-attention with dimension-wise gating, and expert fusion.<n>We show that Co-AttenDWG achieves state-of-the-art performance and superior cross-modal alignment.
  arXiv  Detail & Related papers  (2025-05-25T07:26:00Z)
• Cross Paradigm Representation and Alignment Transformer for Image Deraining [40.66823807648992]
  We propose a novel Cross Paradigm Representation and Alignment Transformer (CPRAformer)<n>Its core idea is the hierarchical representation and alignment, leveraging the strengths of both paradigms to aid image reconstruction.<n>We use two types of self-attention in the Transformer blocks: sparse prompt channel self-attention (SPC-SA) and spatial pixel refinement self-attention (SPR-SA)
  arXiv  Detail & Related papers  (2025-04-23T06:44:46Z)
• XFMamba: Cross-Fusion Mamba for Multi-View Medical Image Classification [31.897467054280504]
  XFMamba is a pure Mamba-based cross-fusion architecture to address the challenge of multi-view medical image classification.<n>XFMamba introduces a novel two-stage fusion strategy, facilitating the learning of single-view features and their cross-view disparity.<n>Results show that it outperforms existing convolution-based and transformer-based multi-view methods.
  arXiv  Detail & Related papers  (2025-03-04T13:38:58Z)
• Dual-scale Enhanced and Cross-generative Consistency Learning for Semi-supervised Medical Image Segmentation [49.57907601086494]
  Medical image segmentation plays a crucial role in computer-aided diagnosis. We propose a novel Dual-scale Enhanced and Cross-generative consistency learning framework for semi-supervised medical image (DEC-Seg)
  arXiv  Detail & Related papers  (2023-12-26T12:56:31Z)
• Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration [67.23451452670282]
  Misalignments between multi-modality images pose challenges in image fusion. We propose a Cross-modality Multi-scale Progressive Dense Registration scheme. This scheme accomplishes the coarse-to-fine registration exclusively using a one-stage optimization.
  arXiv  Detail & Related papers  (2023-08-22T03:46:24Z)
• Efficient Bilateral Cross-Modality Cluster Matching for Unsupervised Visible-Infrared Person ReID [56.573905143954015]
  We propose a novel bilateral cluster matching-based learning framework to reduce the modality gap by matching cross-modality clusters. Under such a supervisory signal, a Modality-Specific and Modality-Agnostic (MSMA) contrastive learning framework is proposed to align features jointly at a cluster-level. Experiments on the public SYSU-MM01 and RegDB datasets demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2023-05-22T03:27:46Z)
• FER-former: Multi-modal Transformer for Facial Expression Recognition [14.219492977523682]
  A novel multifarious supervision-steering Transformer for Facial Expression Recognition is proposed in this paper. Our approach features multi-granularity embedding integration, hybrid self-attention scheme, and heterogeneous domain-steering supervision. Experiments on popular benchmarks demonstrate the superiority of the proposed FER-former over the existing state-of-the-arts.
  arXiv  Detail & Related papers  (2023-03-23T02:29:53Z)
• Cascaded Human-Object Interaction Recognition [175.60439054047043]
  We introduce a cascade architecture for a multi-stage, coarse-to-fine HOI understanding. At each stage, an instance localization network progressively refines HOI proposals and feeds them into an interaction recognition network. With our carefully-designed human-centric relation features, these two modules work collaboratively towards effective interaction understanding.
  arXiv  Detail & Related papers  (2020-03-09T17:05:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.