Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

• URL: http://arxiv.org/abs/2606.03879v1
• Date: Tue, 02 Jun 2026 16:46:42 GMT
• Title: Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs
• Abstract summary: We retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline.<n>We decompose each encoder's contribution into two axes, Capacity and Necessity.<n>Per-encoder pre-projector effective rank explains the residual score variation.
• Score: 50.3757247538081
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large vision-language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline (~20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder's contribution into two axes, Capacity, the score an encoder reaches on its own, and Necessity, the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank expands under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder-projector interface. Together, the Capacity-Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it.

Related papers

• LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics [50.45297537821907]
  We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective.<n>A single encoder is trained with an invariance loss over global and local views of a clip.<n>LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute.
  arXiv  Detail & Related papers  (2026-08-27T17:26:24Z)
• Scope: Selective Cross-modal Orchestration of Visual Perception Experts [51.29218658993354]
  SCOPE is a Mixture-of-Encoders framework that dynamically selects one specialized encoder per image-text pair via instance-level routing.<n>A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders.<n>Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49%.
  arXiv  Detail & Related papers  (2025-10-14T20:33:01Z)
• Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models [37.59115132356727]
  We propose aligning pretrained visual encoders to serve as tokenizers for latent diffusion models in image generation.<n>On ImageNet 256$times$256, our tokenizer accelerates the convergence of diffusion models, reaching a gFID of 1.90 within just 64 epochs.<n>Our method is simple, scalable, and establishes a semantically grounded paradigm for continuous tokenizer design.
  arXiv  Detail & Related papers  (2025-09-29T17:57:39Z)
• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models [92.37117312251755]
  We propose a progressive pruning framework, namely Multi-Encoder collaboraTivE tOken pRuning (METEOR)<n>For multi-vision encoding, we discard redundant tokens within each encoder via a rank guided collaborative token assignment strategy.<n>For multi-vision fusion, we combine the visual features from different encoders while reducing cross-encoder redundancy with cooperative pruning.
  arXiv  Detail & Related papers  (2025-07-28T13:50:53Z)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders [54.03921875874111]
  We show that performance degrades gracefully when selected encoders are masked, revealing pervasive encoder redundancy.<n>We observe strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater than 90%.<n>Masking specific encoders can yield up to 16% higher accuracy on a specific task category and 3.6% overall performance boost compared to the full model.
  arXiv  Detail & Related papers  (2025-07-04T02:38:59Z)
• Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction [0.7237068561453082]
  We propose DeCon, an efficient encoder-decoder self-supervised learning (SSL) framework that supports joint contrastive pre-training.<n>By adapting an established contrastive SSL framework for dense prediction tasks, DeCon achieves new state-of-the-art results.<n>Our results demonstrate that joint pre-training enhances the representation power of the encoder and improves performance in dense prediction tasks.
  arXiv  Detail & Related papers  (2025-03-21T20:19:13Z)
• DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models [22.276574156358084]
  We build a multi-exit encoder-decoder transformer model which is trained with deep supervision so that each of its decoder layers is capable of generating plausible predictions. We show our approach can reduce overall inference latency by 30%-60% with comparable or even higher accuracy compared to baselines.
  arXiv  Detail & Related papers  (2023-11-15T01:01:02Z)
• Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised Learning [18.10704604275133]
  Masked Autoencoders (MAE) have demonstrated promising performance in self-supervised learning for 2D and 3D computer vision. We propose Point Regress AutoEncoder (Point-RAE), a new scheme for regressive autoencoders for point cloud self-supervised learning. Our approach is efficient during pre-training and generalizes well on various downstream tasks.
  arXiv  Detail & Related papers  (2023-09-25T17:23:33Z)
• Think Twice before Driving: Towards Scalable Decoders for End-to-End Autonomous Driving [74.28510044056706]
  Existing methods usually adopt the decoupled encoder-decoder paradigm. In this work, we aim to alleviate the problem by two principles. We first predict a coarse-grained future position and action based on the encoder features. Then, conditioned on the position and action, the future scene is imagined to check the ramification if we drive accordingly.
  arXiv  Detail & Related papers  (2023-05-10T15:22:02Z)
• MUSTER: A Multi-scale Transformer-based Decoder for Semantic Segmentation [19.83103856355554]
  MUSTER is a transformer-based decoder that seamlessly integrates with hierarchical encoders. MSKA units enable the fusion of multi-scale features from the encoder and decoder, facilitating comprehensive information integration. On the challenging ADE20K dataset, our best model achieves a single-scale mIoU of 50.23 and a multi-scale mIoU of 51.88.
  arXiv  Detail & Related papers  (2022-11-25T06:51:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.