Scope: Selective Cross-modal Orchestration of Visual Perception Experts

• URL: http://arxiv.org/abs/2510.12974v2
• Date: Fri, 17 Oct 2025 03:30:31 GMT
• Title: Scope: Selective Cross-modal Orchestration of Visual Perception Experts
• Authors: Tianyu Zhang, Suyuchen Wang, Chao Wang, Juan Rodriguez, Ahmed Masry, Xiangru Jian, Yoshua Bengio, Perouz Taslakian,
• Abstract summary: SCOPE is a Mixture-of-Encoders framework that dynamically selects one specialized encoder per image-text pair via instance-level routing.<n>A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders.<n>Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49%.
• Score: 51.29218658993354
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49\%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs.

Related papers

• METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Models [92.37117312251755]
  We propose a progressive pruning framework, namely Multi-Encoder collaboraTivE tOken pRuning (METEOR)<n>For multi-vision encoding, we discard redundant tokens within each encoder via a rank guided collaborative token assignment strategy.<n>For multi-vision fusion, we combine the visual features from different encoders while reducing cross-encoder redundancy with cooperative pruning.
  arXiv  Detail & Related papers  (2025-07-28T13:50:53Z)
• Efficient Encoder-Decoder Transformer Decoding for Decomposable Tasks [53.550782959908524]
  We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and decomposable tasks. Our method, prompt-in-decoder (PiD), encodes the input once and decodes the output in parallel, boosting both training and inference efficiency.
  arXiv  Detail & Related papers  (2024-03-19T19:27:23Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Reducing Redundancy in the Bottleneck Representation of the Autoencoders [98.78384185493624]
  Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks. We propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST.
  arXiv  Detail & Related papers  (2022-02-09T18:48:02Z)
• Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network [99.03895740754402]
  We propose a two-stream decoupled design of encoder-decoder structure, in which two decoupled cross-modal encoder and decoder are involved. As an alternative, we propose a primary scheduled sampling strategy that mitigates such discrepancy via pretraining encoder-decoder in a two-pass manner.
  arXiv  Detail & Related papers  (2021-01-27T17:36:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.