PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization

• URL: http://arxiv.org/abs/2606.03444v1
• Date: Tue, 02 Jun 2026 10:28:32 GMT
• Title: PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization
• Abstract summary: We introduce textbfPRISM, a novel dual-stream Mixture-of-Experts (MoE) framework that synergizes VFMs via modular specialization.<n>Experiments on PASCAL-Context and NYUD-v2 show that textbfPRISM establishes a new state of the art, validating that sparse, emergent specialization is a scalable approach for integrating diverse visual knowledge.
• Score: 25.22639372242625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unifying the complementary strengths of diverse Vision Foundation Models (VFMs) into a single efficient model is highly desirable but challenged by the negative transfer inherent in monolithic distillation. To address these feature conflicts, we introduce \textbf{PRISM}, a novel dual-stream Mixture-of-Experts (MoE) framework that synergizes VFMs via modular specialization. We propose a two-stage paradigm: (1) expertise deconstruction, where a teacher-conditional router guides experts to specialize in distinct representational subspaces to mitigate interference, followed by (2) dynamic recomposition, where the router learns to assemble these experts into tailored computational pathways for downstream tasks. Experiments on PASCAL-Context and NYUD-v2 show that \textbf{PRISM} establishes a new state of the art, validating that sparse, emergent specialization is a scalable approach for integrating diverse visual knowledge.

Related papers

• Rethinking Expert Training for Model Merging with Prompt Learning [29.003417422329715]
  Model merging aims to combine multiple domain-specialized experts trained from a shared foundation model into a single multi-task model.<n>Existing approaches largely focus on improving the merging procedure itself.<n>We introduce Dual-Tuned Experts (DTEs), a two-stage training strategy that first learns prompts and then fine-tunes the vision encoder.
  arXiv  Detail & Related papers  (2026-07-27T14:04:48Z)
• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• MME: Mixture of Mesh Experts with Random Walk Transformer Gating [13.564417897372875]
  We present a novel Mixture of Experts (MoE) framework designed to harness the complementary strengths of diverse approaches.<n>We propose a new gate architecture that encourages each expert to specialise in the classes it excels in.<n>Our framework achieves state-of-the-art results in mesh classification, retrieval, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2026-02-28T22:13:00Z)
• $n$-Musketeers: Reinforcement Learning Shapes Collaboration Among Language Models [17.293403543585413]
  We introduce soft hidden-state collaboration, where multiple frozen SLM experts are integrated via a trainable attention interface.<n>Experiments on Reasoning Gym and GSM8K show that this latent integration is competitive with strong single-model RLVR baselines.<n>Overall, hidden-state collaboration provides a compact mechanism for leveraging frozen experts, while offering an observational window into expert utilization patterns and their evolution under RLVR.
  arXiv  Detail & Related papers  (2026-02-09T20:27:52Z)
• DRFormer: A Dual-Regularized Bidirectional Transformer for Person Re-identification [37.61695934257133]
  Fine-grained discriminative details and global semantic features can contribute to solving person re-identification challenges.<n>Vision foundation models excel at mining local textures, and vision-language models capture strong global semantic difference.<n>We propose a framework to synergize their strengths by a textbfDual-textbfRegularized Bidirectional textbfTransformer.
  arXiv  Detail & Related papers  (2026-02-01T06:59:53Z)
• Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance [79.21541758879012]
  Mixture-of-Experts (MoE) has emerged as a powerful paradigm for scaling model capacity while preserving computational efficiency.<n>We present ProMoE, an MoE framework featuring a two-step router with explicit routing guidance that promotes expert specialization.
  arXiv  Detail & Related papers  (2025-10-28T17:59:02Z)
• UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception [54.53657134205492]
  UniAlignment is a unified multimodal generation framework within a single diffusion transformer.<n>It incorporates both intrinsic-modal semantic alignment and cross-modal semantic alignment, thereby enhancing the model's cross-modal consistency and instruction-following robustness.<n>We present SemGen-Bench, a new benchmark specifically designed to evaluate multimodal semantic consistency under complex textual instructions.
  arXiv  Detail & Related papers  (2025-09-28T09:11:30Z)
• A Tale of Two Experts: Cooperative Learning for Source-Free Unsupervised Domain Adaptation [59.88864205383671]
  Source-Free Unsupervised Domain Adaptation (SFUDA) addresses the realistic challenge of adapting a source-trained model to a target domain without access to the source data.<n>Existing SFUDA methods either exploit only the source model's predictions or fine-tune large multimodal models.<n>We propose the Experts Cooperative Learning (EXCL) to exploit complementary insights and the latent structure of target data.
  arXiv  Detail & Related papers  (2025-09-26T11:39:50Z)
• Astrea: A MOE-based Visual Understanding Model with Progressive Alignment [10.943104653307294]
  Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures have emerged as a pivotal paradigm in multimodal understanding.<n>We propose Astrea, a novel multi-expert collaborative VLM architecture based on progressive pre-alignment.
  arXiv  Detail & Related papers  (2025-03-12T14:44:52Z)
• Robust Training of Federated Models with Extremely Label Deficiency [84.00832527512148]
  Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. We propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings.
  arXiv  Detail & Related papers  (2024-02-22T10:19:34Z)
• HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization [69.33162366130887]
  Domain Generalization (DG) endeavors to create machine learning models that excel in unseen scenarios by learning invariant features. We introduce a novel method designed to supplement the model with domain-level and task-specific characteristics. This approach aims to guide the model in more effectively separating invariant features from specific characteristics, thereby boosting the generalization.
  arXiv  Detail & Related papers  (2024-01-18T04:23:21Z)
• Universal Information Extraction as Unified Semantic Matching [54.19974454019611]
  We decouple information extraction into two abilities, structuring and conceptualizing, which are shared by different tasks and schemas. Based on this paradigm, we propose to universally model various IE tasks with Unified Semantic Matching framework. In this way, USM can jointly encode schema and input text, uniformly extract substructures in parallel, and controllably decode target structures on demand.
  arXiv  Detail & Related papers  (2023-01-09T11:51:31Z)
• Diversified Dynamic Routing for Vision Tasks [36.199659460868496]
  We propose a novel architecture where each layer is composed of a set of experts. In our method, the model is explicitly trained to solve the challenge of finding relevant partitioning of the data. We conduct several experiments on semantic segmentation on Cityscapes and object detection and instance segmentation on MS-COCO.
  arXiv  Detail & Related papers  (2022-09-26T23:27:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.