Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts

• URL: http://arxiv.org/abs/2505.24541v1
• Date: Fri, 30 May 2025 12:48:07 GMT
• Title: Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts
• Authors: Xin He, Xumeng Han, Longhui Wei, Lingxi Xie, Qi Tian,
• Abstract summary: Multimodal large language models (MLLMs) require nuanced interpretation of complex image information.<n> relying solely on a single vision encoder to handle diverse task domains proves difficult and inevitably leads to conflicts.<n>We introduce Mixpert, an efficient mixture-of-vision-experts architecture that inherits the joint learning advantages from a single vision encoder.
• Score: 104.73983712940816
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) require a nuanced interpretation of complex image information, typically leveraging a vision encoder to perceive various visual scenarios. However, relying solely on a single vision encoder to handle diverse task domains proves difficult and inevitably leads to conflicts. Recent work enhances data perception by directly integrating multiple domain-specific vision encoders, yet this structure adds complexity and limits the potential for joint optimization. In this paper, we introduce Mixpert, an efficient mixture-of-vision-experts architecture that inherits the joint learning advantages from a single vision encoder while being restructured into a multi-expert paradigm for task-specific fine-tuning across different visual tasks. Additionally, we design a dynamic routing mechanism that allocates input images to the most suitable visual expert. Mixpert effectively alleviates domain conflicts encountered by a single vision encoder in multi-task learning with minimal additional computational cost, making it more efficient than multiple encoders. Furthermore, Mixpert integrates seamlessly into any MLLM, with experimental results demonstrating substantial performance gains across various tasks.

Related papers

• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders [17.14555102933619]
  Multimodal Large Language Models (MLLMs) increasingly adopt multiple vision encoders to capture diverse visual information.<n>We observe that the performance gains from adding encoders often diminish and can even lead to performance degradation.<n>To quantify each encoder's unique contribution, we propose a metric: the Conditional Utilization Rate (CUR)
  arXiv  Detail & Related papers  (2025-07-04T02:38:59Z)
• An Efficient and Mixed Heterogeneous Model for Image Restoration [71.85124734060665]
  Current mainstream approaches are based on three architectural paradigms: CNNs, Transformers, and Mambas.<n>We propose RestorMixer, an efficient and general-purpose IR model based on mixed-architecture fusion.
  arXiv  Detail & Related papers  (2025-04-15T08:19:12Z)
• MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing [2.0249250133493195]
  Multimodal language models (MLMs) integrate visual and textual information by coupling a vision encoder with a large language model through the specific adapter.<n>We propose MOVE (Mixture of Visions) to leverage multiple pre-trained encoders for specialized tasks.
  arXiv  Detail & Related papers  (2025-02-21T11:05:30Z)
• VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use [74.39058448757645]
  We present VipAct, an agent framework that enhances vision-language models (VLMs) VipAct consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks. We evaluate VipAct on benchmarks featuring a diverse set of visual perception tasks, with experimental results demonstrating significant performance improvements.
  arXiv  Detail & Related papers  (2024-10-21T18:10:26Z)
• Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders [89.41055673919895]
  This study explores the design space for MLLMs using a mixture of vision encoders and resolutions.<n>We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies.<n>The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.
  arXiv  Detail & Related papers  (2024-08-28T17:59:31Z)
• DeepInteraction++: Multi-Modality Interaction for Autonomous Driving [80.8837864849534]
  We introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout.<n>DeepInteraction++ is a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder.<n>Experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:04:21Z)
• MoVA: Adapting Mixture of Vision Experts to Multimodal Context [38.8308841469793]
  We propose the MoVA, a powerful and novel MLLM, adaptively routing and fusing task-specific vision experts with a coarse-to-fine mechanism. In the coarse-grained stage, we design a context-aware expert routing strategy to dynamically select the most suitable vision experts. In the fine-grained stage, we elaborately conduct the mixture-of-vision-expert adapter (MoV-Adapter) to extract and fuse task-specific knowledge.
  arXiv  Detail & Related papers  (2024-04-19T17:59:48Z)
• MouSi: Poly-Visual-Expert Vision-Language Models [132.58949014605477]
  This paper proposes the use of ensemble experts technique to synergize the capabilities of individual visual encoders. This technique introduces a fusion network to unify the processing of outputs from different visual experts. In our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1.
  arXiv  Detail & Related papers  (2024-01-30T18:09:11Z)
• M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition [39.92547393649842]
  We introduce a novel Multimodal, Multi-task CLIP adapting framework named name to address these challenges. We demonstrate exceptional performance in supervised learning while maintaining strong generalization in zero-shot scenarios.
  arXiv  Detail & Related papers  (2024-01-22T02:03:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.