Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder

• URL: http://arxiv.org/abs/2603.05528v1
• Date: Fri, 27 Feb 2026 06:41:25 GMT
• Title: Omni-C: Compressing Heterogeneous Modalities into a Single Dense Encoder
• Abstract summary: We propose Omni-C, a single dense Transformer-based encoder that learns competitive shared representations across heterogeneous modalities.<n>By maximizing parameter sharing in the backbone, Omni-C effectively mitigates inter-modality conflicts without requiring MoE, paired supervision, or routing.<n>Experiments show Omni-C achieves performance comparable to expert models in unimodal and cross-model tasks.
• Score: 18.13981375699951
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent multimodal systems often rely on separate expert modality encoders which cause linearly scaling complexity and computational overhead with added modalities. While unified Omni-models address this via Mixture-of-Expert (MoE) architectures with specialized experts and routing, they still inflate parameter counts and introduce routing overhead. In this paper, we propose Omni-C (Omni-Compress), a single dense Transformer-based encoder that learns competitive shared representations across heterogeneous modalities--images, audio, and text--through unimodal contrastive pretraining on large-scale unaligned data. By maximizing parameter sharing in the backbone and using lightweight modality-specific projection heads, Omni-C effectively mitigates inter-modality conflicts without requiring MoE, paired supervision, or routing. This design supports efficient deployment on memory-constrained systems via sequential modality processing and low-memory inference, eliminating the need for parallel expert loading or specialized hardware. Experiments show Omni-C achieves performance comparable to expert models in unimodal and cross-model tasks, with modest zero-shot degradation on audio and text that is largely recovered through lightweight linear probing or parameter efficient fine-tuning. The unified architecture substantially reduces inference memory usage compared to multi-encoder baselines, advancing efficient and scalable multimodal learning.

Related papers

• OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models [49.34557936004057]
  We propose a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs.<n>Our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.
  arXiv  Detail & Related papers  (2026-05-18T08:33:52Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation [8.840077295284393]
  MoBaNet is a parameter-efficient and modality-balanced symmetric fusion framework.<n>Built upon a largely frozen VFM backbone, MoBaNet adopts a symmetric dual-stream architecture to preserve generalizable representations.<n>Experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that MoBaNet achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-18T13:23:58Z)
• Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking [16.123153889076104]
  We propose a novel Sparse-Dense Mixture of Experts Adapter (SDMoEA) framework for PEFT-based multi-modal tracking.<n>To overcome limitations of existing tracking methods in modeling high-order correlations during multi-level multi-modal fusion, we introduce a Gram-based Semantic Alignment Hypergraph Fusion (GSAHF) module.<n>The proposed method achieves superior performance compared with other PEFT approaches on several multi-modal tracking benchmarks.
  arXiv  Detail & Related papers  (2026-03-14T02:51:10Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Dynamic Expert Sharing: Decoupling Memory from Parallelism in Mixture-of-Experts Diffusion LLMs [22.399470395813577]
  Dynamic Expert Sharing (DES) is a novel technique that shifts MoE optimization from token-centric pruning to sequence-level coreset selection.<n>DES reduces unique expert activations by over 55% and latency by up to 38%, while retaining 99% of vanilla accuracy.
  arXiv  Detail & Related papers  (2026-01-31T20:01:47Z)
• MM-DETR: An Efficient Multimodal Detection Transformer with Mamba-Driven Dual-Granularity Fusion and Frequency-Aware Modality Adapters [12.063966356953186]
  Multimodal remote sensing object detection aims to achieve more accurate and robust perception under challenging conditions.<n>Existing approaches that rely on attention-based or deformable convolution fusion blocks still struggle to balance performance and lightweight design.<n>We propose MM-DETR, a lightweight and efficient framework for multimodal object detection.
  arXiv  Detail & Related papers  (2025-11-29T07:23:01Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation [91.45421429922506]
  OneCAT is a unified multimodal model that seamlessly integrates understanding, generation, and editing.<n>Our framework eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference.
  arXiv  Detail & Related papers  (2025-09-03T17:29:50Z)
• Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking [17.095655627061934]
  We present Decom-Renorm-Merge (DRM), a simple yet effective approach that leverages Singular Value Decomposition to decompose and coordinate weight matrices into an aligned joint space.<n>Our experimental results show that DRM outperforms several state-of-the-art merging techniques across full finetuning and low-rank adaptation settings.
  arXiv  Detail & Related papers  (2025-05-29T05:37:53Z)
• Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition [95.54406667705999]
  Pangu Embedded is an efficient Large Language Model (LLM) reasoner developed on Ascend Neural Processing Units (NPUs)<n>It addresses the significant computational costs and inference latency challenges prevalent in existing reasoning-optimized LLMs.<n>It delivers rapid responses and state-of-the-art reasoning quality within a single, unified model architecture.
  arXiv  Detail & Related papers  (2025-05-28T14:03:02Z)
• Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregation [15.35494431928751]
  Transformer-based large language models (LLMs) exhibit impressive performance in generative tasks but also introduce significant challenges in real-world serving.<n>We introduce model-attention disaggregation to enhance the efficiency of LLM decoding.<n>We develop and deploy Lamina, an LLM inference system that incorporates model-attention disaggregation in a distributed heterogeneous cluster.
  arXiv  Detail & Related papers  (2024-05-03T02:15:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.