Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling

• URL: http://arxiv.org/abs/2608.29291v2
• Date: Tue, 01 Sep 2026 05:51:02 GMT
• Title: Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling
• Abstract summary: Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps.<n>We propose a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment.<n> Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03% of dense understanding performance with a 1.93times$ end-to-end inference speedup.
• Score: 61.28599393707943
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps. Through token-importance probing, we identify an asymmetric core-expansion structure: understanding exhibits a stable importance component, while generation largely shares this component but requires progress-dependent corrections. We therefore propose CE-Router, which uses a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment. At inference, CE-Router compacts token computation and supplies a learned routing signal to Unified Computation Scheduling, which coordinates layer skipping, FFN pruning, diffusion-head cache reuse, and denoising-step early exit. Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03\% of dense understanding performance with a 1.93$\times$ end-to-end inference speedup.

Related papers

• Empowering Cross-Domain Sequential Recommendation with Hybrid Tokenization and Serial-Parallel Decoding [63.74998371740272]
  Cross-domain sequential recommendation (CDSR) aims to model users' dynamic interest transitions and sequential patterns across multiple domains.<n>We propose GenCDSR, an effective and efficient generative framework for CDSR.<n>We show that GenCDSR achieves an average accuracy improvement of 1.5 percent and an average inference latency reduction of 85.1 percent compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-07-21T08:01:15Z)
• DRDN: Decoupled Representation Dynamic Network for From-Scratch ViT Class-Incremental Learning [6.097306160389619]
  Dynamic expansion methods for class-incremental learning (CIL) protect task-specific knowledge by growing dedicated tokens or computationworks.<n>We identify two intertwined challenges: cross-task confusion from sequential training on predominantly current-task data, and under-optimized shared representations in the backbone.<n>We propose the Decoupled Representation Dynamic Network (DRDN), which addresses these challenges via two mechanisms.
  arXiv  Detail & Related papers  (2026-07-02T02:57:58Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• DCAU-Net: Differential Cross Attention and Channel-Spatial Feature Fusion for Medical Image Segmentation [2.2015188658021003]
  We propose DCAU-Net, a novel yet efficient segmentation framework.<n>First, a new Differential Cross Attention (DCA) is designed to compute the difference between two independent softmax attention maps.<n>Second, a Channel-Spatial Feature Fusion (CSFF) strategy is introduced to adaptively recalibrate features.
  arXiv  Detail & Related papers  (2026-03-10T11:37:10Z)
• UniX: Unifying Autoregression and Diffusion for Chest X-Ray Understanding and Generation [98.93314262366681]
  We present UniX, a next-generation unified medical foundation model for chest X-ray understanding and generation.<n>UniX decouples the two tasks into an autoregressive branch for understanding and a diffusion branch for high-fidelity generation.<n>On two representative benchmarks, UniX achieves a 46.1% improvement in understanding performance and a 24.2% gain in generation quality.
  arXiv  Detail & Related papers  (2026-01-16T18:59:58Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• Adapformer: Adaptive Channel Management for Multivariate Time Series Forecasting [49.40321003932633]
  Adapformer is an advanced Transformer-based framework that merges the benefits of CI and CD methodologies through effective channel management.<n>Adapformer achieves superior performance over existing models, enhancing both predictive accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-11-18T16:24:05Z)
• Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing [8.705453442427585]
  Large language models (LLMs) have achieved near-human performance across diverse reasoning tasks.<n>Their deployment on resource-constrained Internet-of-Things (IoT) devices remains impractical due to massive parameter footprints and memory-intensive autoregressive decoding.<n>This work introduces the first autoregressive-aware split computing framework designed explicitly for LLM deployment on edge devices.
  arXiv  Detail & Related papers  (2025-11-06T02:55:07Z)
• Time-Shifted Token Scheduling for Symbolic Music Generation [6.835980208223368]
  We adapt a delay-based scheduling mechanism to expand compound-like tokens across decoding steps.<n>Our method improves all metrics over standard compound tokenizations and narrows the gap to fine-grained tokenizations.
  arXiv  Detail & Related papers  (2025-09-28T08:52:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.