Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity

• URL: http://arxiv.org/abs/2603.12707v1
• Date: Fri, 13 Mar 2026 06:42:35 GMT
• Title: Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
• Abstract summary: Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands.<n>We show that under standard transformer KV caching, the modality boundary minimizes cross-device transfer.<n>We build HeteroServe, a phase-aware runtime with modality-level partitioning and cross-tier scheduling.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal large language model (MLLM) inference splits into two phases with opposing hardware demands: vision encoding is compute-bound, while language generation is memory-bandwidth-bound. We show that under standard transformer KV caching, the modality boundary (between vision encoder and language model) minimizes cross-device transfer among all partition points that preserve standard stage-based execution. Partitioning here reduces transfer complexity from $O(L * s_ctx)$ bytes (GB-scale KV caches under stage-level disaggregation) to $O(N_v * d)$ bytes (MB-scale embeddings), an O(L) reduction where L is the transformer depth. The result holds across attention mechanisms (MHA/GQA), dynamic vision resolutions, and model scales, and the advantage grows as models deepen. A direct implication is that existing stage-level disaggregation systems are constrained to high-bandwidth interconnects (e.g., NVLink), whereas modality-level disaggregation enables cross-tier heterogeneous serving over commodity PCIe. A closed-form cost model shows that heterogeneous deployment is cost-optimal under phase-separable workloads (predicts 31.4% savings; observed 40.6%). We build HeteroServe, a phase-aware runtime with modality-level partitioning and cross-tier scheduling, and evaluate it on LLaVA-1.5-7B and Qwen2.5-VL against vLLM v0.3.0. On identical 4xA100 hardware, engine optimizations raise throughput by up to 54%. Under a fixed budget, a heterogeneous cluster (\$38k) improves Tokens/\$ by 37% over a homogeneous baseline (\$64k) without degrading latency.

Related papers

• Multimodal Resource-Exhaustion Attacks on Vision-Language Models via Joint Pixel-Prompt Optimization [38.37373243718745]
  We introduce Joint Pixel-Prompt Optimization (JPPO), the first compound adversarial framework elevating the visible prompt to a first-class adversarial variable alongside image perturbations.<n>Under a restricted joint-input threat model, JPPO performs coupled, stagewise optimization over both the pixel and prompt surfaces.<n>This produces synergistic cost amplification, mechanistically distinct from loop-dependent failures.
  arXiv  Detail & Related papers  (2026-09-05T05:20:53Z)
• Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware [5.066906328188419]
  Inference with transformer-based large language models (LLMs) is often limited by the memory-bound KV cache and quadratic attention cost.<n>We introduce a method that induces sparse neural activity in heavily quantized linear-attention models with minimal performance loss.
  arXiv  Detail & Related papers  (2026-08-31T08:28:01Z)
• YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition [78.52944812151664]
  Large language models (LLMs) drive significant financial innovations, yet their deployment is bottlenecked by KV cache memory overhead.<n>We propose YouZhi-LLM, a highly efficient financial LLM empowered by a comprehensive structural transition and training pipeline built on the Huawei Ascend ecosystem.
  arXiv  Detail & Related papers  (2026-06-04T08:44:37Z)
• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• Tiny Inference-Time Scaling with Latent Verifiers [56.696619768584675]
  Verifier on Hidden States ( VHS) operates on intermediate hidden representations of Diffusion Transformer (DiT) single-step generators.<n> VHS analyzes generator features without decoding to pixel space, thereby reducing the per-candidate verification cost.<n> VHS achieves a +2.7% improvement on GenEval at the same inference-time budget.
  arXiv  Detail & Related papers  (2026-03-23T19:00:02Z)
• Scaling State-Space Models on Multiple GPUs with Tensor Parallelism [0.24148976266903474]
  Selective state space models (SSMs) have rapidly become a compelling backbone for large language models.<n>But in deployment, their inference performance is often bounded by the memory capacity, bandwidth, and latency limits of a single GPU.<n>This paper presents a communication-efficient TP design for selective SSM inference that addresses three practical engineering challenges.
  arXiv  Detail & Related papers  (2026-02-24T17:47:54Z)
• ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding [37.86179431483446]
  Autoregressive models (ARMs) are hindered by slow sequential inference.<n>We introduce ReFusion, a novel masked diffusion model that achieves superior performance and efficiency.<n>ReFusion bridges the performance gap to strong ARMs while maintaining a 2.33$times$ average speedup.
  arXiv  Detail & Related papers  (2025-12-15T17:41:19Z)
• Memory- and Latency-Constrained Inference of Large Language Models via Adaptive Split Computing [8.705453442427585]
  Large language models (LLMs) have achieved near-human performance across diverse reasoning tasks.<n>Their deployment on resource-constrained Internet-of-Things (IoT) devices remains impractical due to massive parameter footprints and memory-intensive autoregressive decoding.<n>This work introduces the first autoregressive-aware split computing framework designed explicitly for LLM deployment on edge devices.
  arXiv  Detail & Related papers  (2025-11-06T02:55:07Z)
• I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation [0.0]
  Quantization offers an effective strategy to improve efficiency, but ViT-based segmentation models are notoriously fragile under low precision.<n>We introduce I-Segmenter, the first fully integer-only ViT segmentation framework.<n>I-Segmenter achieves competitive accuracy even in one-shot PTQ with a single calibration image.
  arXiv  Detail & Related papers  (2025-09-12T15:14:19Z)
• Flexiffusion: Training-Free Segment-Wise Neural Architecture Search for Efficient Diffusion Models [50.260693393896716]
  Diffusion models (DMs) are powerful generative models capable of producing high-fidelity images but constrained by high computational costs.<n>We propose Flexiffusion, a training-free NAS framework that jointly optimize generation schedules and model architectures without modifying pre-trained parameters.<n>Our work pioneers a resource-efficient paradigm for searching high-speed DMs without sacrificing quality.
  arXiv  Detail & Related papers  (2025-06-03T06:02:50Z)
• semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage [6.805644270436825]
  We propose a novel large language model (LLM) serving system, semi-PD, characterized by disaggregated computation and unified storage.<n>Compared to state-of-the-art systems, semi-PD maintains lower latency at higher request rates, reducing the average end-to-end latency per request by 1.27-2.58x.
  arXiv  Detail & Related papers  (2025-04-28T15:00:03Z)
• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models [84.84277196012907]
  We propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes.<n>We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V.
  arXiv  Detail & Related papers  (2024-12-02T18:58:25Z)
• Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts [60.1586169973792]
  M$3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE) MoE achieves better accuracy and over 80% reduction computation but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations.
  arXiv  Detail & Related papers  (2023-05-30T02:24:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.