CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding

• URL: http://arxiv.org/abs/2605.02218v1
• Date: Mon, 04 May 2026 04:43:15 GMT
• Title: CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding
• Abstract summary: CoVSpec is an efficient collaborative speculative decoding framework for vision-language models (VLMs) inference.<n>We show that CoVSpec achieves up to 2.21x higher throughput than target-only inference and reduces communication overhead by more than 96% compared with baselines.
• Score: 5.680072178589744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) have demonstrated strong capabilities in multimodal perception and reasoning. However, deploying large VLMs on mobile devices remains challenging due to their substantial computational and memory demands. A practical alternative is device-edge co-inference, where a lightweight draft VLM on the mobile device collaborates with a larger target VLM on the edge server via speculative decoding. Nevertheless, directly extending speculative decoding to VLMs suffers from severe inefficiency due to excessive visual-token computation and high communication overhead. To address these challenges, we propose CoVSpec, an efficient collaborative speculative decoding framework for VLM inference. Specifically, we first develop a training-free visual token reduction framework that prunes redundant visual tokens on the mobile device by jointly considering query relevance, token activity, and low-rank dependency. Moreover, we design an adaptive drafting strategy that dynamically adjusts both the verification frequency and the draft length. In addition, we introduce a parallel branching mechanism with decoupled verification-correction to improve draft-side utilization during target-side verification and reduce correction-related transmission overhead. Experiments on multiple benchmarks show that CoVSpec achieves up to 2.21x higher throughput than target-only inference and reduces communication overhead by more than 96% compared with baselines, without compromising task accuracy.

Related papers

• CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning [79.07656918674684]
  CoME-VL is a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder.<n>Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin.
  arXiv  Detail & Related papers  (2026-04-03T17:59:51Z)
• Efficient Reasoning on the Edge [22.390056779820295]
  Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks.<n>Their verbose reasoning traces and large context requirements make them impractical for edge deployment.<n>We propose a lightweight approach to enable reasoning in small LLMs using LoRA adapters combined with supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-03-17T17:59:51Z)
• Empirical Recipes for Efficient and Compact Vision-Language Models [54.92440500651415]
  Vision-language models (VLMs) in resource-constrained settings demand low latency and high throughput.<n>We conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks.<n>We develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy.
  arXiv  Detail & Related papers  (2026-03-17T17:17:40Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders [40.81958598891815]
  Vision Language Model (VLM) development has largely relied on scaling model size.<n>We present Penguin-VL, whose vision encoder is from a text-only LLM.<n>Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining.
  arXiv  Detail & Related papers  (2026-03-06T18:58:04Z)
• ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning [8.933549837045932]
  Large Vision-Language Models incur high computational costs due to significant redundancy in their visual tokens.<n>We propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the Large Language Models.
  arXiv  Detail & Related papers  (2026-01-25T12:47:30Z)
• Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization [61.55616421408666]
  Low-Altitude Economy Networks (LAENets) have enabled a variety of applications, including aerial surveillance, environmental sensing, and semantic data collection.<n> onboard vision (VLMs) offer inference for real-time inference but limited onboard dynamic network conditions.<n>We propose a UAV-enabled LAENet system that improves communication efficiency under dynamic LAENet conditions.
  arXiv  Detail & Related papers  (2025-10-11T05:11:21Z)
• SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving [7.69991252191073]
  acronym is a framework that allows lightweight edge devices to draft multiple candidate tokens locally using diverse draft models.<n>A single, shared edge server verifies the tokens utilizing a more precise target model.<n>Our initial experiments with Jetson Orin Nano, Raspberry Pi 4B/5, and an edge server equipped with 4 Nvidia A100 GPUs indicate substantial benefits.
  arXiv  Detail & Related papers  (2025-06-11T04:55:54Z)
• Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference [54.53508601749513]
  We propose a task-oriented feature compression (TOFC) method for multimodal understanding in a device-edge co-inference framework.<n>To enhance compression efficiency, multiple entropy models are adaptively selected based on the characteristics of the visual features.<n>Results show that TOFC achieves up to 52% reduction in data transmission overhead and 63% reduction in system latency.
  arXiv  Detail & Related papers  (2025-03-17T08:37:22Z)
• EVEv2: Improved Baselines for Encoder-Free Vision-Language Models [72.07868838411474]
  Existing encoder-free vision-language models (VLMs) are narrowing the performance gap with their encoder-based counterparts.<n>We develop efficient strategies for encoder-free VLMs that rival mainstream encoder-based ones.<n>We show that properly and hierarchically associating vision and language within a unified model reduces interference between modalities.
  arXiv  Detail & Related papers  (2025-02-10T18:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.