Aligned Vector Quantization for Edge-Cloud Collabrative Vision-Language Models

• URL: http://arxiv.org/abs/2411.05961v1
• Date: Fri, 08 Nov 2024 20:48:37 GMT
• Title: Aligned Vector Quantization for Edge-Cloud Collabrative Vision-Language Models
• Abstract summary: We introduce an edge-cloud collaborative VQA system called LLaVA-AlignedVQ. LLaVA-AlignedVQ features a novel Aligned Vector Quantization algorithm (AlignedVQ) that efficiently compress intermediate features. Experiments demonstrate that LLaVA-AlignedVQ achieves approximately 1365x compression rate of intermediate features.
• Score: 19.938589623698338
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models (VLMs) are central to Visual Question Answering (VQA) systems and are typically deployed in the cloud due to their high computational demands. However, this cloud-only approach underutilizes edge computational resources and requires significant bandwidth for transmitting raw images. In this paper, we introduce an edge-cloud collaborative VQA system, called LLaVA-AlignedVQ, which features a novel Aligned Vector Quantization algorithm (AlignedVQ) that efficiently compress intermediate features without compromising accuracy to support partitioned execution. Our experiments demonstrate that LLaVA-AlignedVQ achieves approximately 1365x compression rate of intermediate features, reducing data transmission overhead by 96.8% compared to transmitting JPEG90-compressed images to the cloud. LLaVA-AlignedVQ achieves an inference speedup of 2-15x while maintaining high accuracy, remaining within -2.23% to +1.6% of the original model's accuracy performance across eight VQA datasets, compared to the cloud-only solution.

Related papers

• DynaFilter: Cloud-driven Dynamic Filtering for Satellite Edge Intelligence [24.492347669238175]
  DynaFilter is a dynamic filtering technique that enables satellite edge devices to perform selective region-of-interest (RoI) inference directly in the compressed-domain.<n>We show that DynaFilter reduces the total volume of pixel data for decoding and subsequent inference by 1.6x-7.1x for images.<n>It achieves 92.0% bandwidth savings for video streams compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-07-11T03:28:25Z)
• INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inference [0.8633013637160062]
  We present INAR-VL, a lightweight edge-cloud routing system for multimodal inference in a two-tier deployment.<n>We show that INAR-VL executes 36% of requests on the edge, reduces latency by 24%, lowers energy by 26%, and preserves 97% of cloud-level accuracy.
  arXiv  Detail & Related papers  (2026-05-13T12:44:10Z)
• Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression [53.041405703471895]
  We propose RDVQ, a unified framework that enables end-to-end RD optimization for low-bitrate image compression.<n>We develop an autoregressive entropy model that supports accurate entropy modeling and test-time rate control.<n>Compared with RD2K-val, RDVQ reduces by up to 75.71% on DISTS and 37.63% on LPIPS on DIV2K-val.
  arXiv  Detail & Related papers  (2026-04-12T09:25:24Z)
• DiffusionX: Efficient Edge-Cloud Collaborative Image Generation with Multi-Round Prompt Evolution [10.001414172068623]
  DiffusionX is a cloud-edge collaborative framework for efficient multi-round, prompt-based generation.<n>A lightweight on-device diffusion model interacts with users by rapidly producing preview images, while a high-capacity cloud model performs final refinements.<n>Experiments show that DiffusionX reduces average generation time by 15.8% compared with Stable Diffusion v1.5, while maintaining comparable image quality.
  arXiv  Detail & Related papers  (2025-10-18T03:20:39Z)
• QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models [13.850959421148273]
  Vision-Language Models (VLMs) are integral to tasks such as image captioning and visual question answering.<n>Their high computational cost, driven by large memory footprints and processing time, limits their scalability and real-time applicability.<n>We propose leveraging Singular-Value Decomposition (SVD) over the joint query (Q), key (K), and value (V) weight matrices to reduce KV cache size and computational overhead.
  arXiv  Detail & Related papers  (2025-10-18T01:31:14Z)
• Codebook-Based Adaptive Feature Compression With Semantic Enhancement for Edge-Cloud Systems [20.88178006915217]
  CAFC-SE is a Codebook-based Adaptive Feature Compression framework with Semantic Enhancement.<n>It maps continuous visual features to discrete indices with a codebook at the edge via Vector Quantization (VQ) and selectively transmits them to the cloud.<n>The VQ operation that projects feature vectors onto the nearest visual primitives enables us to preserve more informative visual patterns under low-bitrate conditions.
  arXiv  Detail & Related papers  (2025-09-23T00:34:12Z)
• EC-Diff: Fast and High-Quality Edge-Cloud Collaborative Inference for Diffusion Models [57.059991285047296]
  hybrid edge-cloud collaborative framework was recently proposed to realize fast inference and high-quality generation.<n>Excessive cloud denoising prolongs inference time, while insufficient steps cause semantic ambiguity, leading to inconsistency in edge model output.<n>We propose EC-Diff that accelerates cloud inference through gradient-based noise estimation.<n>Our method significantly enhances generation quality compared to edge inference, while achieving up to an average $2times$ speedup in inference compared to cloud inference.
  arXiv  Detail & Related papers  (2025-07-16T07:23:14Z)
• Lightweight Cloud Masking Models for On-Board Inference in Hyperspectral Imaging [0.6775616141339018]
  Cloud and cloud shadow masking is a crucial preprocessing step in hyperspectral satellite imaging.<n>This study evaluates various machine learning approaches, including gradient boosting and convolutional neural networks (CNNs)<n>CNNs with feature reduction emerged as the most efficient, offering a balance of high accuracy, low storage requirements, and rapid inference times.
  arXiv  Detail & Related papers  (2025-07-10T08:10:11Z)
• EEPNet-V2: Patch-to-Pixel Solution for Efficient Cross-Modal Registration between LiDAR Point Cloud and Camera Image [15.975048012603914]
  We propose a framework that projects point clouds into several 2D representations for matching with camera images.<n>To tackle the challenges of cross modal differences and the limited overlap between LiDAR point clouds and images in the image matching task, we introduce a multi-scale feature extraction network.<n>We validate the performance of our model through experiments on the KITTI and nuScenes datasets.
  arXiv  Detail & Related papers  (2025-03-19T15:04:01Z)
• Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference [49.77734021302196]
  We propose a task-oriented feature compression (TOFC) method for multimodal understanding in a device-edge co-inference framework. To enhance compression efficiency, multiple entropy models are adaptively selected based on the characteristics of the visual features. Results show that TOFC achieves up to 60% reduction in data transmission overhead and 50% reduction in system latency.
  arXiv  Detail & Related papers  (2025-03-17T08:37:22Z)
• Point Cloud-Assisted Neural Image Compression [35.46346027449056]
  In this paper, we increase image compression performance with the assistance of point cloud. We propose the point cloud-assisted neural image (PCA-NIC) to enhance the preservation of image texture and structure. Our work is the first to improve image compression performance using point cloud and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-16T13:44:26Z)
• CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration [1.6021932740447968]
  Large Language Models (LLMs) exhibit remarkable human-like predictive capabilities.<n>It is challenging to deploy LLMs to provide efficient and adaptive inference services at the edge.<n>This paper proposes a novel Cloud-Edge Collaboration framework for LLMs (CE-CoLLM) to tackle these challenges.
  arXiv  Detail & Related papers  (2024-11-05T06:00:27Z)
• Distributed Semantic Segmentation with Efficient Joint Source and Task Decoding [18.8803233019656]
  Deep neural networks (DNNs) execute one part of the network on edge devices and the other part on a large-scale cloud platform. In this work, we propose joint source and task decoding, as it allows for a smaller network size in the cloud. We demonstrate the effectiveness of our method by achieving a distributed semantic segmentation SOTA over a wide range of intersections.
  arXiv  Detail & Related papers  (2024-07-15T20:20:04Z)
• IDF-CR: Iterative Diffusion Process for Divide-and-Conquer Cloud Removal in Remote-sensing Images [55.40601468843028]
  We present an iterative diffusion process for cloud removal (IDF-CR) IDF-CR is divided into two-stage models that address pixel space and latent space. In the latent space stage, the diffusion model transforms low-quality cloud removal into high-quality clean output.
  arXiv  Detail & Related papers  (2024-03-18T15:23:48Z)
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models [65.37846460916042]
  We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs. We introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency.
  arXiv  Detail & Related papers  (2024-03-11T14:35:32Z)
• Distribution-aware Interactive Attention Network and Large-scale Cloud Recognition Benchmark on FY-4A Satellite Image [24.09239785062109]
  We develop a novel dataset for accurate cloud recognition. We use domain adaptation methods to align 70,419 image-label pairs in terms of projection, temporal resolution, and spatial resolution. We also introduce a Distribution-aware Interactive-Attention Network (DIAnet), which preserves pixel-level details through a high-resolution branch and a parallel cross-branch.
  arXiv  Detail & Related papers  (2024-01-06T09:58:09Z)
• Towards Efficient Object Re-Identification with A Novel Cloud-Edge Collaborative Framework [25.587445307574132]
  Object re-identification (ReID) is committed to searching for objects of the same identity across cameras. Current ReID methods assume that the deployed system follows the centralized processing paradigm. We propose a cloud-edge collaborative framework for ReID systems, aiming to expedite the return of the desired image to the cloud server.
  arXiv  Detail & Related papers  (2024-01-04T02:56:50Z)
• Does Vector Quantization Fail in Spatio-Temporal Forecasting? Exploring a Differentiable Sparse Soft-Vector Quantization Approach [22.070533429289334]
  We present Sparse Soft- Quantization (SVQ), the first VQ method to enhance temporal forecasting.<n>SVQ balances with sparse noise reduction, offering fulliability and a solid foundation in sparse regression.<n>Our approach employs a two-layer dataset and an extensive codebook to streamline the sparse regression process.
  arXiv  Detail & Related papers  (2023-12-06T10:42:40Z)
• Simple Baselines for Projection-based Full-reference and No-reference Point Cloud Quality Assessment [60.2709006613171]
  We propose simple baselines for projection-based point cloud quality assessment (PCQA) We use multi-projections obtained via a common cube-like projection process from the point clouds for both full-reference (FR) and no-reference (NR) PCQA tasks. Taking part in the ICIP 2023 PCVQA Challenge, we succeeded in achieving the top spot in four out of the five competition tracks.
  arXiv  Detail & Related papers  (2023-10-26T04:42:57Z)
• Boosting Point Clouds Rendering via Radiance Mapping [49.24193509772339]
  We focus on boosting the image quality of point clouds rendering with a compact model design. We simplify the NeRF representation to a spatial mapping function which only requires single evaluation per pixel. Our method achieves the state-of-the-art rendering on point clouds, outperforming prior works by notable margins.
  arXiv  Detail & Related papers  (2022-10-27T01:25:57Z)
• Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment [60.57703721744873]
  The increased resolution of real-world videos presents a dilemma between efficiency and accuracy for deep Video Quality Assessment (VQA) In this work, we propose a unified scheme, spatial-temporal grid mini-cube sampling (St-GMS) to get a novel type of sample, named fragments. With fragments and FANet, the proposed efficient end-to-end FAST-VQA and FasterVQA achieve significantly better performance than existing approaches on all VQA benchmarks.
  arXiv  Detail & Related papers  (2022-10-11T11:38:07Z)
• FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling [54.31355080688127]
  Current deep video quality assessment (VQA) methods are usually with high computational costs when evaluating high-resolution videos. We propose Grid Mini-patch Sampling (GMS), which allows consideration of local quality by sampling patches at their raw resolution. We build the Fragment Attention Network (FANet) specially designed to accommodate fragments as inputs. FAST-VQA improves state-of-the-art accuracy by around 10% while reducing 99.5% FLOPs on 1080P high-resolution videos.
  arXiv  Detail & Related papers  (2022-07-06T11:11:43Z)
• Identity Preserving Loss for Learned Image Compression [0.0]
  This work proposes an end-to-end image compression framework that learns domain-specific features to achieve higher compression ratios. We present a novel Identity Preserving Reconstruction (IPR) loss function which achieves Bits-Per-Pixel (BPP) values that are 38% and 42% of CRF-23 HEVC compression. We show at-par recognition performance on the LFW dataset with an unseen recognition model while retaining a lower BPP value of 38% of CRF-23 HEVC compression.
  arXiv  Detail & Related papers  (2022-04-22T18:01:01Z)
• Feature Compression for Rate Constrained Object Detection on the Edge [20.18227104333772]
  An emerging approach to solve this problem is to offload the computation of neural networks to computing resources at an edge server. In this work, we consider a "split computation" system to offload a part of the computation of the YOLO object detection model. We train the feature compression and decompression module together with the YOLO model to optimize the object detection accuracy under a rate constraint.
  arXiv  Detail & Related papers  (2022-04-15T03:39:30Z)
• Multiscale deep context modeling for lossless point cloud geometry compression [11.69103847045569]
  MSVoxelDNN is a deep generative approach for point cloud geometry compression. It significantly reduces the rate compared to the MPEG G-PCC.
  arXiv  Detail & Related papers  (2021-04-20T09:48:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.