DIVE: Embedding Compression via Self-Limiting Gradient Updates

• URL: http://arxiv.org/abs/2605.20689v1
• Date: Wed, 20 May 2026 04:35:28 GMT
• Title: DIVE: Embedding Compression via Self-Limiting Gradient Updates
• Abstract summary: High-dimensional embeddings from large language models impose significant storage and computational costs on vector search systems.<n>Recent embedding compression methods, including Matryoshka-Adaptor (EMNLP 2024), enable dimensionality reduction through lightweight residual adapters, but their training objectives cause severe overfitting when labeled data is scarce.<n>We propose textscDIVE (textbfDimensionality reduction with textbfImplicit textbfView textbfEnsembles), a compression adapter that addresses this failure through two mechanisms.
• Score: 0.9179857807576733
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: High-dimensional embeddings from large language models impose significant storage and computational costs on vector search systems. Recent embedding compression methods, including Matryoshka-Adaptor (EMNLP 2024), Search-Adaptor (ACL 2024), and SMEC (EMNLP 2025), enable dimensionality reduction through lightweight residual adapters, but their training objectives cause severe overfitting when labeled data is scarce, degrading retrieval performance below the frozen baseline. We propose \textsc{DIVE} (\textbf{D}imensionality reduction with \textbf{I}mplicit \textbf{V}iew \textbf{E}nsembles), a compression adapter that addresses this failure through two mechanisms. First, a self-limiting hinge-based triplet loss produces zero gradient once a triplet satisfies the margin constraint, bounding the total perturbation applied to the pretrained embedding space. Second, a head-wise NT-Xent contrastive loss treats multiple learned projections of each embedding as implicit views, providing dense self-supervised gradients that compensate for the sparsity of the triplet signal on small datasets. Across six BEIR datasets, \textsc{DIVE} outperforms all three baseline adapters on every dataset and at every evaluated compression ratio, with a 14M-parameter open-source implementation.

Related papers

• Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression [8.890261387138215]
  We introduce a three-level chain that widens optimization scope from individual matrices to Transformer blocks to the full model.<n>On LLaMA-7B at 60% compression, the chain reduces WikiText-2 perplexity from 42.1 to 19.1 to 11.4.<n> skipping the block-level stage becomes competitive, revealing an offline compute--data trade-off.
  arXiv  Detail & Related papers  (2026-09-14T16:36:46Z)
• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Prediction [9.96771578356719]
  Existing approaches rely on increasingly complex network architectures to regress depth maps.<n>We propose a multilevel conditional random field (CRF) model built upon the Swin Transformer backbone.<n>Our method achieves state-of-the-art performance, reducing Abs Rel to 0.088 ($-$7.4%) and RMSE to 0.316 ($-$5.4%) on NYU Depth v2, while attaining near-perfect threshold accuracy.
  arXiv  Detail & Related papers  (2026-04-03T07:59:26Z)
• LACE: Loss-Adaptive Capacity Expansion for Continual Learning [0.0]
  LACE (Loss-Adaptive Capacity Expansion) is an online mechanism that expands a model's representational capacity during training.<n>LACE requires no labels, no replay buffers, and no external controllers, making it suitable for on-device continual learning under resource constraints.
  arXiv  Detail & Related papers  (2026-03-30T15:58:33Z)
• CSGaussian: Progressive Rate-Distortion Compression and Segmentation for 3D Gaussian Splatting [57.73006852239138]
  We present the first unified framework for rate-distortion-optimized compression and segmentation of 3D Gaussian Splatting (3DGS)<n>Inspired by recent advances in rate-distortion-optimized 3DGS compression, this work integrates semantic learning into the compression pipeline to support decoder-side applications.<n>Our scheme features a lightweight implicit neural representation-based hyperprior, enabling efficient entropy coding of both color and semantic attributes.
  arXiv  Detail & Related papers  (2026-01-19T08:21:45Z)
• Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression [57.71917274869577]
  UltraDelta is a data-free delta compression pipeline that achieves both ultra-high compression and strong performance.<n>UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions.
  arXiv  Detail & Related papers  (2025-05-19T10:37:22Z)
• MoDeGPT: Modular Decomposition for Large Language Model Compression [59.361006801465344]
  This paper introduces textbfModular bfDecomposition (MoDeGPT), a novel structured compression framework.<n>MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions.<n>Our experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods.
  arXiv  Detail & Related papers  (2024-08-19T01:30:14Z)
• Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depth [83.15263499262824]
  We prove that gradient descent converges to a solution that completely disregards the sparse structure of the input. We show how to improve upon Gaussian performance for the compression of sparse data by adding a denoising function to a shallow architecture. We validate our findings on image datasets, such as CIFAR-10 and MNIST.
  arXiv  Detail & Related papers  (2024-02-07T16:32:29Z)
• Error Feedback Can Accurately Compress Preconditioners [43.60787513716217]
  Leveraging second-order information about the loss at the scale of deep networks is one of the main lines of approach for improving the performance of currents for deep learning. Yet, existing approaches for accurate full-matrix preconditioning, such as Full-Matrix Adagrad (GGT) or Matrix-Free Approximate Curvature (M-FAC) suffer from massive storage costs when applied even to small-scale models. In this paper, we address this issue via a novel and efficient error-feedback technique that can be applied to compress preconditioners by up to two orders of magnitude in practice, without loss of convergence.
  arXiv  Detail & Related papers  (2023-06-09T17:58:47Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.