CSRv2: Unlocking Ultra-Sparse Embeddings

• URL: http://arxiv.org/abs/2602.05735v3
• Date: Tue, 10 Feb 2026 01:24:48 GMT
• Title: CSRv2: Unlocking Ultra-Sparse Embeddings
• Abstract summary: Contrastive Sparse Representation (CSR) is proposed as a promising direction to map dense embeddings into high-dimensional but k-sparse vectors.<n>CSR suffers severe degradation in the ultra-sparse regime, where over 80% of neurons remain inactive.<n>We introduce CSRv2, a principled training approach designed to make ultra-sparse embeddings viable.
• Score: 52.553928856110296
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the era of large foundation models, the quality of embeddings has become a central determinant of downstream task performance and overall system capability. Yet widely used dense embeddings are often extremely high-dimensional, incurring substantial costs in storage, memory, and inference latency. To address these, Contrastive Sparse Representation (CSR) is recently proposed as a promising direction, mapping dense embeddings into high-dimensional but k-sparse vectors, in contrast to compact dense embeddings such as Matryoshka Representation Learning (MRL). Despite its promise, CSR suffers severe degradation in the ultra-sparse regime, where over 80% of neurons remain inactive, leaving much of its efficiency potential unrealized. In this paper, we introduce CSRv2, a principled training approach designed to make ultra-sparse embeddings viable. CSRv2 stabilizes sparsity learning through progressive k-annealing, enhances representational quality via supervised contrastive objectives, and ensures end-to-end adaptability with full backbone finetuning. CSRv2 reduces dead neurons from 80% to 20% and delivers a 14% accuracy gain at k=2, bringing ultra-sparse embeddings on par with CSR at k=8 and MRL at 32 dimensions, all with only two active features. While maintaining comparable performance, CSRv2 delivers a 7x speedup over MRL, and yields up to 300x improvements in compute and memory efficiency relative to dense embeddings in text representation. Extensive experiments across text and vision demonstrate that CSRv2 makes ultra-sparse embeddings practical without compromising performance, where CSRv2 achieves 7%/4% improvement over CSR when k=4 and further increases this gap to 14%/6% when k=2 in text/vision representation. By making extreme sparsity viable, CSRv2 broadens the design space for real-time and edge-deployable AI systems where both embedding quality and efficiency are critical.

Related papers

• Coupled Continuous-Discrete Generation for Scene Text Image Super-Resolution [51.31474187379532]
  Scene text image super-resolution (STISR) aims to recover visually plausible appearance while preserving character semantics from degraded inputs.<n>We present DualTSR, a unified framework that formulates STISR as coupled continuous-discrete generation.<n> Conditional flow matching restores continuous image latents, while absorbing-state discrete diffusion reconstructs text tokens.
  arXiv  Detail & Related papers  (2026-08-05T06:57:30Z)
• Efficient One-Step Diffusion Restoration Model with Compact Token Compression and Linear Attention [66.63806505114263]
  Existing Real-ISR methods inherit the dense latent representations and quadratic-cost global modeling paradigm.<n>We argue that the key bottleneck lies in excessive token redundancy and costly token interactions during high-resolution restoration.<n>We introduce a linear-attention DiT with LoRA fine-tuning, enabling efficient high-resolution restoration with linear-complexity token mixing.
  arXiv  Detail & Related papers  (2026-05-22T10:07:02Z)
• QuantSR+: Pushing the Limit of Quantized Image Super-Resolution Networks [63.04549067734431]
  Low-bit quantization is widely used to compress super-resolution (SR) models.<n>However, when SR models are pushed to ultra-low precision (2-4 bits), performance can drop sharply.<n>We propose QuantSR+, a unified framework that improves quantization operators, network design, and training optimization.
  arXiv  Detail & Related papers  (2026-05-21T11:38:52Z)
• R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning [40.03346193264488]
  We propose Robust Representation via Redundancy Reduction (R2R2) as a regularization method within Self-Predictive Learning (SPL)<n>We verify R2R2 on SPL-native algorithms like TD7.<n> Experiments across 11 continuous control tasks confirm that R2R2 effectively mitigates overfitting.
  arXiv  Detail & Related papers  (2026-05-13T18:38:32Z)
• DARE: Diffusion Language Model Activation Reuse for Efficient Inference [26.967107786841936]
  Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to auto-regressive (AR) models.<n>We introduce DARE, with two complementary mechanisms: DARE-KV, which reuses cached key-value (KV) activations, and DARE-O, which reuses output activations.<n>DARE achieves up to 1.20x per-layer latency reduction and reuses up to 87% of attention activations, with negligible degradation on reasoning and code-generation benchmarks.
  arXiv  Detail & Related papers  (2026-05-01T19:15:45Z)
• DeFRiS: Silo-Cooperative IoT Applications Scheduling via Decentralized Federated Reinforcement Learning [62.347535250646196]
  This paper proposes DeFRiS, a Decentralized Federated Reinforcement Learning framework for robust and scalable Silo-cooperative IoT application scheduling.<n>DeFRiS integrates three innovations: (i) an action-space-agnostic policy utilizing candidate resource scoring to enable seamless knowledge transfer across heterogeneous silos; (ii) a silo-optimized local learning mechanism combining Generalized Advantage Estimation with clipped policy updates to resolve sparse delayed reward challenges; and (iii) a Dual-Track Non-IID robust decentralized aggregation protocol leveraging gradient fingerprints for similarity-aware knowledge transfer and anomaly detection.
  arXiv  Detail & Related papers  (2026-03-16T02:02:38Z)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity [54.95089105944234]
  Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens.<n>We propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline.<n>We demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT.
  arXiv  Detail & Related papers  (2026-03-10T10:31:58Z)
• Efficient-LVSM: Faster, Cheaper, and Better Large View Synthesis Model via Decoupled Co-Refinement Attention [105.11288339285154]
  Efficient-LVSM is a dual-stream architecture that applies intra-view self-attention for input views and self-then-cross attention for target views.<n>It achieves 29.86 dB PSNR on RealEstate10K with 2 input views, surpassing LVSM by 0.2 dB, with 2x faster training convergence and 4.4x faster inference speed.
  arXiv  Detail & Related papers  (2026-02-06T08:11:58Z)
• SRSR: Enhancing Semantic Accuracy in Real-World Image Super-Resolution with Spatially Re-Focused Text-Conditioning [59.013863248600046]
  We propose a spatially re-focused super-resolution framework that refines text conditioning at inference time.<n>Second, we introduce a Spatially Targeted-Free Guidance mechanism that selectively bypasses text influences on ungrounded pixels to prevent hallucinations.
  arXiv  Detail & Related papers  (2025-10-26T05:03:55Z)
• KeyKnowledgeRAG (K^2RAG): An Enhanced RAG method for improved LLM question-answering capabilities [2.4874078867686085]
  KeyKnowledgeRAG (K2RAG) is a novel framework designed to overcome limitations in RAG implementations.<n>It integrates dense and sparse vector search, knowledge graphs, and text summarization to improve retrieval quality and system efficiency.<n>K2RAG achieved the highest mean answer similarity score of 0.57, and reached the highest third quartile (Q3) similarity of 0.82, indicating better alignment with ground-truth answers.
  arXiv  Detail & Related papers  (2025-07-10T12:19:03Z)
• Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation [42.590255022001145]
  Matryoshka Representation Learning (MRL) recently emerged as a solution for adaptive embedding lengths.<n>We show that sparse coding offers a compelling alternative for achieving adaptive representation with minimal overhead and higher fidelity.
  arXiv  Detail & Related papers  (2025-03-03T17:59:48Z)
• NVS-SQA: Exploring Self-Supervised Quality Representation Learning for Neurally Synthesized Scenes without References [57.0432939964225]
  We propose NVS-SQA, a quality assessment method to learn no-reference quality representations through self-supervision.<n>Traditional self-supervised learning predominantly relies on the "same instance, similar representation" assumption and extensive datasets.<n>We employ photorealistic cues and quality scores as learning objectives, along with a specialized contrastive pair preparation process to improve the effectiveness and efficiency of learning.
  arXiv  Detail & Related papers  (2025-01-11T09:12:43Z)
• VICON: Vision In-Context Operator Networks for Multi-Physics Fluid Dynamics Prediction [30.201826592090885]
  In-Context Operator Networks (ICONs) learn operators across diverse partial differential equations using few-shot, in-context learning.<n>Existing ICONs process each spatial point as an individual token, severely limiting computational efficiency when handling dense data in higher spatial dimensions.<n>We propose Vision In-Context Operator Networks (VICON), which integrates vision transformer architectures to efficiently process 2D data through patch-wise operations.
  arXiv  Detail & Related papers  (2024-11-25T03:25:17Z)
• Sebica: Lightweight Spatial and Efficient Bidirectional Channel Attention Super Resolution Network [0.0]
  Single Image Super-Resolution (SISR) is a vital technique for improving the visual quality of low-resolution images. We present Sebica, a lightweight network that incorporates spatial and efficient bidirectional channel attention mechanisms. Sebica significantly reduces computational costs while maintaining high reconstruction quality.
  arXiv  Detail & Related papers  (2024-10-27T18:27:07Z)
• Structured Pruning for Efficient Visual Place Recognition [24.433604332415204]
  Visual Place Recognition (VPR) is fundamental for the global re-localization of robots and devices. Our work introduces a novel structured pruning method to streamline common VPR architectures. This dual focus significantly enhances the efficiency of the system, reducing both map and model memory requirements and decreasing feature extraction and retrieval latencies.
  arXiv  Detail & Related papers  (2024-09-12T08:32:25Z)
• Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach [58.57026686186709]
  We introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR) CFSR inherits the advantages of both convolution-based and transformer-based approaches. Experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance.
  arXiv  Detail & Related papers  (2024-01-11T03:08:00Z)
• BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance [54.214426436283134]
  Deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. We present a strong yet efficient binary neural network for KWS, namely BiFSMNv2, pushing it to the real-network accuracy performance. We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1x speedup and 20.2x storage-saving on edge hardware.
  arXiv  Detail & Related papers  (2022-11-13T18:31:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.