The Rescue Effect: Spatio-Semantic Early Exit Bypasses Quantization Collapse in CLIP

• URL: http://arxiv.org/abs/2605.26415v1
• Date: Tue, 26 May 2026 00:45:46 GMT
• Title: The Rescue Effect: Spatio-Semantic Early Exit Bypasses Quantization Collapse in CLIP
• Abstract summary: In joint-embedding architectures such as CLIP this introduces a failure mode distinct from quantized CNN classifiers.<n>We characterize this as Quantization-Induced Representation Collapse (QIRC) and quantify it on INT8 replacing CLIP ViT-B/32.<n>We propose LRA-EE (Layer-wise Representation-Aware Early Exit), which bypasses noise-saturated deep layers.<n>On ImageNet-1K zero-shot classification, LRA-EE reduces FLOPs by 13.4% and improves Top-1 accuracy by +2.44%p (58.72% -> 61.16%) over the INT8 baseline
• Score: 6.450516062706025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying Vision-Language Models on resource-constrained hardware typically requires INT8 quantization, but in joint-embedding architectures such as CLIP this introduces a failure mode distinct from quantized CNN classifiers: activation noise accumulated across transformer blocks perturbs the direction of the multimodal embedding, eroding the cosine alignment on which zero-shot retrieval depends. We characterize this as Quantization-Induced Representation Collapse (QIRC) and quantify it on INT8 CLIP ViT-B/32, where the layer-wise noise-to-signal ratio grows from below 10% in shallow blocks to 52% at Layer 11. We propose LRA-EE (Layer-wise Representation-Aware Early Exit), which bypasses noise-saturated deep layers via Spatio-Semantic Aggregation (replacing the immature shallow [CLS] with a global patch-token average), a learned multi-feature gate (confidence, top-2 margin, spatial-activation variance), and Layer-adaptive Confidence Thresholding calibrated to each layer's Information-to-Noise Ratio. On ImageNet-1K zero-shot classification, LRA-EE reduces FLOPs by 13.4% and improves Top-1 accuracy by +2.44%p (58.72% -> 61.16%) over the INT8 baseline. A four-quadrant decomposition isolates the Rescue Effect: 9.5% of samples are correctly classified at shallow exits but lost to noise at full depth, against only 7.1% suffering the inverse.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks [0.20313422213594776]
  We show that smooth two-layer feed-forward networks (FFNs) expose an additional structural model extraction channel.<n>We exploit a second-order leakage channel in which projected input Hessians form different mixtures of the same hidden symmetric rank-one factors.<n>On independently trained CIFAR-10 vision transformers, only 16 projected Hessians, corresponding to 8193 black-box queries, recover the hidden FFN directions.
  arXiv  Detail & Related papers  (2026-08-28T20:26:07Z)
• SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs [18.494055578192015]
  We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter at each pruning site.<n>Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction.<n>Experiments on five model families, six layer-selection criteria, and seven zero-shot benchmarks show that SHIFT-LLM consistently recovers accuracy lost to depth pruning.
  arXiv  Detail & Related papers  (2026-08-25T19:01:58Z)
• Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale [17.610398782149293]
  In vision models, where spatial correlation concentrates gradient energy into a low-rank subspace, most of this noise falls in directions that carry little signal.<n> Spectral gradient orthogonalization via polar decomposition is introduced as a post-processing step.<n>It improves accuracy only when the per-direction spectral signal-to-noise ratio (SNR) suffices for singular vector recovery.
  arXiv  Detail & Related papers  (2026-08-18T06:35:26Z)
• Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift [3.704942101286518]
  We introduce PRISM, a training-free, source-free Test-Time Adaptation (TTA) framework grounded in the Affine Noise Hypothesis.<n>At inference, adaptation reduces to one matrix-vector multiplication in 0.0009 ms, making it substantially faster than gradient-based TTA.<n>On UrbanSound8K, PRISM improves over the zero-shot baseline by 12.94 percentage points and surpasses an oracle-assisted TTA baseline by 9.41 percentage points.
  arXiv  Detail & Related papers  (2026-08-15T04:42:14Z)
• Cross-Layer Error Compensation and Finite-Sample Feature-Statistics Matching for Extreme Low-Bit Quantization of Large Language Models [0.0]
  We formulate quantization as a joint optimization over the discrete codes and scales of all layers.<n>We show that error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher.<n>On Qwen2.5-1.5B with 1.125-bit group-binary weights, error compensation alone reaches a perplexity ratio of 9.56 +/- 0.15 over the FP16 teacher.
  arXiv  Detail & Related papers  (2026-07-16T06:47:52Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Attention Sinks and Outliers in Attention Residuals [39.45624457227565]
  We propose an outlier- and sink-aware technique built on inter-layer null signaling.<n>AttnResidual architectures introduce an additional depth-wise normalization channel.<n>We show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness.
  arXiv  Detail & Related papers  (2026-05-18T05:53:09Z)
• Layer Collapse in Diffusion Language Models [54.880703002010144]
  Diffusion language models (DLMs) have emerged as competitive alternatives to autoregressive (AR) language models.<n>We show that layer collapse in DLMs is not driven by undertraining but by overtraining.<n>Our findings have strong practical implications.
  arXiv  Detail & Related papers  (2026-05-07T14:39:40Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference [2.954681536906518]
  KV caching is a ubiquitous optimization in autoregressive transformer inference.<n>We show that FP16 KV cache inference is fundamentally non-equivalent to recomputation.
  arXiv  Detail & Related papers  (2026-04-16T15:59:40Z)
• Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking [40.652380579951206]
  In-Context Re-ranking (ICR) has recently been proposed as an $O(1)$ alternative method.<n>ICR extracts internal attention signals directly, avoiding the overhead of text generation.<n>No unified study has compared internal attention with traditional generative and likelihood-based mechanisms.
  arXiv  Detail & Related papers  (2026-02-26T03:51:31Z)
• Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks [0.0]
  We present a practical system for privacy-aware large language model (LLM) inference that splits a transformer between a trusted local GPU and a cloud GPU.<n>Our system addresses the unique challenges of autoregressive LLM decoding over high-latency wide-area networks (WANs)<n>System generalizes to larger models with only 4.9 GB local VRAM and matching 7B throughput.
  arXiv  Detail & Related papers  (2026-02-18T14:13:08Z)
• Ultrafast Deep Learning-Based Scatter Estimation in Cone-Beam Computed Tomography [7.864992877255044]
  scatter artifacts drastically degrade the image quality of cone-beam computed tomography (CBCT) scans.<n>Deep learning-based methods show promise in estimating scatter from CBCT measurements.<n>Their deployment in mobile CBCT systems or edge devices is still limited due to the large memory footprint of the networks.
  arXiv  Detail & Related papers  (2025-09-10T20:07:56Z)
• Robust Residual Finite Scalar Quantization for Neural Compression [46.574899938569125]
  Finite Scalar Quantization (FSQ) offers simplified training but suffers from residual magnitude decay in multi-stage settings.<n>We propose Robust Residual Finite Scalar Quantization (RFSQ), addressing this fundamental limitation through two novel conditioning strategies.<n>Our experiments across audio and image modalities demonstrate RFSQ's effectiveness and generalizability.
  arXiv  Detail & Related papers  (2025-08-20T15:18:59Z)
• Sub-token ViT Embedding via Stochastic Resonance Transformers [51.12001699637727]
  Vision Transformer (ViT) architectures represent images as collections of high-dimensional vectorized tokens, each corresponding to a rectangular non-overlapping patch. We propose a training-free method inspired by "stochastic resonance" The resulting "Stochastic Resonance Transformer" (SRT) retains the rich semantic information of the original representation, but grounds it on a finer-scale spatial domain, partly mitigating the coarse effect of spatial tokenization.
  arXiv  Detail & Related papers  (2023-10-06T01:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.