Enabling Fully Integer-Only Inference for Lightweight Detection Transformers

• URL: http://arxiv.org/abs/2607.24981v1
• Date: Mon, 27 Jul 2026 18:33:18 GMT
• Title: Enabling Fully Integer-Only Inference for Lightweight Detection Transformers
• Abstract summary: Key components, including deformable attention, feature fusion, and nonlinear activation functions, are not compatible with integer arithmetic.<n>We develop I-LW-DETR, the first fully integer-only lightweight DETR.<n>I-LW-DETR is built upon three key components: a scale-preserving split convolution that assigns independent activation scale to each branch of the multi-scale projector; SD-ShiftGELU, a sign-dependent GELU approximation that preserves element-wise behavior while avoiding the accuracy degradation; and a constrained Shiftmax that maintains stable Softmax normalization.
• Score: 0.19116784879310025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformer detectors now approach the accuracy of CNNs but remain difficult to deploy on NPUs and microcontrollers because key components, including deformable attention, feature fusion, and nonlinear activation functions, are not natively compatible with integer arithmetic. Existing quantized detectors either retain operators such as Softmax, GELU, and LayerNorm or focus on heavyweight backbones, leaving lightweight detection transformers without an end-to-end integer implementation. We address this gap with I-LW-DETR, the first fully integer-only lightweight DETR, in which every operation in the forward pass, including transformer nonlinearities, is executed in integer arithmetic. I-LW-DETR is built upon three key components: a scale-preserving split convolution that assigns independent activation scale to each branch of the multi-scale projector; SD-ShiftGELU, a sign-dependent GELU approximation that preserves element-wise behavior while avoiding the accuracy degradation; and a constrained Shiftmax that maintains stable Softmax normalization. Experimental results demonstrate that the proposed quantization pipeline consistently produces efficient fully integer-only models across different model scales. Across all model scales, the proposed pipeline incurs only a moderate accuracy degradation while reducing the model size by approximately $3.6\times$ and the computational cost by more than one order of magnitude.

Related papers

• Low-Rank Ternary Adaptation for Fine-Tuning Transformers [54.3230438745856]
  Ternary transformers offer extreme memory and compute efficiency.<n>Existing low-bit LoRA-based methods cannot directly fine-tune ternary weights.<n>We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights.
  arXiv  Detail & Related papers  (2026-08-25T12:15:56Z)
• Sparse Weight Decomposition for Efficient Circuit Extraction [36.87892448834478]
  Dense pretrained transformers do not naturally expose interpretable units for circuit extraction.<n>We propose Sparse Weight Decomposition, which factorizes pretrained projections by factorizing each weight matrix into two sparse factors.<n>We show that SWD remains effective for full-model replacement of all attention and weight matrices after fine-tuning the nonzero factor values.
  arXiv  Detail & Related papers  (2026-08-04T16:40:48Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• Quantized Visual Geometry Grounded Transformer [67.15451442018258]
  This paper proposes the first Quantization framework for VGGTs, namely QuantVGGT.<n>We introduce Dual-Smoothed Fine-Grained Quantization, which integrates pre-global Hadamard rotation and post-local channel smoothing.<n>We also design Noise-Filtered Diverse Sampling, which filters outliers via deep-layer statistics.
  arXiv  Detail & Related papers  (2025-09-25T15:17:11Z)
• I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentation [0.0]
  Quantization offers an effective strategy to improve efficiency, but ViT-based segmentation models are notoriously fragile under low precision.<n>We introduce I-Segmenter, the first fully integer-only ViT segmentation framework.<n>I-Segmenter achieves competitive accuracy even in one-shot PTQ with a single calibration image.
  arXiv  Detail & Related papers  (2025-09-12T15:14:19Z)
• Kolmogorov-Arnold Transformer [72.88137795439407]
  We introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces layers with Kolmogorov-Arnold Network (KAN) layers. We identify three key challenges: (C1) Base function, (C2) Inefficiency, and (C3) Weight. With these designs, KAT outperforms traditional-based transformers.
  arXiv  Detail & Related papers  (2024-09-16T17:54:51Z)
• I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models [20.070306492164427]
  Post-training quantization serves as a potent technique to accelerate the inference of large language models. Existing works still necessitate a considerable number of floating-point (FP) operations during inference. This limitation hinders the deployment of large language models on the edge and cloud devices. We propose I-LLM, a novel integer-only fully-quantized PTQ framework tailored for large language models.
  arXiv  Detail & Related papers  (2024-05-28T05:56:11Z)
• Sparse Binary Transformers for Multivariate Time Series Modeling [1.3965477771846404]
  We show that lightweight Compressed Neural Networks can achieve accuracy comparable to dense floating-point Transformers. Our model achieves favorable results across three time series learning tasks: classification, anomaly detection, and single-step forecasting. We measure the computational savings of our approach over a range of metrics including parameter count, bit size, and floating point operation (FLOPs) count.
  arXiv  Detail & Related papers  (2023-08-09T00:23:04Z)
• I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference [3.067607520161916]
  Vision Transformers (ViTs) have achieved state-of-the-art performance on various computer vision applications. These models have considerable storage and computational overheads, making their deployment and efficient inference on edge devices challenging. We propose I-ViT, an integer-only quantization scheme for ViTs, to enable ViTs to perform the entire computational graph of inference with integer arithmetic and bit-shifting.
  arXiv  Detail & Related papers  (2022-07-04T13:37:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.