Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment

• URL: http://arxiv.org/abs/2606.24173v1
• Date: Tue, 23 Jun 2026 05:54:33 GMT
• Title: Lightweight Transformer Models for On-Device Fault Detection: A Benchmark Study on Resource-Constrained Deployment
• Abstract summary: On-device fault detection enables real-time diagnostics without cloud dependency, but deploying machine learning models on resource-constrained hardware demands careful tradeoffs between accuracy, latency, and model size.<n>We present a benchmark comparing traditional ML methods against lightweight transformer architectures for binary fault detection.<n>Our results reveal that on well-separated sensor data, lightweight transformers match traditional ML at 87.8% F1 but at 100x the model size and 9000x the latency.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-device fault detection enables real-time diagnostics without cloud dependency, but deploying machine learning models on resource-constrained hardware demands careful tradeoffs between accuracy, latency, and model size. We present a benchmark comparing traditional ML methods (Random Forest, XGBoost, SVM, Logistic Regression) against lightweight transformer architectures (DistilBERT, TinyBERT-6L, TinyBERT-4L, MobileBERT) for binary fault detection across three public datasets: NASA C-MAPSS turbofan degradation, SECOM semiconductor manufacturing, and UCI AI4I 2020 predictive maintenance. We evaluate classification performance (F1-score, AUC), model size, and CPU inference latency, and further assess INT8 dynamic quantization and a two-stage adaptive inference pipeline. Our results reveal that on well-separated sensor data (C-MAPSS), lightweight transformers match traditional ML at 87.8% F1 but at 100x the model size and 9000x the latency. TinyBERT-4L emerges as the most deployment-friendly transformer at 55 MB and 18 ms CPU latency. INT8 quantization reduces size by 25% while preserving 86.9% F1. Our adaptive pipeline, routing 97.9% of predictions through a quantized triage model and only 2.1% to a larger expert, achieves 87.6% F1 at 19.5 ms average latency. On severely imbalanced datasets (SECOM, UCI-PM), both traditional and transformer methods struggle significantly, highlighting fundamental limitations of current approaches for extreme class imbalance in fault detection. All code is publicly available.

Related papers

• Multi-Stage Mamba-Based Architecture for Fast and Scalable Superconducting Qubit Readout [2.7180447022087932]
  In superconducting quantum processors, readout operations are both error-prone and high-latency.<n>We present a multi-stage qubit state discriminator based on the Mamba model.<n>Our model achieves a geometric mean readout fidelity of 0.906, outperforming the best-reported state-of-the-art method.
  arXiv  Detail & Related papers  (2026-07-13T15:12:07Z)
• MicroBi-ConvLSTM: An Ultra-Lightweight Efficient Model for Human Activity Recognition on Resource Constrained Devices [0.0]
  Human Activity Recognition (HAR) on resource constrained wearables requires models that balance accuracy against strict memory and computational budgets.<n>We present MicroBi-ConvLSTM, an ultra-lightweight convolutional-recurrent architecture achieving 11.4K parameters on average.
  arXiv  Detail & Related papers  (2026-02-06T09:26:29Z)
• LiQSS: Post-Transformer Linear Quantum-Inspired State-Space Tensor Networks for Real-Time 6G [85.58816960936069]
  Proactive and agentic control in Sixth-Generation (6G) Open Radio Access Networks (O-RAN) requires control-grade prediction under stringent Near-Time (Near-RT) latency and computational constraints.<n>This paper investigates a post-Transformer paradigm for efficient radio telemetry forecasting.<n>We propose a quantum-inspired state-space tensor network that replaces self-attention with stable structured state-space dynamics kernels.
  arXiv  Detail & Related papers  (2026-01-18T12:08:38Z)
• Lightweight Transformer Architectures for Edge Devices in Real-Time Applications [0.0]
  This survey examines lightweight transformer architectures specifically designed for edge deployment.<n>We systematically review prominent lightweight variants including MobileBERT, TinyBERT, DistilBERT, EfficientFormer, EdgeFormer, and MobileViT.<n> Experimental results demonstrate that modern lightweight transformers can achieve 75-96% of full-model accuracy while reducing model size by 4-10x and inference latency by 3-9x.
  arXiv  Detail & Related papers  (2026-01-05T01:04:25Z)
• Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts [6.221156050218661]
  We present a curiosity-driven quantized Mixture-of-Experts framework for deep neural networks on resource-constrained devices.<n>Our 4-bit quantization maintains 99.9 percent of 16-bit accuracy (0.858 vs 0.859 F1) with 4x compression and 41 percent energy savings.<n>Our information-theoretic routing demonstrates that adaptive quantization yields accurate (0.858 F1, 1.2M params), energy-efficient (3.87 F1/mJ), and predictable edge models.
  arXiv  Detail & Related papers  (2025-11-13T15:32:41Z)
• Automated Energy-Aware Time-Series Model Deployment on Embedded FPGAs for Resilient Combined Sewer Overflow Management [17.903318666906728]
  Extreme weather events, intensified by climate change, increasingly challenge aging combined sewer systems.<n>Forecasting of sewer overflow basin filling levels can provide actionable insights for early intervention.<n>We propose an end-to-end forecasting framework that enables energy-efficient inference directly on edge devices.
  arXiv  Detail & Related papers  (2025-08-19T15:06:04Z)
• Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning [73.73967342609603]
  We introduce a predictor-corrector learning framework to minimize truncation errors. We also propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor. Our model surpasses a robust 3.8B DeepNet by an average of 2.9 SacreBLEU, using only 1/3 parameters.
  arXiv  Detail & Related papers  (2024-11-05T12:26:25Z)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models [76.64055251296548]
  We show how to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources.<n>The resulting hybrid model achieves performance comparable to the original Transformer in chat benchmarks.<n>We also introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models.
  arXiv  Detail & Related papers  (2024-08-27T17:56:11Z)
• SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation [53.675725490807615]
  We introduce SDPose, a new self-distillation method for improving the performance of small transformer-based models. SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs, while SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset.
  arXiv  Detail & Related papers  (2024-04-04T15:23:14Z)
• Transformer Multivariate Forecasting: Less is More? [42.558736426375056]
  The paper focuses on reducing redundant information to elevate forecasting accuracy while optimizing runtime efficiency. The framework is evaluated by five state-of-the-art (SOTA) models and four diverse real-world datasets. From the model perspective, one of the PCA-enhanced models: PCA+Crossformer, reduces mean square errors (MSE) by 33.3% and decreases runtime by 49.2% on average.
  arXiv  Detail & Related papers  (2023-12-30T13:44:23Z)
• Global Vision Transformer Pruning with Hessian-Aware Saliency [93.33895899995224]
  This work challenges the common design philosophy of the Vision Transformer (ViT) model with uniform dimension across all the stacked blocks in a model stage. We derive a novel Hessian-based structural pruning criteria comparable across all layers and structures, with latency-aware regularization for direct latency reduction. Performing iterative pruning on the DeiT-Base model leads to a new architecture family called NViT (Novel ViT), with a novel parameter that utilizes parameters more efficiently.
  arXiv  Detail & Related papers  (2021-10-10T18:04:59Z)
• CoDeNet: Efficient Deployment of Input-Adaptive Object Detection on Embedded FPGAs [41.43273142203345]
  We harness the flexibility of FPGAs to develop a novel object detection pipeline with deformable convolutions. With our high-efficiency implementation, our solution reaches 26.9 frames per second with a tiny model size of 0.76 MB. Our model gets to 67.1 AP50 on Pascal VOC with only 2.9 MB of parameters-20.9x smaller but 10% more accurate than Tiny-YOLO.
  arXiv  Detail & Related papers  (2020-06-12T17:56:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.