QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring

• URL: http://arxiv.org/abs/2404.12599v2
• Date: Sat, 16 Nov 2024 06:34:58 GMT
• Title: QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring
• Abstract summary: Uncertainty quantification (UQ) provides a resource-efficient solution for on-device monitoring of tinyML models deployed without access to true labels. We propose QUTE, a novel resource-efficient early-exit-assisted ensemble architecture for optimized tinyML models. We show that QUTE delivers superior uncertainty quality on tiny models, achieving comparable performance on larger models with 59% smaller model sizes than the closest prior work.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Uncertainty quantification (UQ) provides a resource-efficient solution for on-device monitoring of tinyML models deployed without access to true labels. However, existing UQ methods impose significant memory and compute demands, making them impractical for ultra-low-power, KB-sized TinyML devices. Prior work has attempted to reduce overhead by using early-exit ensembles to quantify uncertainty in a single forward pass, but these approaches still carry prohibitive costs. To address this, we propose QUTE, a novel resource-efficient early-exit-assisted ensemble architecture optimized for tinyML models. QUTE introduces additional output blocks at the final exit of the base network, distilling early-exit knowledge into these blocks to form a diverse yet lightweight ensemble. We show that QUTE delivers superior uncertainty quality on tiny models, achieving comparable performance on larger models with 59% smaller model sizes than the closest prior work. When deployed on a microcontroller, QUTE demonstrates a 31% reduction in latency on average. In addition, we show that QUTE excels at detecting accuracy-drop events, outperforming all prior works.

Related papers

• LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs [52.1276403258812]
  We introduce Logit-aware Final-block Quantization (LFQ), a simple yet effective enhancement to block-wise PTQ.<n>LFQ consistently improves the accuracy of complex generation tasks over state-of-the-art block-wise PTQ.
  arXiv  Detail & Related papers  (2026-05-28T11:02:23Z)
• One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking [7.856998585396422]
  Large language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines.<n>We propose textbfmulti-task learning (MTL) as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly.
  arXiv  Detail & Related papers  (2026-01-16T13:44:25Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• SNAP-UQ: Self-supervised Next-Activation Prediction for Single-Pass Uncertainty in TinyML [0.0]
  We introduce textbfSNAP-UQ, a single-pass, label-free uncertainty method for TinyML.<n>It estimates risk from emphdepth-wise next-activation prediction: tiny int8 heads forecast the statistics of the next layer from a compressed view of the previous one.<n>The design requires no temporal buffers, auxiliary exits, or repeated forward passes, and adds only a few tens of kilobytes to MCU deployments.
  arXiv  Detail & Related papers  (2025-08-18T13:14:20Z)
• Collaborative LLM Inference via Planning for Efficient Reasoning [50.04696654679751]
  We propose a test-time collaboration framework in which a planner model first generates a plan, defined as a distilled and high-level abstraction of the problem.<n>Small and large models take turns acting as planner and reasoner, exchanging plans in a multi-round cascade to collaboratively solve complex tasks.<n>Our method achieves accuracy comparable to strong proprietary models alone, while significantly reducing reliance on paid inference.
  arXiv  Detail & Related papers  (2025-06-13T08:35:50Z)
• Harnessing On-Device Large Language Model: Empirical Results and Implications for AI PC [8.837470787975308]
  Large Language Models (LLMs) on edge devices offer significant privacy benefits.<n>These on-device LLMs inherently face performance limitations due to reduced model capacity and necessary compression techniques.<n>We introduce a systematic methodology -- encompassing model capability, development efficiency, and system resources -- for evaluating on-device LLMs.
  arXiv  Detail & Related papers  (2025-05-21T02:23:01Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Predicting Emergent Capabilities by Finetuning [98.9684114851891]
  We find that finetuning language models can shift the point in scaling at which emergence occurs towards less capable models. We validate this approach using four standard NLP benchmarks. We find that, in some cases, we can accurately predict whether models trained with up to 4x more compute have emerged.
  arXiv  Detail & Related papers  (2024-11-25T01:48:09Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• Quantized Transformer Language Model Implementations on Edge Devices [1.2979415757860164]
  Large-scale transformer-based models like the Bidirectional Representations from Transformers (BERT) are widely used for Natural Language Processing (NLP) applications. These models are initially pre-trained with a large corpus with millions of parameters and then fine-tuned for a downstream NLP task. One of the major limitations of these large-scale models is that they cannot be deployed on resource-constrained devices due to their large model size and increased inference latency.
  arXiv  Detail & Related papers  (2023-10-06T01:59:19Z)
• FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs [9.072821427818557]
  Large Language Models (LLMs) have achieved state-of-the-art performance across various language tasks but pose challenges for practical deployment. We propose an efficient weight-only quantization method that reduces memory consumption and accelerates inference for LLMs. We evaluate our approach on large-scale open source models such as OPT-175B and internal MoE models, showcasing minimal accuracy loss while achieving up to 3.65 times higher throughput.
  arXiv  Detail & Related papers  (2023-08-16T23:57:41Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• T-RECX: Tiny-Resource Efficient Convolutional neural networks with early-eXit [0.0]
  We show how an early exit intermediate classifier can be enhanced by the addition of an early exit intermediate classifier. Our technique is optimized specifically for tiny-CNN sized models. Our results show that T-RecX 1) improves the accuracy of baseline network, 2) achieves 31.58% average reduction in FLOPS in exchange for one percent accuracy across all evaluated models.
  arXiv  Detail & Related papers  (2022-07-14T02:05:43Z)
• An Empirical Study of Low Precision Quantization for TinyML [8.939851623894334]
  We focus on post-training quantization (PTQ) algorithms that quantize a model to low-bit (less than 8-bit) precision with only a small set of calibration data. To achieve a fair comparison, we build a simulated quantization framework to investigate recent PTQ algorithms. With ablation study on different alternatives of components in the pipeline, we reveal key design choices when performing low precision quantization.
  arXiv  Detail & Related papers  (2022-03-10T17:22:08Z)
• BSC: Block-based Stochastic Computing to Enable Accurate and Efficient TinyML [10.294484356351152]
  Machine learning (ML) has been successfully applied to edge applications, such as smart phones and automated driving. Today, more applications require ML on tiny devices with extremely limited resources, like implantable cardioverter defibrillator (ICD) which is known as TinyML. Unlike ML on the edge, TinyML with a limited energy supply has higher demands on low-power execution.
  arXiv  Detail & Related papers  (2021-11-12T12:28:05Z)
• A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays [66.62377866022221]
  Latent Replay-based Continual Learning (CL) techniques enable online, serverless adaptation in principle. We introduce a HW/SW platform for end-to-end CL based on a 10-core FP32-enabled parallel ultra-low-power processor. Our results show that by combining these techniques, continual learning can be achieved in practice using less than 64MB of memory.
  arXiv  Detail & Related papers  (2021-10-20T11:01:23Z)
• Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Model [50.38446482252857]
  This paper is concerned with the sample efficiency of reinforcement learning, assuming access to a generative model (or simulator) We first consider $gamma$-discounted infinite-horizon Markov decision processes (MDPs) with state space $mathcalS$ and action space $mathcalA$. We prove that a plain model-based planning algorithm suffices to achieve minimax-optimal sample complexity given any target accuracy level.
  arXiv  Detail & Related papers  (2020-05-26T17:53:18Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.