CARB: A Characterization-Guided Framework for CNN Inference Cost Prediction and Deployment Screening

• URL: http://arxiv.org/abs/2608.10506v1
• Date: Tue, 11 Aug 2026 05:27:53 GMT
• Title: CARB: A Characterization-Guided Framework for CNN Inference Cost Prediction and Deployment Screening
• Abstract summary: Existing approaches rely on FLOPs, latency measurements, or single-device profiling as energy proxies.<n>We present a workload characterization study of 13 419 CNN configurations on two GPU platforms.<n>We develop CARB, a cascade-blended ensemble that jointly predicts all three targets with R2 0.99, and a two-stage deployment screening workflow that eliminates over 90% of candidates in seconds.
• Score: 3.4577090606397163
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate pre-deployment estimation of CNN inference cost--energy, latency, and peak memory--is increasingly critical as models are deployed on resource-constrained GPU platforms. Existing approaches rely on FLOPs, latency measurements, or single-device profiling as energy proxies, overlooking the non-linear interactions between architectural design and hardware load. We present a workload characterization study of 13 419 CNN configurations on two GPU platforms (RTX 5090 and RTX 3080) under GPU telemetry, revealing that energy, latency, and memory exhibit fundamentally distinct scaling behaviors: energy and latency diverge by 3x under high computational demand, and cross-GPU transferability differs by target--energy and latency require platform-specific models while memory transfers well across the two tested platforms. Building on these characterization findings, we develop CARB, a cascade-blended ensemble that jointly predicts all three targets with R2 ~0.99, and a two-stage deployment screening workflow that eliminates over 90% of candidates in seconds, reducing large design spaces to a Pareto-prioritized shortlist validated against real hardware.

Related papers

• HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications [2.050075597129042]
  Foundation models are increasingly deployed on embedded and edge platforms for CPS and IoT applications.<n>Existing benchmarking tools evaluate model capability in isolation, reporting accuracy assuming sufficient compute.<n>We present HoliBench, a modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy across platforms.
  arXiv  Detail & Related papers  (2026-09-11T04:03:20Z)
• CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems [2.870762512009438]
  We introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems.<n>As a latency-oriented spatial scaling technique, CTA-pipelining leverages the Cooperative Thread Array level.<n>Results show CTA-pipelining reduces latency by up to 31.8% compared to micro-batching.
  arXiv  Detail & Related papers  (2026-07-08T18:54:08Z)
• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference [0.0]
  This paper presents a systematic empirical analysis of the Nvidia and Apple Silicon ecosystems.<n>We characterize the distinct intra-architecture trade-offs required to deploy massive models.<n>We conclude that for consumer-grade inference, the optimal hardware is defined by a complex interplay between compute density (Nvidia) and memory capacity (Apple)
  arXiv  Detail & Related papers  (2026-05-01T08:45:47Z)
• Scalable Generative Game Engine: Breaking the Resolution Wall via Hardware-Algorithm Co-Design [17.941176878609337]
  We bridge the gap between generative models and high-resolution neural simulations by introducing a scalable textitHardware-Algorithm Co-Design framework.<n>Our system delivers fluid 26.4 FPS and 48.3 FPS respectively, with an amortized effective latency of 2.7 ms.
  arXiv  Detail & Related papers  (2026-01-31T08:52:51Z)
• Energy Scaling Laws for Diffusion Models: Quantifying Compute and Carbon Emissions in Image Generation [50.21021246855702]
  We propose an adaptation of Kaplan scaling laws to predict GPU energy consumption for diffusion models based on computational complexity (FLOPs)<n>Our approach decomposes diffusion model inference into text encoding, iterative denoising, and decoding components, with the hypothesis that denoising operations dominate energy consumption due to their repeated execution across multiple inference steps.<n>Our results validate the compute-bound nature of diffusion inference and provide a foundation for sustainable AI deployment planning and carbon footprint estimation.
  arXiv  Detail & Related papers  (2025-11-21T08:12:47Z)
• Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity [39.483346492111515]
  Linear recurrent neural networks enable powerful long-range sequence modeling with constant memory usage and time-per-token during inference.<n>Unstructured sparsity offers a compelling solution, enabling substantial reductions in compute and memory requirements when accelerated by compatible hardware platforms.<n>We find that highly sparse linear RNNs consistently achieve better efficiency-performance trade-offs than dense baselines.
  arXiv  Detail & Related papers  (2025-02-03T13:09:21Z)
• EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction [67.11722682878722]
  This work presents EfficientViT, a new family of high-resolution vision models with novel multi-scale linear attention. Our multi-scale linear attention achieves the global receptive field and multi-scale learning. EfficientViT delivers remarkable performance gains over previous state-of-the-art models.
  arXiv  Detail & Related papers  (2022-05-29T20:07:23Z)
• Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining [58.10436813430554]
  Mini-batch training of graph neural networks (GNNs) requires a lot of computation and data movement. We argue in favor of performing mini-batch training with neighborhood sampling in a distributed multi-GPU environment. We present a sequence of improvements to mitigate these bottlenecks, including a performance-engineered neighborhood sampler. We also conduct an empirical analysis that supports the use of sampling for inference, showing that test accuracies are not materially compromised.
  arXiv  Detail & Related papers  (2021-10-16T02:41:35Z)
• Multi-Exit Semantic Segmentation Networks [78.44441236864057]
  We propose a framework for converting state-of-the-art segmentation models to MESS networks. specially trained CNNs that employ parametrised early exits along their depth to save during inference on easier samples. We co-optimise the number, placement and architecture of the attached segmentation heads, along with the exit policy, to adapt to the device capabilities and application-specific requirements.
  arXiv  Detail & Related papers  (2021-06-07T11:37:03Z)
• FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation [81.76975488010213]
  Dense optical flow estimation plays a key role in many robotic vision tasks. Current networks often occupy large number of parameters and require heavy computation costs. Our proposed FastFlowNet works in the well-known coarse-to-fine manner with following innovations.
  arXiv  Detail & Related papers  (2021-03-08T03:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.