DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos
- URL: http://arxiv.org/abs/2203.03996v2
- Date: Sat, 2 Sep 2023 07:01:11 GMT
- Title: DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos
- Authors: Mathias Parger, Chengcheng Tang, Christopher D. Twigg, Cem Keskin,
Robert Wang, Markus Steinberger
- Abstract summary: Convolutional neural network inference on video data requires powerful hardware for real-time processing.
We present a sparse convolutional neural network framework that enables sparse frame-by-frame updates.
We are the first to significantly outperform the dense reference, cuDNN, in practical settings, achieving speedups of up to 7x with only marginal differences in accuracy.
- Score: 16.644938608211202
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Convolutional neural network inference on video data requires powerful
hardware for real-time processing. Given the inherent coherence across
consecutive frames, large parts of a video typically change little. By skipping
identical image regions and truncating insignificant pixel updates,
computational redundancy can in theory be reduced significantly. However, these
theoretical savings have been difficult to translate into practice, as sparse
updates hamper computational consistency and memory access coherence; which are
key for efficiency on real hardware. With DeltaCNN, we present a sparse
convolutional neural network framework that enables sparse frame-by-frame
updates to accelerate video inference in practice. We provide sparse
implementations for all typical CNN layers and propagate sparse feature updates
end-to-end - without accumulating errors over time. DeltaCNN is applicable to
all convolutional neural networks without retraining. To the best of our
knowledge, we are the first to significantly outperform the dense reference,
cuDNN, in practical settings, achieving speedups of up to 7x with only marginal
differences in accuracy.
Related papers
- SparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity [15.872209884833977]
We propose a memory-efficient scheduling method to eliminate memory overhead and an online adjustment mechanism to minimize accuracy degradation.
SparseTem achieves speedup of 1.79x for EfficientDet and 4.72x for CRNN, with minimal accuracy drop and no additional memory overhead.
arXiv Detail & Related papers (2024-10-28T07:13:25Z) - A Converting Autoencoder Toward Low-latency and Energy-efficient DNN
Inference at the Edge [4.11949030493552]
We present CBNet, a low-latency and energy-efficient deep neural network (DNN) inference framework tailored for edge devices.
It utilizes a "converting" autoencoder to efficiently transform hard images into easy ones.
CBNet achieves up to 4.8x speedup in inference latency and 79% reduction in energy usage compared to competing techniques.
arXiv Detail & Related papers (2024-03-11T08:13:42Z) - Compressing CNN Kernels for Videos Using Tucker Decompositions: Towards
Lightweight CNN Applications [2.191505742658975]
Convolutional Neural Networks (CNN) are the state-of-theart in the field of visual computing.
A major problem with CNNs is the large number of floating point operations (FLOPs) required to perform convolutions for large inputs.
We propose a Tuckerdecomposition to compress the convolutional kernel of a pre-trained network for images.
arXiv Detail & Related papers (2022-03-10T11:53:53Z) - Instant Neural Graphics Primitives with a Multiresolution Hash Encoding [67.33850633281803]
We present a versatile new input encoding that permits the use of a smaller network without sacrificing quality.
A small neural network is augmented by a multiresolution hash table of trainable feature vectors whose values are optimized through a gradient descent.
We achieve a combined speed of several orders of magnitude, enabling training of high-quality neural graphics primitives in a matter of seconds.
arXiv Detail & Related papers (2022-01-16T07:22:47Z) - Content-Aware Convolutional Neural Networks [98.97634685964819]
Convolutional Neural Networks (CNNs) have achieved great success due to the powerful feature learning ability of convolution layers.
We propose a Content-aware Convolution (CAC) that automatically detects the smooth windows and applies a 1x1 convolutional kernel to replace the original large kernel.
arXiv Detail & Related papers (2021-06-30T03:54:35Z) - Continual 3D Convolutional Neural Networks for Real-time Processing of
Videos [93.73198973454944]
We introduce Continual 3D Contemporalal Neural Networks (Co3D CNNs)
Co3D CNNs process videos frame-by-frame rather than by clip by clip.
We show that Co3D CNNs initialised on the weights from preexisting state-of-the-art video recognition models reduce floating point operations for frame-wise computations by 10.0-12.4x while improving accuracy on Kinetics-400 by 2.3-3.8x.
arXiv Detail & Related papers (2021-05-31T18:30:52Z) - MoViNets: Mobile Video Networks for Efficient Video Recognition [52.49314494202433]
3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets.
We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs.
arXiv Detail & Related papers (2021-03-21T23:06:38Z) - SparseDNN: Fast Sparse Deep Learning Inference on CPUs [1.6244541005112747]
We present SparseDNN, a sparse deep learning inference engine targeting CPUs.
We show that our sparse code generator can achieve significant speedups over state-of-the-art sparse and dense libraries.
arXiv Detail & Related papers (2021-01-20T03:27:35Z) - Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization.
Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
arXiv Detail & Related papers (2020-06-21T08:23:03Z) - Approximation and Non-parametric Estimation of ResNet-type Convolutional
Neural Networks [52.972605601174955]
We show a ResNet-type CNN can attain the minimax optimal error rates in important function classes.
We derive approximation and estimation error rates of the aformentioned type of CNNs for the Barron and H"older classes.
arXiv Detail & Related papers (2019-03-24T19:42:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.