FrameHopper: Selective Processing of Video Frames in Detection-driven Real-Time Video Analytics

• URL: http://arxiv.org/abs/2203.11493v1
• Date: Tue, 22 Mar 2022 07:05:57 GMT
• Title: FrameHopper: Selective Processing of Video Frames in Detection-driven Real-Time Video Analytics
• Authors: Md Adnan Arefeen, Sumaiya Tabassum Nimi, and Md Yusuf Sarwar Uddin
• Abstract summary: Detection-driven real-time video analytics require continuous detection of objects contained in the video frames. Running these detectors on each and every frame in resource-constrained edge devices is computationally intensive. We propose an off-line Reinforcement Learning (RL)-based algorithm to determine these skip-lengths.
• Score: 2.5119455331413376
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detection-driven real-time video analytics require continuous detection of objects contained in the video frames using deep learning models like YOLOV3, EfficientDet. However, running these detectors on each and every frame in resource-constrained edge devices is computationally intensive. By taking the temporal correlation between consecutive video frames into account, we note that detection outputs tend to be overlapping in successive frames. Elimination of similar consecutive frames will lead to a negligible drop in performance while offering significant performance benefits by reducing overall computation and communication costs. The key technical questions are, therefore, (a) how to identify which frames to be processed by the object detector, and (b) how many successive frames can be skipped (called skip-length) once a frame is selected to be processed. The overall goal of the process is to keep the error due to skipping frames as small as possible. We introduce a novel error vs processing rate optimization problem with respect to the object detection task that balances between the error rate and the fraction of frames filtering. Subsequently, we propose an off-line Reinforcement Learning (RL)-based algorithm to determine these skip-lengths as a state-action policy of the RL agent from a recorded video and then deploy the agent online for live video streams. To this end, we develop FrameHopper, an edge-cloud collaborative video analytics framework, that runs a lightweight trained RL agent on the camera and passes filtered frames to the server where the object detection model runs for a set of applications. We have tested our approach on a number of live videos captured from real-life scenarios and show that FrameHopper processes only a handful of frames but produces detection results closer to the oracle solution and outperforms recent state-of-the-art solutions in most cases.

Related papers

• Practical Video Object Detection via Feature Selection and Aggregation [18.15061460125668]
  Video object detection (VOD) needs to concern the high across-frame variation in object appearance, and the diverse deterioration in some frames. Most of contemporary aggregation methods are tailored for two-stage detectors, suffering from high computational costs. This study invents a very simple yet potent strategy of feature selection and aggregation, gaining significant accuracy at marginal computational expense.
  arXiv  Detail & Related papers  (2024-07-29T02:12:11Z)
• Tracking by Associating Clips [110.08925274049409]
  In this paper, we investigate an alternative by treating object association as clip-wise matching. Our new perspective views a single long video sequence as multiple short clips, and then the tracking is performed both within and between the clips. The benefits of this new approach are two folds. First, our method is robust to tracking error accumulation or propagation, as the video chunking allows bypassing the interrupted frames. Second, the multiple frame information is aggregated during the clip-wise matching, resulting in a more accurate long-range track association than the current frame-wise matching.
  arXiv  Detail & Related papers  (2022-12-20T10:33:17Z)
• Look at Adjacent Frames: Video Anomaly Detection without Offline Training [21.334952965297667]
  We propose a solution to detect anomalous events in videos without the need to train a model offline. Specifically, our solution is based on a randomly-d multilayer perceptron that is optimized online to reconstruct video frames, pixel-by-pixel, from their frequency information. An incremental learner is used to update parameters of the multilayer perceptron after observing each frame, thus allowing to detect anomalous events along the video stream.
  arXiv  Detail & Related papers  (2022-07-27T21:18:58Z)
• Distortion-Aware Network Pruning and Feature Reuse for Real-time Video Segmentation [49.17930380106643]
  We propose a novel framework to speed up any architecture with skip-connections for real-time vision tasks. Specifically, at the arrival of each frame, we transform the features from the previous frame to reuse them at specific spatial bins. We then perform partial computation of the backbone network on the regions of the current frame that captures temporal differences between the current and previous frame.
  arXiv  Detail & Related papers  (2022-06-20T07:20:02Z)
• Learning Trajectory-Aware Transformer for Video Super-Resolution [50.49396123016185]
  Video super-resolution aims to restore a sequence of high-resolution (HR) frames from their low-resolution (LR) counterparts. Existing approaches usually align and aggregate video frames from limited adjacent frames. We propose a novel Transformer for Video Super-Resolution (TTVSR)
  arXiv  Detail & Related papers  (2022-04-08T03:37:39Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• OCSampler: Compressing Videos to One Clip with Single-step Sampling [82.0417131211353]
  We propose a framework named OCSampler to explore a compact yet effective video representation with one short clip. Our basic motivation is that the efficient video recognition task lies in processing a whole sequence at once rather than picking up frames sequentially.
  arXiv  Detail & Related papers  (2022-01-12T09:50:38Z)
• Parallel Detection for Efficient Video Analytics at the Edge [5.547133811014004]
  Deep Neural Network (DNN) trained object detectors are widely deployed in mission-critical systems for real time video analytics at the edge. A common performance requirement in mission-critical edge services is the near real-time latency of online object detection on edge devices. This paper addresses these problems by exploiting multi-model multi-device detection parallelism for fast object detection in edge systems.
  arXiv  Detail & Related papers  (2021-07-27T02:50:46Z)
• Temporal Early Exits for Efficient Video Object Detection [1.1470070927586016]
  We propose temporal early exits to reduce the computational complexity of per-frame video object detection. Our method significantly reduces the computational complexity and execution of per-frame video object detection up to $34 times$ compared to existing methods.
  arXiv  Detail & Related papers  (2021-06-21T15:49:46Z)
• Efficient Semantic Video Segmentation with Per-frame Inference [117.97423110566963]
  In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed.
  arXiv  Detail & Related papers  (2020-02-26T12:24:32Z)
• Pack and Detect: Fast Object Detection in Videos Using Region-of-Interest Packing [15.162117090697006]
  We propose Pack and Detect, an approach to reduce the computational requirements of object detection in videos. Experiments using the ImageNet video object detection dataset indicate that PaD can potentially reduce the number of FLOPS required for a frame by $4times$.
  arXiv  Detail & Related papers  (2018-09-05T19:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.