A Multi-Task Deep Learning Framework for Real-Time Intelligent Video Surveillance with Temporal Event Validation

• URL: http://arxiv.org/abs/2607.03131v1
• Date: Fri, 03 Jul 2026 09:20:58 GMT
• Title: A Multi-Task Deep Learning Framework for Real-Time Intelligent Video Surveillance with Temporal Event Validation
• Abstract summary: This paper presents a unified multi-task deep learning framework that simultaneously performs face recognition, license plate recognition, weapon detection, fire and smoke detection, and human action recognition.<n>To improve robustness in continuous video, all detection modules are integrated into a temporal event-validation architecture.<n>Results show that the framework maintains real-time operation with a per-frame latency below 100 ms on commodity hardware.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern video surveillance systems generate far more video streams than human operators can effectively monitor, making automated analysis essential for timely detection of security events. This paper presents a unified multi-task deep learning framework that simultaneously performs face recognition with zone-based authorization, automatic license plate recognition, weapon detection, fire and smoke detection, and human action recognition on a shared GPU platform. Among the integrated modules, two task-specific deep-learning models are proposed in this work to address scenarios that are insufficiently represented in publicly available datasets: a single-class weapon detector fine-tuned on a merged and relabeled dataset, achieving a mean average precision (mAP@0.5) of 0.947, and a SlowFast-R50 action recognition model trained on a purpose-built vandalism dataset comprising 614 video clips, achieving 94.33% classification accuracy. To improve robustness in continuous video, all detection modules are integrated into a temporal event-validation architecture based on multi-frame confirmation, confidence-weighted voting, and cascaded filtering, transforming frame-level predictions into reliable security events. Each module is evaluated independently on established public datasets (LFW, D-Fire, FIRESENSE, and UCF-Crime), followed by integrated end-to-end system evaluation. The proposed temporal validation strategy reduces the fire and smoke false-alarm rate from 52% to 4% and improves video license plate exact-match accuracy from 66.7% to 81.8%, while the complete framework maintains real-time operation with a per-frame latency below 100 ms on commodity hardware. These results demonstrate that combining specialized deep-learning models with temporal event validation provides an effective and practical solution for reliable real-time intelligent video surveillance.

Related papers

• City Sentinel: A Unified AI-Based Smart Surveillance Framework for Real-Time Multi-Threat Detection Using Deep Learning [0.06999740786886534]
  City Sentinel is a unified AI-based surveillance framework that integrates six detection capabilities into one scalable platform.<n>System achieves median end-to-end latency of 743 ms and supports four concurrent RTSP streams.<n>It achieves a 91.2% face-match rate, 85.7% plate-reading accuracy, and mAP@0.5 scores of 0.846 to 0.889 across the fire, knife, and weapon detection modules.
  arXiv  Detail & Related papers  (2026-08-09T19:59:31Z)
• VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference [2.538209532048867]
  We present a unified framework that combines deformable attention with causal temporal modeling to detect anomalies in video surveillance footage.<n>We evaluate our framework on VigilFormer, ShanghaiTech, and CUHK Avenue, at 41.5 FPS on a single GPU, recently outperforming weakly-supervised methods in both accuracy and speed.
  arXiv  Detail & Related papers  (2026-05-31T14:27:57Z)
• Real-Time Multi-Modal Embedded Vision Framework for Object Detection Facial Emotion Recognition and Biometric Identification on Low-Power Edge Platforms [0.44219509596259216]
  We present a real-time multi-modal vision framework that integrates object detection, owner-specific face recognition, and emotion detection into a unified pipeline deployed on a Raspberry Pi 5 edge platform.<n>Our work demonstrates that context-aware scheduling is the key to unlocking complex multi-modal AI on cost-effective edge hardware.
  arXiv  Detail & Related papers  (2026-01-17T09:06:47Z)
• Short-Window Sliding Learning for Real-Time Violence Detection via LLM-based Auto-Labeling [0.6999740786886536]
  This paper proposes a Short-Window Sliding Learning framework for real-time violence detection in CCTV footages.<n>The proposed method divides videos into 1-2 second clips and applies Large Language Model (LLM)-based auto-caption labeling to construct fine-grained datasets.
  arXiv  Detail & Related papers  (2025-11-14T00:29:31Z)
• Lie Detector: Unified Backdoor Detection via Cross-Examination Framework [68.45399098884364]
  We propose a unified backdoor detection framework in the semi-honest setting.<n>Our method achieves superior detection performance, improving accuracy by 5.4%, 1.6%, and 11.9% over SoTA baselines.<n> Notably, it is the first to effectively detect backdoors in multimodal large language models.
  arXiv  Detail & Related papers  (2025-03-21T06:12:06Z)
• CorrDiff: Adaptive Delay-aware Detector with Temporal Cue Inputs for Real-time Object Detection [11.714072240331518]
  CorrDiff is designed to tackle the challenge of delays in real-time detection systems.<n>It is able to utilize runtime-estimated temporal cues to predict objects' locations for multiple future frames.<n>It meets the stringent real-time processing requirements on all kinds of devices.
  arXiv  Detail & Related papers  (2025-01-09T10:34:25Z)
• Spatiotemporal Attention-based Semantic Compression for Real-time Video Recognition [117.98023585449808]
  We propose a temporal attention-based autoencoder (STAE) architecture to evaluate the importance of frames and pixels in each frame. We develop a lightweight decoder that leverages a 3D-2D CNN combined to reconstruct missing information. Experimental results show that ViT_STAE can compress the video dataset H51 by 104x with only 5% accuracy loss.
  arXiv  Detail & Related papers  (2023-05-22T07:47:27Z)
• Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequences [49.91741677556553]
  We propose TempO, a temporal ordering pretext task for pre-training region-level feature representations for perception tasks. We embed each frame by an unordered set of proposal feature vectors, a representation that is natural for object detection or tracking systems. Extensive evaluations on the BDD100K, nuImages, and MOT17 datasets show that our TempO pre-training approach outperforms single-frame self-supervised learning methods.
  arXiv  Detail & Related papers  (2023-02-17T18:18:27Z)
• NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition [89.84188594758588]
  A novel Non-saliency Suppression Network (NSNet) is proposed to suppress the responses of non-salient frames. NSNet achieves the state-of-the-art accuracy-efficiency trade-off and presents a significantly faster (2.44.3x) practical inference speed than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-21T09:41:22Z)
• ACDnet: An action detection network for real-time edge computing based on flow-guided feature approximation and memory aggregation [8.013823319651395]
  ACDnet is a compact action detection network targeting real-time edge computing. It exploits the temporal coherence between successive video frames to approximate CNN features rather than naively extracting them. It can robustly achieve detection well above real-time (75 FPS)
  arXiv  Detail & Related papers  (2021-02-26T14:06:31Z)
• Identity-Aware Attribute Recognition via Real-Time Distributed Inference in Mobile Edge Clouds [53.07042574352251]
  We design novel models for pedestrian attribute recognition with re-ID in an MEC-enabled camera monitoring system. We propose a novel inference framework with a set of distributed modules, by jointly considering the attribute recognition and person re-ID. We then devise a learning-based algorithm for the distributions of the modules of the proposed distributed inference framework.
  arXiv  Detail & Related papers  (2020-08-12T12:03:27Z)
• Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos [82.02074241700728]
  In this paper, we present a prohibitive-level action recognition model that is trained with only video-frame labels. Our method per person detectors have been trained on large image datasets within Multiple Instance Learning framework. We show how we can apply our method in cases where the standard Multiple Instance Learning assumption, that each bag contains at least one instance with the specified label, is invalid.
  arXiv  Detail & Related papers  (2020-07-21T10:45:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.