SENSE-VAD: Sentient and Semantic Video Anomaly Detection for Autonomous Driving

• URL: http://arxiv.org/abs/2606.31875v1
• Date: Tue, 30 Jun 2026 15:57:50 GMT
• Title: SENSE-VAD: Sentient and Semantic Video Anomaly Detection for Autonomous Driving
• Abstract summary: We introduce SENSE-VAD, the first synthetic video anomaly detection benchmark for autonomous driving.<n>We generate distinct anomaly scenarios across multiple categories: individual behaviors, group behaviors, person--object interactions, cyclist interactions, vehicle & agent, each annotated with per-frame binary labels.<n>We evaluate state-of-the-art video anomaly detection baselines and demonstrate that socially complex anomalies constitute a distinct and currently unsolved challenge.
• Score: 12.729229824969904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous vehicles (AVs) must navigate not only motion-based hazards but also socially complex situations whose danger is constituted by inter-agent relationships rather than movement statistics alone. A child running away from a guardian, a person being carried by another, or a pursuer chasing a pedestrian across a sidewalk are all anomalous in social context, yet none produces an obvious motion signal that current anomaly detectors are equipped to flag. We introduce SENSE-VAD, the first synthetic video anomaly detection benchmark for autonomous driving explicitly designed around socially complex anomalies. Using the CARLA simulator and Unreal Engine (UE), we generate distinct anomaly scenarios across multiple categories: individual behaviors, group behaviors, person--object interactions, cyclist interactions, vehicle & agent, each annotated with per-frame binary labels. A key design principle is the separation of social anomaly from motion-based or appearance-based anomaly: many scenarios involve motion of objects that appears unremarkable in isolation but is anomalous in relational context. We additionally provide real-world normal and anomalous videos as a sim-to-real transfer probe. We evaluate state-of-the-art video anomaly detection baselines and demonstrate that socially complex anomalies constitute a distinct and currently unsolved challenge. Our dataset, annotations, and generation code are publicly available.

Related papers

• CausalDrive: Real-time Causal World Models for Autonomous Driving [60.66609721457312]
  We present CausalDrive, a controllable, real-time foundation driving world.<n>CaulDrive operates solely on the initial front-view frame, the ego-vehicle's trajectory, and a macroscopic text prompt.<n>We demonstrate its versatility across three downstream applications: (1) generative closed-loop evaluation with significantly mitigated collision artifacts, (2) large-scale Reinforcement Learning (RL) post-training driven by a Video2Reward module, and (3) real-time human-in-the-loop simulation.
  arXiv  Detail & Related papers  (2026-06-13T15:04:44Z)
• Out of Context: Reliability in Multimodal Anomaly Detection Requires Contextual Inference [54.774686416991386]
  We argue that multimodal anomaly detection should be reframed as a cross-modal contextual inference problem.<n>This perspective has implications for model design, evaluation protocols, and benchmark construction.
  arXiv  Detail & Related papers  (2026-04-14T19:32:55Z)
• Track Any Anomalous Object: A Granular Video Anomaly Detection Pipeline [63.96226274616927]
  A new framework called Track Any Anomalous Object (TAO) introduces a granular video anomaly detection pipeline.<n>Unlike methods that assign anomaly scores to every pixel, our approach transforms the problem into pixel-level tracking of anomalous objects.<n>Experiments demonstrate that TAO sets new benchmarks in accuracy and robustness.
  arXiv  Detail & Related papers  (2025-06-05T15:49:39Z)
• An Expert Ensemble for Detecting Anomalous Scenes, Interactions, and Behaviors in Autonomous Driving [5.424441980205082]
  Safety in a near-infinite number of driving scenarios becomes one of the major concerns for the widespread adoption of fully autonomous driving.<n>On-road anomaly detection in egocentric videos remains a challenging problem due to the difficulties introduced by complex and interactive scenarios.<n>We propose three unsupervised anomaly detection experts: a scene expert that focuses on frame-level appearances to detect abnormal scenes and unexpected scene motions; an interaction expert that models normal relative motions between two road participants and raises alarms whenever anomalous interactions emerge; and a behavior expert which monitors abnormal behaviors of individual objects by future trajectory prediction.
  arXiv  Detail & Related papers  (2025-02-23T00:43:23Z)
• Hybrid Video Anomaly Detection for Anomalous Scenarios in Autonomous Driving [16.94473342644408]
  We present a variation of the HF$2$-VAD surveillance video anomaly detection method for autonomous driving.<n>We learn a representation of normality from a vehicle's ego perspective and evaluate pixel-wise anomaly detections in rare and critical scenarios.
  arXiv  Detail & Related papers  (2024-06-10T16:14:33Z)
• Hawk: Learning to Understand Open-World Video Anomalies [76.9631436818573]
  Video Anomaly Detection (VAD) systems can autonomously monitor and identify disturbances, reducing the need for manual labor and associated costs. We introduce Hawk, a novel framework that leverages interactive large Visual Language Models (VLM) to interpret video anomalies precisely. We have annotated over 8,000 anomaly videos with language descriptions, enabling effective training across diverse open-world scenarios, and also created 8,000 question-answering pairs for users' open-world questions.
  arXiv  Detail & Related papers  (2024-05-27T07:08:58Z)
• Learn Suspected Anomalies from Event Prompts for Video Anomaly Detection [16.77262005540559]
  A novel framework is proposed to guide the learning of suspected anomalies from event prompts. It enables a new multi-prompt learning process to constrain the visual-semantic features across all videos. Our proposed model outperforms most state-of-the-art methods in terms of AP or AUC.
  arXiv  Detail & Related papers  (2024-03-02T10:42:47Z)
• UniSim: A Neural Closed-Loop Sensor Simulator [76.79818601389992]
  We present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene. We incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions.
  arXiv  Detail & Related papers  (2023-08-03T17:56:06Z)
• Contracting Skeletal Kinematics for Human-Related Video Anomaly Detection [43.72042594414675]
  COSKAD is a novel model that encodes skeletal human motion by a graph convolutional network. It learns to COntract SKeletal embeddings onto a latent hypersphere of minimum volume for Video Anomaly Detection. All variants outperform the state-of-the-art on the most recent UBnormal dataset.
  arXiv  Detail & Related papers  (2023-01-23T15:32:27Z)
• Anomaly Detection in Multi-Agent Trajectories for Automated Driving [2.5211566369910967]
  Similar to humans, automated vehicles are supposed to perform anomaly detection. Our innovation is the ability to jointly learn multiple trajectories of a dynamic number of agents.
  arXiv  Detail & Related papers  (2021-10-15T08:07:31Z)
• Learning Sparse Interaction Graphs of Partially Observed Pedestrians for Trajectory Prediction [0.3025231207150811]
  Multi-pedestrian trajectory prediction is an indispensable safety element of autonomous systems that interact with crowds in unstructured environments. We propose Gumbel Social Transformer, in which an Edge Gumbel Selector samples a sparse graph of partially observed pedestrians at each time step. We demonstrate that our model overcomes the potential problems caused by the assumptions, and our approach outperforms the related works in benchmark evaluation.
  arXiv  Detail & Related papers  (2021-07-15T00:45:11Z)
• Unsupervised Video Anomaly Detection via Normalizing Flows with Implicit Latent Features [8.407188666535506]
  Most existing methods use an autoencoder to learn to reconstruct normal videos. We propose an implicit two-path AE (ITAE), a structure in which two encoders implicitly model appearance and motion features. For the complex distribution of normal scenes, we suggest normal density estimation of ITAE features. NF models intensify ITAE performance by learning normality through implicitly learned features.
  arXiv  Detail & Related papers  (2020-10-15T05:02:02Z)
• A Flow Base Bi-path Network for Cross-scene Video Crowd Understanding in Aerial View [93.23947591795897]
  In this paper, we strive to tackle the challenges and automatically understand the crowd from the visual data collected from drones. To alleviate the background noise generated in cross-scene testing, a double-stream crowd counting model is proposed. To tackle the crowd density estimation problem under extreme dark environments, we introduce synthetic data generated by game Grand Theft Auto V(GTAV)
  arXiv  Detail & Related papers  (2020-09-29T01:48:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.