A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection

• URL: http://arxiv.org/abs/2603.28888v1
• Date: Mon, 30 Mar 2026 18:14:03 GMT
• Title: A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection
• Abstract summary: We propose a quantized vision-language model (VLM) running at 1--2,Hz alongside the primary AV control loop.<n>We achieve 500 ms inference a 50x speedup over the unoptimized FP16 baseline.<n>The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms.
• Score: 0.8466401378239364
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic anomalies-context-dependent hazards that pixel-level detectors cannot reason about-pose a critical safety risk in autonomous driving. We propose a \emph{semantic observer layer}: a quantized vision-language model (VLM) running at 1--2\,Hz alongside the primary AV control loop, monitoring for semantic edge cases, and triggering fail-safe handoffs when detected. Using Nvidia Cosmos-Reason1-7B with NVFP4 quantization and FlashAttention2, we achieve ~500 ms inference a ~50x speedup over the unoptimized FP16 baseline (no quantization, standard PyTorch attention) on the same hardware--satisfying the observer timing budget. We benchmark accuracy, latency, and quantization behavior in static and video conditions, identify NF4 recall collapse (10.6%) as a hard deployment constraint, and a hazard analysis mapping performance metrics to safety goals. The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms.

Related papers

• SafeAtlas-VL: Beyond Binary Multimodal Safety with Large-Scale Data and Guard Models [73.04773649437375]
  Multimodal safety moderation requires distinguishing risks arising from visual content, user intent, and assistant behavior.<n>We introduce SafeAtlas-VL, a dataset of 1.5M training instances that places image-, request-, and response-level judgments on a five-level ordered scale.<n>We also construct SafeAtlas-Bench, a set of 5,000 instances for evaluating five-level predictions and continuous risk scores.
  arXiv  Detail & Related papers  (2026-08-29T07:08:01Z)
• VigilFormer: Deformable Attention for Video Anomaly Detection with Causal Risk Inference [2.538209532048867]
  We present a unified framework that combines deformable attention with causal temporal modeling to detect anomalies in video surveillance footage.<n>We evaluate our framework on VigilFormer, ShanghaiTech, and CUHK Avenue, at 41.5 FPS on a single GPU, recently outperforming weakly-supervised methods in both accuracy and speed.
  arXiv  Detail & Related papers  (2026-05-31T14:27:57Z)
• Send a SCOUT First: Pre-hoc Reasoning for Adaptive Detector Allocation in Prompt-Injection Defense [37.10272384523361]
  We develop a framework for predicting each detector's per-sample reliability and latency from how it behaved on similar past inputs.<n>We build SCOUT-450, a benchmark that captures the structurally complex, agent-facing injections that older prompt-injection sets under-represent.<n>On SCOUT-450, a safety-oriented operating point reduces attack-success rate by 46% and total wall-clock by 40% relative to an always-on GPT-4o judge.
  arXiv  Detail & Related papers  (2026-05-29T04:49:20Z)
• How VLAs Fail Differently: Black-Box Action Monitoring Reveals Architecture-Specific Failure Signatures [0.0]
  We find that VLA architectures fail in fundamentally different, predictable ways at the motor-command level.<n>We quantify consequence of the well-known discrete/continuous VLA distinction: the two families produce qualitatively different failure monitor works.
  arXiv  Detail & Related papers  (2026-05-27T16:44:55Z)
• From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection [9.404376027901277]
  Pose-based Video Anomaly Detection (VAD) has gained significant attention for its privacy-preserving nature and robustness to environmental variations.<n>Traditional frame-level evaluations treat video as a collection of isolated frames, fundamentally misaligned with how anomalies manifest and are acted upon in the real world.<n>We introduce two strategies for temporal event localization: a score-refinement pipeline with hierarchical Gaussian smoothing and adaptive binarization, and an end-to-end Dual-Branch Model that directly generates event-level detections.
  arXiv  Detail & Related papers  (2026-04-10T13:52:18Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• When Sensors Fail: Temporal Sequence Models for Robust PPO under Sensor Drift [64.37959940809633]
  We study robustness of Proximal Policy Optimization (PPO) under temporally persistent sensor failures.<n>We show Transformer-based sequence policies substantially outperform, RNN, and SSMs in robustness, maintaining high returns even when large fractions of sensors are unavailable.
  arXiv  Detail & Related papers  (2026-03-04T22:21:54Z)
• Conditional Flow Matching for Continuous Anomaly Detection in Autonomous Driving on a Manifold-Aware Spectral Space [0.0]
  We present Deep-Flow, an unsupervised framework for safety-critical anomaly detection.<n>Unlike standard generative approaches that operate in unstable, high-dimensional coordinate spaces, Deep-Flow constrains the generative process to a low-rank spectral manifold.<n>This ensures kinematic smoothness by design and enables the computation of numerically stable, deterministic log-likelihood estimation.
  arXiv  Detail & Related papers  (2026-02-19T18:10:16Z)
• Detecting Object Tracking Failure via Sequential Hypothesis Testing [80.7891291021747]
  Real-time online object tracking in videos constitutes a core task in computer vision.<n>We propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time.<n>We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information.
  arXiv  Detail & Related papers  (2026-02-13T14:57:15Z)
• STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning [65.36458157092207]
  In vision-language models (VLMs), misalignment between textual descriptions and visual coordinates often induces hallucinations.<n>We propose a novel visual prompting paradigm that avoids the difficult problem of aligning coordinates across modalities.<n>We introduce STVG-R1, the first reinforcement learning framework for STVG, which employs a task-driven reward to jointly optimize temporal accuracy, spatial consistency, and structural format regularization.
  arXiv  Detail & Related papers  (2026-02-12T08:53:32Z)
• Unsupervised Anomaly Detection in Multi-Agent Trajectory Prediction via Transformer-Based Models [45.08545174556591]
  We propose an unsupervised anomaly detection framework based on a multi-agent Transformer.<n>A dual evaluation scheme has been proposed to assess both detection stability and physical alignment.<n>Our framework identifies 388 unique anomalies missed by Time-to-Collision and statistical baselines.
  arXiv  Detail & Related papers  (2026-01-28T08:33:10Z)
• EVEREST: An Evidential, Tail-Aware Transformer for Rare-Event Time-Series Forecasting [4.551615447454767]
  EVEREST is a transformer-based architecture for probabilistic rare-event forecasting.<n>It delivers calibrated predictions and tail-aware risk estimation.<n>It is applicable to high-stakes domains such as industrial monitoring, weather, and satellite diagnostics.
  arXiv  Detail & Related papers  (2026-01-26T23:15:20Z)
• LiQSS: Post-Transformer Linear Quantum-Inspired State-Space Tensor Networks for Real-Time 6G [85.58816960936069]
  Proactive and agentic control in Sixth-Generation (6G) Open Radio Access Networks (O-RAN) requires control-grade prediction under stringent Near-Time (Near-RT) latency and computational constraints.<n>This paper investigates a post-Transformer paradigm for efficient radio telemetry forecasting.<n>We propose a quantum-inspired state-space tensor network that replaces self-attention with stable structured state-space dynamics kernels.
  arXiv  Detail & Related papers  (2026-01-18T12:08:38Z)
• SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection [6.806105013817923]
  SAVANT is a structured reasoning framework that achieves high accuracy and recall in detecting anomalous driving scenarios.<n>By automatically labeling over 9,640 real-world images with high accuracy, SAVANT addresses the critical data scarcity problem in anomaly detection.
  arXiv  Detail & Related papers  (2025-10-20T19:14:29Z)
• Adaptive t Design Dummy-Gate Obfuscation for Cryogenic Scale Enforcement [0.0]
  Cloud quantum services can reveal circuit structure and timing through scheduler metadata, latency patterns, and co-tenant interference.<n>We introduce NADGO, a scheduling and obfuscation stack that enforces operational privacy for gate-model workloads.<n>We prototype the approach on a 4-qubit superconducting tile with cryo-CMOS control and evaluate both depth-varied local-random circuits and small QAOA instances.
  arXiv  Detail & Related papers  (2025-08-31T12:12:48Z)
• SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Model [52.47816604709358]
  Video anomaly detection (VAD) aims to identify unexpected events in videos and has wide applications in safety-critical domains.<n> vision-language models (VLMs) have demonstrated strong multimodal reasoning capabilities, offering new opportunities for anomaly detection.<n>We propose SlowFastVAD, a hybrid framework that integrates a fast anomaly detector with a slow anomaly detector.
  arXiv  Detail & Related papers  (2025-04-14T15:30:03Z)
• Evaluating ML Robustness in GNSS Interference Classification, Characterization & Localization [42.14439854721613]
  Jamming devices disrupt signals from the global navigation satellite system (GNSS)<n>This paper introduces an extensive dataset comprising snapshots obtained from a low-frequency antenna.<n>Our objective is to assess the resilience of machine learning (ML) models against environmental changes.
  arXiv  Detail & Related papers  (2024-09-23T15:20:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.