Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models

• URL: http://arxiv.org/abs/2407.10299v2
• Date: Sat, 20 Jul 2024 07:56:47 GMT
• Title: Follow the Rules: Reasoning for Video Anomaly Detection with Large Language Models
• Authors: Yuchen Yang, Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, Shao-Yuan Lo,
• Abstract summary: Video Anomaly Detection is crucial for applications such as security surveillance and autonomous driving. Existing VAD methods provide little rationale behind detection, hindering public trust in real-world deployments. We propose AnomalyRuler, a rule-based reasoning framework for VAD with Large Language Models.
• Score: 21.48544455321618
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Anomaly Detection (VAD) is crucial for applications such as security surveillance and autonomous driving. However, existing VAD methods provide little rationale behind detection, hindering public trust in real-world deployments. In this paper, we approach VAD with a reasoning framework. Although Large Language Models (LLMs) have shown revolutionary reasoning ability, we find that their direct use falls short of VAD. Specifically, the implicit knowledge pre-trained in LLMs focuses on general context and thus may not apply to every specific real-world VAD scenario, leading to inflexibility and inaccuracy. To address this, we propose AnomalyRuler, a novel rule-based reasoning framework for VAD with LLMs. AnomalyRuler comprises two main stages: induction and deduction. In the induction stage, the LLM is fed with few-shot normal reference samples and then summarizes these normal patterns to induce a set of rules for detecting anomalies. The deduction stage follows the induced rules to spot anomalous frames in test videos. Additionally, we design rule aggregation, perception smoothing, and robust reasoning strategies to further enhance AnomalyRuler's robustness. AnomalyRuler is the first reasoning approach for the one-class VAD task, which requires only few-normal-shot prompting without the need for full-shot training, thereby enabling fast adaption to various VAD scenarios. Comprehensive experiments across four VAD benchmarks demonstrate AnomalyRuler's state-of-the-art detection performance and reasoning ability. AnomalyRuler is open-source and available at: https://github.com/Yuchen413/AnomalyRuler

Related papers

• Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection [52.5174167737992]
  Video anomaly detection (VAD) aims to identify abnormal events in videos.<n>We propose SteerVAD, which advances MLLM-based VAD by shifting from passively reading to actively steering and rectifying internal representations.<n>Our method achieves state-of-the-art performance among tuning-free approaches requiring only 1% of training data.
  arXiv  Detail & Related papers  (2026-02-27T13:48:50Z)
• Reason-IAD: Knowledge-Guided Dynamic Latent Reasoning for Explainable Industrial Anomaly Detection [85.29900916231655]
  Reason-IAD is a knowledge-guided dynamic latent reasoning framework for explainable industrial anomaly detection.<n>Experiments demonstrate that Reason-IAD consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-02-10T14:54:17Z)
• From Gameplay Traces to Game Mechanics: Causal Induction with Large Language Models [64.43268969806098]
  We investigate Causal Induction: the ability to infer governing laws from observational data.<n>We compare two approaches to VGDL generation: direct code generation from observations, and a two-stage method that first infers a structural causal model (SCM) and then translates it into VGDL.<n>Results show that the SCM-based approach more often produces VGDL descriptions closer to the ground truth than direct generation.
  arXiv  Detail & Related papers  (2026-01-30T08:48:23Z)
• Directional Reasoning Injection for Fine-Tuning MLLMs [51.53222423215055]
  Multimodal large language models (MLLMs) are rapidly advancing, yet their reasoning ability often lags behind that of strong text-only counterparts.<n>Existing methods to bridge this gap rely on supervised fine-tuning over large-scale multimodal reasoning data or reinforcement learning.<n>We propose Directional Reasoning Injection for Fine-Tuning (DRIFT) to solve this problem.
  arXiv  Detail & Related papers  (2025-10-16T18:06:46Z)
• PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoning [57.89188317734747]
  PrismRAG trains the model with distractor-aware QA pairs mixing gold evidence with subtle distractor passages.<n>It instills reasoning-centric habits that make the LLM plan, rationalize, and synthesize without relying on extensive human engineered instructions.
  arXiv  Detail & Related papers  (2025-07-25T00:15:31Z)
• HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs [8.18063726177317]
  Video Anomaly Detection (VAD) aims to identify and locate deviations from normal patterns in video sequences.<n>We propose HiProbe-VAD, a novel framework that leverages pre-trained Multimodal Large Language Models (MLLMs) for VAD without requiring fine-tuning.
  arXiv  Detail & Related papers  (2025-07-23T10:41:46Z)
• VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning [12.293826084601115]
  Video anomaly understanding is essential for smart cities, security surveillance, and disaster alert systems.<n>Despite advances in anomaly detection, existing methods often lack interpretability and struggle to capture the causal and contextual aspects of abnormal events.<n>We introduce VAU-R1, a data-efficient framework built upon Multimodal Large Language Models (MLLMs), which enhances anomaly reasoning through Reinforcement Fine-Tuning (RFT)
  arXiv  Detail & Related papers  (2025-05-29T14:48:10Z)
• Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought [58.321044666612174]
  Vad-R1 is an end-to-end MLLM-based framework for Video Anomaly Reasoning.<n>We design a Perception-to-Cognition Chain-of-Thought (P2C-CoT) that simulates the human process of recognizing anomalies.<n>We also propose an improved reinforcement learning algorithm AVA-GRPO, which explicitly incentivizes the anomaly reasoning capability of MLLMs.
  arXiv  Detail & Related papers  (2025-05-26T12:05:16Z)
• SlowFastVAD: Video Anomaly Detection via Integrating Simple Detector and RAG-Enhanced Vision-Language Model [52.47816604709358]
  Video anomaly detection (VAD) aims to identify unexpected events in videos and has wide applications in safety-critical domains. vision-language models (VLMs) have demonstrated strong multimodal reasoning capabilities, offering new opportunities for anomaly detection. We propose SlowFastVAD, a hybrid framework that integrates a fast anomaly detector with a slow anomaly detector.
  arXiv  Detail & Related papers  (2025-04-14T15:30:03Z)
• Patterns Over Principles: The Fragility of Inductive Reasoning in LLMs under Noisy Observations [43.491353243991284]
  We introduce Robust Rule Induction, a task that evaluates large language models' capability in inferring rules from data fused with noisy examples. We also propose Sample-steered Rule Refinement (SRR), a method enhancing reasoning stability via observation diversification and execution-guided feedback. Our findings challenge LLMs' reasoning, revealing susceptibility to hypothesis drift and pattern overfitting, while providing empirical evidence critical for developing human-like inductive systems.
  arXiv  Detail & Related papers  (2025-02-22T10:03:19Z)
• Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models [29.078437003042357]
  Zero-Shot Anomaly Detection (ZSAD) is an emerging AD paradigm. We propose Anomaly-OneVision (Anomaly-OV), the first specialist visual assistant for ZSAD and reasoning.
  arXiv  Detail & Related papers  (2025-02-11T14:50:43Z)
• RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios [58.90106984375913]
  RuleArena is a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning.<n> Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions.
  arXiv  Detail & Related papers  (2024-12-12T06:08:46Z)
• Effort: Efficient Orthogonal Modeling for Generalizable AI-Generated Image Detection [66.16595174895802]
  Existing AI-generated image (AIGI) detection methods often suffer from limited generalization performance. In this paper, we identify a crucial yet previously overlooked asymmetry phenomenon in AIGI detection.
  arXiv  Detail & Related papers  (2024-11-23T19:10:32Z)
• From Explicit Rules to Implicit Reasoning in Weakly Supervised Video Anomaly Detection [1.8274323268621635]
  This paper introduces Rule-based Violence Monitoring (RuleVM), a novel weakly supervised video anomaly detection (WVAD) paradigm. RuleVM employs a dual-branch architecture: an implicit branch using visual features for coarse-grained binary classification, with feature extraction split into scene frames and action channels, and an explicit branch leveraging language-image alignment for fine-grained classification. The explicit branch utilizes the state-of-the-art YOLO-World model for object detection in video frames, with association rules mined from data as video descriptors.
  arXiv  Detail & Related papers  (2024-10-29T12:22:07Z)
• Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM [35.06386971859359]
  Holmes-VAD is a novel framework that leverages precise temporal supervision and rich multimodal instructions. We construct the first large-scale multimodal VAD instruction-tuning benchmark, VAD-Instruct50k. Building upon the VAD-Instruct50k dataset, we develop a customized solution for interpretable video anomaly detection.
  arXiv  Detail & Related papers  (2024-06-18T03:19:24Z)
• PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection [59.34973469354926]
  This paper proposes a one-class prompt learning method for few-shot anomaly detection, termed PromptAD. For image-level/pixel-level anomaly detection, PromptAD achieves first place in 11/12 few-shot settings on MVTec and VisA.
  arXiv  Detail & Related papers  (2024-04-08T06:53:30Z)
• Harnessing Large Language Models for Training-free Video Anomaly Detection [34.76811491190446]
  Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Training-based methods are prone to be domain-specific, thus being costly for practical deployment. We propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training-free paradigm.
  arXiv  Detail & Related papers  (2024-04-01T09:34:55Z)
• Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning [74.90592233107712]
  We propose a Direct-Indirect Reasoning (DIR) method, which considers Direct Reasoning (DR) and Indirect Reasoning (IR) as multiple parallel reasoning paths that are merged to derive the final answer. Our DIR method is simple yet effective and can be straightforwardly integrated with existing variants of CoT methods.
  arXiv  Detail & Related papers  (2024-02-06T03:41:12Z)
• InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance [56.184255657175335]
  We develop textbfInferAligner, a novel inference-time alignment method that utilizes cross-model guidance for harmlessness alignment. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics. It significantly diminishes the Attack Success Rate (ASR) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks.
  arXiv  Detail & Related papers  (2024-01-20T10:41:03Z)
• Can LLMs Follow Simple Rules? [28.73820874333199]
  Rule-following Language Evaluation Scenarios (RuLES) is a framework for measuring rule-following ability in Large Language Models. RuLES consists of 14 simple text scenarios in which the model is instructed to obey various rules while interacting with the user. We show that almost all current models struggle to follow scenario rules, even on straightforward test cases.
  arXiv  Detail & Related papers  (2023-11-06T08:50:29Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• Machine Learning with Probabilistic Law Discovery: A Concise Introduction [77.34726150561087]
  Probabilistic Law Discovery (PLD) is a logic based Machine Learning method, which implements a variant of probabilistic rule learning. PLD is close to Decision Tree/Random Forest methods, but it differs significantly in how relevant rules are defined. This paper outlines the main principles of PLD, highlight its benefits and limitations and provide some application guidelines.
  arXiv  Detail & Related papers  (2022-12-22T17:40:13Z)
• Towards Open Set Video Anomaly Detection [11.944167192592905]
  Open Set Video Anomaly Detection (OpenVAD) aims to identify abnormal events from video data where both known anomalies and novel ones exist in testing. We develop a novel weakly supervised method for the OpenVAD problem by integrating evidential deep learning (EDL) and normalizing flows (NFs) into a multiple instance learning (MIL) framework.
  arXiv  Detail & Related papers  (2022-08-23T17:53:34Z)
• A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning [33.623558899286635]
  In deep reinforcement learning (RL) systems, abnormal states pose significant risks by potentially triggering unpredictable behaviors and unsafe actions. We propose a novel Mahalanobis distance-based (MD) anomaly detection framework, called textitMDX, for deep RL algorithms. MDX simultaneously addresses random, adversarial, and out-of-distribution (OOD) state outliers in both offline and online settings.
  arXiv  Detail & Related papers  (2021-09-21T00:09:03Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.