VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation

• URL: http://arxiv.org/abs/2403.12415v1
• Date: Tue, 19 Mar 2024 03:55:39 GMT
• Title: VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation
• Authors: Hao Wang, Jiayou Qin, Ashish Bastola, Xiwen Chen, John Suchanek, Zihao Gong, Abolfazl Razi,
• Abstract summary: This paper explores the potential of Large Language Models in zero-shot anomaly detection for safe visual navigation. The proposed framework can identify anomalies within camera-captured frames that include any possible obstacles, then generate concise, audio-delivered descriptions emphasizing abnormalities.
• Score: 3.837186701755568
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-World and specialized prompts, the proposed framework can identify anomalies within camera-captured frames that include any possible obstacles, then generate concise, audio-delivered descriptions emphasizing abnormalities, assist in safe visual navigation in complex circumstances. Moreover, our proposed framework leverages the advantages of LLMs and the open-vocabulary object detection model to achieve the dynamic scenario switch, which allows users to transition smoothly from scene to scene, which addresses the limitation of traditional visual navigation. Furthermore, this paper explored the performance contribution of different prompt components, provided the vision for future improvement in visual accessibility, and paved the way for LLMs in video anomaly detection and vision-language understanding.

Related papers

• Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines [18.602869210526848]
  Vision Search Assistant is a novel framework that facilitates collaboration between vision-language models and web agents. By integrating visual and textual representations through this collaboration, the model can provide informed responses even when the image is novel to the system.
  arXiv  Detail & Related papers  (2024-10-28T17:04:18Z)
• VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use [74.39058448757645]
  We present VipAct, an agent framework that enhances vision-language models (VLMs) VipAct consists of an orchestrator agent, which manages task requirement analysis, planning, and coordination, along with specialized agents that handle specific tasks. We evaluate VipAct on benchmarks featuring a diverse set of visual perception tasks, with experimental results demonstrating significant performance improvements.
  arXiv  Detail & Related papers  (2024-10-21T18:10:26Z)
• Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models [72.75669790569629]
  Vision-language alignment in Large Vision-Language Models (LVLMs) successfully enables LLMs to understand visual input. We find that existing vision-language alignment methods fail to transfer the existing safety mechanism for text in LLMs to vision. We propose a novel Text-Guided vision-language alignment method (TGA) for LVLMs.
  arXiv  Detail & Related papers  (2024-10-16T15:20:08Z)
• VMAD: Visual-enhanced Multimodal Large Language Model for Zero-Shot Anomaly Detection [19.79027968793026]
  Zero-shot anomaly detection (ZSAD) recognizes and localizes anomalies in previously unseen objects. Existing ZSAD methods are limited by closed-world settings, struggling to unseen defects with predefined prompts. We propose a novel framework VMAD (Visual-enhanced MLLM Anomaly Detection) that enhances MLLM with visual-based IAD knowledge and fine-grained perception.
  arXiv  Detail & Related papers  (2024-09-30T09:51:29Z)
• End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting [68.37943632270505]
  Open-vocabulary video visual relationship detection aims to expand video visual relationship detection beyond categories. Existing methods usually use trajectory detectors trained on closed datasets to detect object trajectories. We propose an open-vocabulary relationship that leverages the rich semantic knowledge of CLIP to discover novel relationships.
  arXiv  Detail & Related papers  (2024-09-19T06:25:01Z)
• Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts [57.01985221057047]
  This paper introduces a novel method that learnstemporal prompt embeddings for weakly supervised video anomaly detection and localization (WSVADL) based on pre-trained vision-language models (VLMs) Our method achieves state-of-theart performance on three public benchmarks for the WSVADL task.
  arXiv  Detail & Related papers  (2024-08-12T03:31:29Z)
• InsightSee: Advancing Multi-agent Vision-Language Models for Enhanced Visual Understanding [12.082379948480257]
  This paper proposes InsightSee, a multi-agent framework to enhance vision-language models' capabilities in handling complex visual understanding scenarios. The framework comprises a description agent, two reasoning agents, and a decision agent, which are integrated to refine the process of visual information interpretation. The proposed framework outperforms state-of-the-art algorithms in 6 out of 9 benchmark tests, with a substantial advancement in multimodal understanding.
  arXiv  Detail & Related papers  (2024-05-31T13:56:55Z)
• Exosense: A Vision-Centric Scene Understanding System For Safe Exoskeleton Navigation [0.0]
  Exosense is a vision-centric scene understanding system. It generates rich, globally-consistent elevation maps, incorporating both semantic and terrain traversability information. We demonstrate the system's robustness to the challenges of typical periodic walking gaits.
  arXiv  Detail & Related papers  (2024-03-21T11:41:39Z)
• Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Models [81.71651422951074]
  Chain-of-Spot (CoS) method is a novel approach that enhances feature extraction by focusing on key regions of interest. This technique allows LVLMs to access more detailed visual information without altering the original image resolution. Our empirical findings demonstrate a significant improvement in LVLMs' ability to understand and reason about visual content.
  arXiv  Detail & Related papers  (2024-03-19T17:59:52Z)
• PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs [55.8550939439138]
  Vision-Language Models (VLMs) have shown immense potential by integrating large language models with vision systems. These models face challenges in the fundamental computer vision task of object localisation, due to their training on multimodal data containing mostly captions. We introduce an input-agnostic Positional Insert (PIN), a learnable spatial prompt, containing a minimal set of parameters that are slid inside the frozen VLM. Our PIN module is trained with a simple next-token prediction task on synthetic data without requiring the introduction of new output heads.
  arXiv  Detail & Related papers  (2024-02-13T18:39:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.