Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detection

• URL: http://arxiv.org/abs/2311.11273v2
• Date: Tue, 30 Jul 2024 05:45:50 GMT
• Title: Chain of Visual Perception: Harnessing Multimodal Large Language Models for Zero-shot Camouflaged Object Detection
• Abstract summary: We introduce a novel multimodal camo-perceptive framework (MMCPF) aimed at handling zero-shot Camouflaged Object Detection (COD) Our research proposes a zero-shot MMCPF that circumvents current COD methodologies. We propose a strategic enhancement called the Chain of Visual Perception (CoVP), which significantly improves the perceptual capabilities of MLLMs in camouflaged scenes.
• Score: 16.836026000473517
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we introduce a novel multimodal camo-perceptive framework (MMCPF) aimed at handling zero-shot Camouflaged Object Detection (COD) by leveraging the powerful capabilities of Multimodal Large Language Models (MLLMs). Recognizing the inherent limitations of current COD methodologies, which predominantly rely on supervised learning models demanding extensive and accurately annotated datasets, resulting in weak generalization, our research proposes a zero-shot MMCPF that circumvents these challenges. Although MLLMs hold significant potential for broad applications, their effectiveness in COD is hindered and they would make misinterpretations of camouflaged objects. To address this challenge, we further propose a strategic enhancement called the Chain of Visual Perception (CoVP), which significantly improves the perceptual capabilities of MLLMs in camouflaged scenes by leveraging both linguistic and visual cues more effectively. We validate the effectiveness of MMCPF on five widely used COD datasets, containing CAMO, COD10K, NC4K, MoCA-Mask and OVCamo. Experiments show that MMCPF can outperform all existing state-of-the-art zero-shot COD methods, and achieve competitive performance compared to weakly-supervised and fully-supervised methods, which demonstrates the potential of MMCPF. The Github link of this paper is \url{https://github.com/luckybird1994/MMCPF}.

Related papers

• Foundation and Multimodal Large Language Models for Face Presentation and Morph Attack Detection [36.05932340652857]
  Presentation attack detection (PAD) and morphing attack detection (MAD) are essential components of trustworthy face biometrics.<n>We investigate whether general-purpose foundation models (FMs) and multimodal large language models (MLLMs) encode PAD-relevant and MAD-relevant information.<n>Experiments show that FMs and MLLMs can achieve significant performance for PAD and MAD.
  arXiv  Detail & Related papers  (2026-08-30T14:07:31Z)
• Weakly Supervised Camouflaged Object Detection Based on the SAM Model and Mask Guidance [49.06684374007753]
  We introduce a new weakly supervised approach for camouflaged object detection to overcome limitations.<n>Specifically, we propose a novel network, MGNet, which tackles edge ambiguity and missed detections.<n>For the weak supervision challenge, we propose BoxSAM, which leverages the Segment Anything Model (SAM) with bounding-box prompts to generate pseudo-labels.
  arXiv  Detail & Related papers  (2026-05-25T03:26:13Z)
• FCL-COD: Weakly Supervised Camouflaged Object Detection with Frequency-aware and Contrastive Learning [49.58338471202117]
  Existing camouflage object detection (COD) methods rely on fully-supervised learning guided by mask annotations.<n>Compared to fully-supervised methods, existing weakly-supervised COD methods exhibit significantly poorer performance.<n>We propose a frequency-aware and contrastive learning-based WSCOD framework in this paper, named FCL-COD.
  arXiv  Detail & Related papers  (2026-03-24T09:05:41Z)
• Multi-Paradigm Collaborative Adversarial Attack Against Multi-Modal Large Language Models [67.45032003041399]
  We propose a novel Multi-Paradigm Collaborative Attack (MPCAttack) framework to boost the transferability of adversarial examples against MLLMs.<n>MPCO adaptively balances the importance of different paradigm representations and guides the global optimisation.<n>Our solution consistently outperforms state-of-the-art methods in both targeted and untargeted attacks on open-source and closed-source MLLMs.
  arXiv  Detail & Related papers  (2026-03-05T06:01:26Z)
• Decoding Large Language Diffusion Models with Foreseeing Movement [36.326379474672564]
  Large Language Diffusion Models (LLDMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models.<n>Existing methods, however, focus mainly on local effects while overlooking long-term impacts.<n>We propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential.
  arXiv  Detail & Related papers  (2025-12-03T14:39:41Z)
• A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Models [85.30893355216486]
  We study how visual token redundancy evolves with different dMLLM architectures and tasks.<n>Our study reveals that visual redundancy emerges only in from-scratch dMLLMs while handling long-answer tasks.<n>Layer-skipping is promising for accelerating AR-to-diffusion dMLLMs, whereas progressive or late-step pruning is more effective for from-scratch dMLLMs.
  arXiv  Detail & Related papers  (2025-11-19T04:13:36Z)
• ImCoref-CeS: An Improved Lightweight Pipeline for Coreference Resolution with LLM-based Checker-Splitter Refinement [45.01372641622595]
  We present ImCoref-CeS, a novel framework that integrates an enhanced supervised model with Large Language Models (LLMs)-based reasoning.<n>First, we present an improved CR method (textbfImCoref) to push the performance boundaries of the supervised neural method.<n>We employ an LLM acting as a multi-role Checker-Splitter agent to validate candidate mentions and coreference results.
  arXiv  Detail & Related papers  (2025-10-11T14:48:08Z)
• Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs [78.09559830840595]
  We present the first systematic study on quantizing diffusion-based language models.<n>We identify the presence of activation outliers, characterized by abnormally large activation values.<n>We implement state-of-the-art PTQ methods and conduct a comprehensive evaluation.
  arXiv  Detail & Related papers  (2025-08-20T17:59:51Z)
• Grounded Chain-of-Thought for Multimodal Large Language Models [66.04061083611863]
  We propose a new learning task for multimodal large language models (MLLMs) called Grounded Chain-of-Thought (GCoT) GCoT is keen to helping MLLMs to recognize and ground the relevant visual cues step by step, thereby predicting the correct answer with grounding coordinates as the intuitive basis. To facilitate this task, we also carefully design and construct a dataset called multimodal grounded chain-of-thought (MM-GCoT) consisting of 24,022 GCoT examples for 5,033 images.
  arXiv  Detail & Related papers  (2025-03-17T04:07:47Z)
• ChatGPT Encounters Morphing Attack Detection: Zero-Shot MAD with Multi-Modal Large Language Models and General Vision Models [13.21801650767302]
  Face Recognition Systems (FRS) are increasingly vulnerable to face-morphing attacks, prompting the development of Morphing Attack Detection (MAD) algorithms. A key challenge in MAD lies in its limited generalizability to unseen data and its lack of explainability-critical for practical application environments. This work explores a novel approach to MAD using zero-shot learning leveraged on Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2025-03-13T22:53:24Z)
• Chain-of-Sketch: Enabling Global Visual Reasoning [37.09634351533441]
  We introduce an expanded set of global visual datasets involving graphs, strings, mazes, and image grids.<n>We show that large vision models still struggle to learn these tasks efficiently.<n>We propose a method called chain-of-sketch (CoS) to mitigate this learning inefficiency.
  arXiv  Detail & Related papers  (2024-10-10T17:44:13Z)
• Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality [69.76121008898677]
  Fine-grained Selective Calibrated CLIP integrates local hard negative loss and selective calibrated regularization. Our evaluations show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities.
  arXiv  Detail & Related papers  (2024-10-07T17:16:20Z)
• SAM-COD: SAM-guided Unified Framework for Weakly-Supervised Camouflaged Object Detection [12.915303163650002]
  Most Camouflaged Object Detection (COD) methods heavily rely on mask annotations, which are time-consuming and labor-intensive to acquire. Existing weakly-supervised COD approaches exhibit significantly inferior performance compared to fully-supervised methods. We propose a unified COD framework, termed SAM-COD, which is capable of supporting arbitrary weakly-supervised labels.
  arXiv  Detail & Related papers  (2024-08-20T11:49:27Z)
• Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models [17.663550432103534]
  Multimodal Large Language Models (MLLMs) extend the capacity of LLMs to understand multimodal information comprehensively. These models are susceptible to jailbreak attacks, where malicious users can break the safety alignment of the target model and generate misleading and harmful answers. We propose Cross-modality Information DEtectoR (CIDER), a plug-and-play jailbreaking detector designed to identify maliciously perturbed image inputs.
  arXiv  Detail & Related papers  (2024-07-31T15:02:46Z)
• CoFiNet: Unveiling Camouflaged Objects with Multi-Scale Finesse [46.79770062391987]
  We introduce a novel method for camouflage object detection, named CoFiNet. Our approach focuses on multi-scale feature fusion and extraction, with special attention to the model's segmentation effectiveness. CoFiNet achieves state-of-the-art performance across all datasets.
  arXiv  Detail & Related papers  (2024-02-03T17:24:55Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Large Model Based Referring Camouflaged Object Detection [51.80619142347807]
  Referring camouflaged object detection (Ref-COD) is a recently-proposed problem aiming to segment out specified camouflaged objects matched with a textual or visual reference. Our motivation is to make full use of the semantic intelligence and intrinsic knowledge of recent Multimodal Large Language Models (MLLMs) to decompose this complex task in a human-like way. We propose a large-model-based Multi-Level Knowledge-Guided multimodal method for Ref-COD termed MLKG.
  arXiv  Detail & Related papers  (2023-11-28T13:45:09Z)
• Towards General Visual-Linguistic Face Forgery Detection [95.73987327101143]
  Deepfakes are realistic face manipulations that can pose serious threats to security, privacy, and trust. Existing methods mostly treat this task as binary classification, which uses digital labels or mask signals to train the detection model. We propose a novel paradigm named Visual-Linguistic Face Forgery Detection(VLFFD), which uses fine-grained sentence-level prompts as the annotation.
  arXiv  Detail & Related papers  (2023-07-31T10:22:33Z)
• CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion Models [72.93652777646233]
  Camouflaged Object Detection (COD) is a challenging task in computer vision due to the high similarity between camouflaged objects and their surroundings. We propose a new paradigm that treats COD as a conditional mask-generation task leveraging diffusion models. Our method, dubbed CamoDiffusion, employs the denoising process of diffusion models to iteratively reduce the noise of the mask.
  arXiv  Detail & Related papers  (2023-05-29T07:49:44Z)
• On Evaluating Adversarial Robustness of Large Vision-Language Models [64.66104342002882]
  We evaluate the robustness of large vision-language models (VLMs) in the most realistic and high-risk setting. In particular, we first craft targeted adversarial examples against pretrained models such as CLIP and BLIP. Black-box queries on these VLMs can further improve the effectiveness of targeted evasion.
  arXiv  Detail & Related papers  (2023-05-26T13:49:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.