VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning

• URL: http://arxiv.org/abs/2311.15011v3
• Date: Thu, 11 Apr 2024 08:16:53 GMT
• Title: VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning
• Authors: Ziyang Luo, Nian Liu, Wangbo Zhao, Xuguang Yang, Dingwen Zhang, Deng-Ping Fan, Fahad Khan, Junwei Han,
• Abstract summary: We introduce VSCode, a model with novel 2D prompt learning to jointly address four SOD tasks and three COD tasks. We utilize VST as the foundation model and introduce 2D prompts within the encoder-decoder architecture to learn domain and task-specific knowledge. VSCode outperforms state-of-the-art methods across six tasks on 26 datasets.
• Score: 104.74705190239119
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Salient object detection (SOD) and camouflaged object detection (COD) are related yet distinct binary mapping tasks. These tasks involve multiple modalities, sharing commonalities and unique cues. Existing research often employs intricate task-specific specialist models, potentially leading to redundancy and suboptimal results. We introduce VSCode, a generalist model with novel 2D prompt learning, to jointly address four SOD tasks and three COD tasks. We utilize VST as the foundation model and introduce 2D prompts within the encoder-decoder architecture to learn domain and task-specific knowledge on two separate dimensions. A prompt discrimination loss helps disentangle peculiarities to benefit model optimization. VSCode outperforms state-of-the-art methods across six tasks on 26 datasets and exhibits zero-shot generalization to unseen tasks by combining 2D prompts, such as RGB-D COD. Source code has been available at https://github.com/Sssssuperior/VSCode.

Related papers

• A Unified Framework for 3D Scene Understanding [50.6762892022386]
  UniSeg3D is a unified 3D segmentation framework that achieves panoptic, semantic, instance, interactive, referring, and open-vocabulary semantic segmentation tasks within a single model. It facilitates inter-task knowledge sharing and promotes comprehensive 3D scene understanding. Experiments on three benchmarks, including the ScanNet20, ScanRefer, and ScanNet200, demonstrate that the UniSeg3D consistently outperforms current SOTA methods.
  arXiv  Detail & Related papers  (2024-07-03T16:50:07Z)
• A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detection [33.30644598646274]
  We propose a simple yet effective network (SENet) based on vision Transformer (ViT) To enhance the Transformer's ability to model local information, we propose a local information capture module (LICM) We also propose a dynamic weighted loss (DW loss) based on Binary Cross-Entropy (BCE) and Intersection over Union (IoU) loss, which guides the network to pay more attention to those smaller and more difficult-to-find target objects.
  arXiv  Detail & Related papers  (2024-02-29T07:29:28Z)
• Multi-task Learning with 3D-Aware Regularization [55.97507478913053]
  We propose a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance.
  arXiv  Detail & Related papers  (2023-10-02T08:49:56Z)
• Pre-train, Adapt and Detect: Multi-Task Adapter Tuning for Camouflaged Object Detection [38.5505943598037]
  We propose a novel pre-train, adapt and detect' paradigm to detect camouflaged objects. By introducing a large pre-trained model, abundant knowledge learned from massive multi-modal data can be directly transferred to COD. Our method outperforms existing state-of-the-art COD models by large margins.
  arXiv  Detail & Related papers  (2023-07-20T08:25:38Z)
• LiDAR-BEVMTN: Real-Time LiDAR Bird's-Eye View Multi-Task Perception Network for Autonomous Driving [12.713417063678335]
  We present a real-time multi-task convolutional neural network for LiDAR-based object detection, semantics, and motion segmentation. We propose a novel Semantic Weighting and Guidance (SWAG) module to transfer semantic features for improved object detection selectively. We achieve state-of-the-art results for two tasks, semantic and motion segmentation, and close to state-of-the-art performance for 3D object detection.
  arXiv  Detail & Related papers  (2023-07-17T21:22:17Z)
• 4D Unsupervised Object Discovery [53.561750858325915]
  We propose 4D unsupervised object discovery, jointly discovering objects from 4D data -- 3D point clouds and 2D RGB images with temporal information. We present the first practical approach for this task by proposing a ClusterNet on 3D point clouds, which is jointly optimized with a 2D localization network.
  arXiv  Detail & Related papers  (2022-10-10T16:05:53Z)
• The Devil is in the Task: Exploiting Reciprocal Appearance-Localization Features for Monocular 3D Object Detection [62.1185839286255]
  Low-cost monocular 3D object detection plays a fundamental role in autonomous driving. We introduce a Dynamic Feature Reflecting Network, named DFR-Net. We rank 1st among all the monocular 3D object detectors in the KITTI test set.
  arXiv  Detail & Related papers  (2021-12-28T07:31:18Z)
• Multi-Task Multi-Sensor Fusion for 3D Object Detection [93.68864606959251]
  We present an end-to-end learnable architecture that reasons about 2D and 3D object detection as well as ground estimation and depth completion. Our experiments show that all these tasks are complementary and help the network learn better representations by fusing information at various levels.
  arXiv  Detail & Related papers  (2020-12-22T22:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.