Semi-Supervised Cross-Modal Salient Object Detection with U-Structure Networks

• URL: http://arxiv.org/abs/2208.04361v1
• Date: Mon, 8 Aug 2022 18:39:37 GMT
• Title: Semi-Supervised Cross-Modal Salient Object Detection with U-Structure Networks
• Authors: Yunqing Bao, Hang Dai, Abdulmotaleb Elsaddik
• Abstract summary: We integrate the linguistic information into the vision-based U-Structure networks designed for salient object detection tasks. We propose a new module called efficient Cross-Modal Self-Attention (eCMSA) to combine visual and linguistic features. To reduce the heavy burden of labeling, we employ a semi-supervised learning method by training an image caption model.
• Score: 18.12933868289846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Salient Object Detection (SOD) is a popular and important topic aimed at precise detection and segmentation of the interesting regions in the images. We integrate the linguistic information into the vision-based U-Structure networks designed for salient object detection tasks. The experiments are based on the newly created DUTS Cross Modal (DUTS-CM) dataset, which contains both visual and linguistic labels. We propose a new module called efficient Cross-Modal Self-Attention (eCMSA) to combine visual and linguistic features and improve the performance of the original U-structure networks. Meanwhile, to reduce the heavy burden of labeling, we employ a semi-supervised learning method by training an image caption model based on the DUTS-CM dataset, which can automatically label other datasets like DUT-OMRON and HKU-IS. The comprehensive experiments show that the performance of SOD can be improved with the natural language input and is competitive compared with other SOD methods.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• LOGO: Video Text Spotting with Language Collaboration and Glyph Perception Model [20.007650672107566]
  Video text spotting (VTS) aims to simultaneously localize, recognize and track text instances in videos. Recent methods track the zero-shot results of state-of-the-art image text spotters directly. Fine-tuning transformer-based text spotters on specific datasets could yield performance enhancements.
  arXiv  Detail & Related papers  (2024-05-29T15:35:09Z)
• Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model [3.3772986620114387]
  We introduce ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features. Our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.
  arXiv  Detail & Related papers  (2024-04-19T07:24:32Z)
• Weakly Supervised Open-Vocabulary Object Detection [31.605276665964787]
  We propose a novel weakly supervised open-vocabulary object detection framework, namely WSOVOD, to extend traditional WSOD. To achieve this, we explore three vital strategies, including dataset-level feature adaptation, image-level salient object localization, and region-level vision-language alignment.
  arXiv  Detail & Related papers  (2023-12-19T18:59:53Z)
• Simple Image-level Classification Improves Open-vocabulary Object Detection [27.131298903486474]
  Open-Vocabulary Object Detection (OVOD) aims to detect novel objects beyond a given set of base categories on which the detection model is trained. Recent OVOD methods focus on adapting the image-level pre-trained vision-language models (VLMs), such as CLIP, to a region-level object detection task via, eg., region-level knowledge distillation, regional prompt learning, or region-text pre-training. These methods have demonstrated remarkable performance in recognizing regional visual concepts, but they are weak in exploiting the VLMs' powerful global scene understanding ability learned from the billion-scale
  arXiv  Detail & Related papers  (2023-12-16T13:06:15Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation [53.4319652364256]
  This paper presents the RefSAM model, which explores the potential of SAM for referring video object segmentation. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-RValModal. We employ a parameter-efficient tuning strategy to align and fuse the language and vision features effectively.
  arXiv  Detail & Related papers  (2023-07-03T13:21:58Z)
• Unified Visual Relationship Detection with Vision and Language Models [89.77838890788638]
  This work focuses on training a single visual relationship detector predicting over the union of label spaces from multiple datasets. We propose UniVRD, a novel bottom-up method for Unified Visual Relationship Detection by leveraging vision and language models. Empirical results on both human-object interaction detection and scene-graph generation demonstrate the competitive performance of our model.
  arXiv  Detail & Related papers  (2023-03-16T00:06:28Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network [17.980765138522322]
  This work introduces OmDet, a novel language-aware object detection architecture. Leveraging natural language as a universal knowledge representation, OmDet accumulates a "visual vocabulary" from diverse datasets. We demonstrate superior performance of OmDet over strong baselines in object detection in the wild, open-vocabulary detection, and phrase grounding.
  arXiv  Detail & Related papers  (2022-09-10T14:25:14Z)
• Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval [152.3504607706575]
  This research aims to conduct weakly-supervised multi-modal instance-level product retrieval for fine-grained product categories. We first contribute the Product1M datasets, and define two real practical instance-level retrieval tasks. We exploit to train a more effective cross-modal model which is adaptively capable of incorporating key concept information from the multi-modal data.
  arXiv  Detail & Related papers  (2022-06-17T15:40:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.