Indoor scene recognition from images under visual corruptions

• URL: http://arxiv.org/abs/2408.13029v1
• Date: Fri, 23 Aug 2024 12:35:45 GMT
• Title: Indoor scene recognition from images under visual corruptions
• Authors: Willams de Lima Costa, Raul Ismayilov, Nicola Strisciuglio, Estefania Talavera Martinez,
• Abstract summary: This paper presents an innovative approach to indoor scene recognition that leverages multimodal data fusion. We examine two multimodal networks that synergize visual features from CNN models with semantic captions via a Graph Convolutional Network (GCN) Our study shows that this fusion improves markedly model performance, with notable gains in Top-1 accuracy when evaluated against a corrupted subset of the Places365 dataset.
• Score: 3.4861209026118836
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The classification of indoor scenes is a critical component in various applications, such as intelligent robotics for assistive living. While deep learning has significantly advanced this field, models often suffer from reduced performance due to image corruption. This paper presents an innovative approach to indoor scene recognition that leverages multimodal data fusion, integrating caption-based semantic features with visual data to enhance both accuracy and robustness against corruption. We examine two multimodal networks that synergize visual features from CNN models with semantic captions via a Graph Convolutional Network (GCN). Our study shows that this fusion markedly improves model performance, with notable gains in Top-1 accuracy when evaluated against a corrupted subset of the Places365 dataset. Moreover, while standalone visual models displayed high accuracy on uncorrupted images, their performance deteriorated significantly with increased corruption severity. Conversely, the multimodal models demonstrated improved accuracy in clean conditions and substantial robustness to a range of image corruptions. These results highlight the efficacy of incorporating high-level contextual information through captions, suggesting a promising direction for enhancing the resilience of classification systems.

Related papers

• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• A Survey on the Robustness of Computer Vision Models against Common Corruptions [3.6486148851646063]
  Computer vision models are susceptible to changes in input images caused by sensor errors or extreme imaging environments. These corruptions can significantly hinder the reliability of these models when deployed in real-world scenarios. We present a comprehensive overview of methods that improve the robustness of computer vision models against common corruptions.
  arXiv  Detail & Related papers  (2023-05-10T10:19:31Z)
• Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment [11.864465182761945]
  We show how data augmentation techniques for generating synthetic noise can address data sparsity in this domain. Second, we enhance the robustness of the model by expanding a state-of-the-art model to a dual network architecture. Third, we evaluate the prediction reliability using confidence calibration on images with different difficulty/noise levels.
  arXiv  Detail & Related papers  (2023-04-28T04:32:28Z)
• Reconstruction-guided attention improves the robustness and shape processing of neural networks [5.156484100374057]
  We build an iterative encoder-decoder network that generates an object reconstruction and uses it as top-down attentional feedback. Our model shows strong generalization performance against various image perturbations. Our study shows that modeling reconstruction-based feedback endows AI systems with a powerful attention mechanism.
  arXiv  Detail & Related papers  (2022-09-27T18:32:22Z)
• Robustness and invariance properties of image classifiers [8.970032486260695]
  Deep neural networks have achieved impressive results in many image classification tasks. Deep networks are not robust to a large variety of semantic-preserving image modifications. The poor robustness of image classifiers to small data distribution shifts raises serious concerns regarding their trustworthiness.
  arXiv  Detail & Related papers  (2022-08-30T11:00:59Z)
• Deep Semantic Statistics Matching (D2SM) Denoising Network [70.01091467628068]
  We introduce the Deep Semantic Statistics Matching (D2SM) Denoising Network. It exploits semantic features of pretrained classification networks, then it implicitly matches the probabilistic distribution of clear images at the semantic feature space. By learning to preserve the semantic distribution of denoised images, we empirically find our method significantly improves the denoising capabilities of networks.
  arXiv  Detail & Related papers  (2022-07-19T14:35:42Z)
• Improving robustness against common corruptions with frequency biased models [112.65717928060195]
  unseen image corruptions can cause a surprisingly large drop in performance. Image corruption types have different characteristics in the frequency spectrum and would benefit from a targeted type of data augmentation. We propose a new regularization scheme that minimizes the total variation (TV) of convolution feature-maps to increase high-frequency robustness.
  arXiv  Detail & Related papers  (2021-03-30T10:44:50Z)
• Towards Unsupervised Deep Image Enhancement with Generative Adversarial Network [92.01145655155374]
  We present an unsupervised image enhancement generative network (UEGAN) It learns the corresponding image-to-image mapping from a set of images with desired characteristics in an unsupervised manner. Results show that the proposed model effectively improves the aesthetic quality of images.
  arXiv  Detail & Related papers  (2020-12-30T03:22:46Z)
• Interpretable Detail-Fidelity Attention Network for Single Image Super-Resolution [89.1947690981471]
  We propose a purposeful and interpretable detail-fidelity attention network to progressively process smoothes and details in divide-and-conquer manner. Particularly, we propose a Hessian filtering for interpretable feature representation which is high-profile for detail inference. Experiments demonstrate that the proposed methods achieve superior performances over the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-09-28T08:31:23Z)
• Stereopagnosia: Fooling Stereo Networks with Adversarial Perturbations [71.00754846434744]
  We show that imperceptible additive perturbations can significantly alter the disparity map. We show that, when used for adversarial data augmentation, our perturbations result in trained models that are more robust.
  arXiv  Detail & Related papers  (2020-09-21T19:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.