Interpreting Object-level Foundation Models via Visual Precision Search

• URL: http://arxiv.org/abs/2411.16198v1
• Date: Mon, 25 Nov 2024 08:54:54 GMT
• Title: Interpreting Object-level Foundation Models via Visual Precision Search
• Authors: Ruoyu Chen, Siyuan Liang, Jingzhi Li, Shiming Liu, Maosen Li, Zheng Huang, Hua Zhang, Xiaochun Cao,
• Abstract summary: We propose a Visual Precision Search method that generates accurate attribution maps with fewer regions. Our method bypasses internal model parameters to overcome attribution issues from multimodal fusion. Our method can interpret failures in visual grounding and object detection tasks, surpassing existing methods across multiple evaluation metrics.
• Score: 53.807678972967224
• License:
• Abstract: Advances in multimodal pre-training have propelled object-level foundation models, such as Grounding DINO and Florence-2, in tasks like visual grounding and object detection. However, interpreting these models\' decisions has grown increasingly challenging. Existing interpretable attribution methods for object-level task interpretation have notable limitations: (1) gradient-based methods lack precise localization due to visual-textual fusion in foundation models, and (2) perturbation-based methods produce noisy saliency maps, limiting fine-grained interpretability. To address these, we propose a Visual Precision Search method that generates accurate attribution maps with fewer regions. Our method bypasses internal model parameters to overcome attribution issues from multimodal fusion, dividing inputs into sparse sub-regions and using consistency and collaboration scores to accurately identify critical decision-making regions. We also conducted a theoretical analysis of the boundary guarantees and scope of applicability of our method. Experiments on RefCOCO, MS COCO, and LVIS show our approach enhances object-level task interpretability over SOTA for Grounding DINO and Florence-2 across various evaluation metrics, with faithfulness gains of 23.7\%, 31.6\%, and 20.1\% on MS COCO, LVIS, and RefCOCO for Grounding DINO, and 102.9\% and 66.9\% on MS COCO and RefCOCO for Florence-2. Additionally, our method can interpret failures in visual grounding and object detection tasks, surpassing existing methods across multiple evaluation metrics. The code will be released at \url{https://github.com/RuoyuChen10/VPS}.

Related papers

• Assessing and improving reliability of neighbor embedding methods: a map-continuity perspective [2.969441406380581]
  Neighbor embedding methods, such as t-SNE, UMAP, and LargeVis, are a family of popular visualization methods. Recent studies suggest that these methods often produce visual artifacts, potentially leading to incorrect scientific conclusions. We introduce a novel conceptual and computational framework, LOO-map, that learns the embedding maps based on a classical statistical idea.
  arXiv  Detail & Related papers  (2024-10-22T01:40:43Z)
• Less is More: Fewer Interpretable Region via Submodular Subset Selection [54.07758302264416]
  This paper re-models the above image attribution problem as a submodular subset selection problem. We construct a novel submodular function to discover more accurate small interpretation regions. For correctly predicted samples, the proposed method improves the Deletion and Insertion scores with an average of 4.9% and 2.5% gain relative to HSIC-Attribution.
  arXiv  Detail & Related papers  (2024-02-14T13:30:02Z)
• Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation [84.62067728093358]
  Weakly supervised object localization and semantic segmentation aim to localize objects using only image-level labels. New paradigm has emerged by generating a foreground prediction map to achieve pixel-level localization. This paper presents two astonishing experimental observations on the object localization learning process.
  arXiv  Detail & Related papers  (2023-09-22T15:44:10Z)
• For A More Comprehensive Evaluation of 6DoF Object Pose Tracking [22.696375341994035]
  We contribute a unified benchmark to address the above problems. For more accurate annotation of YCBV, we propose a multi-view multi-object global pose refinement method. In experiments, we validate the precision and reliability of the proposed global pose refinement method with a realistic semi-synthesized dataset.
  arXiv  Detail & Related papers  (2023-09-14T15:35:08Z)
• View Consistent Purification for Accurate Cross-View Localization [59.48131378244399]
  This paper proposes a fine-grained self-localization method for outdoor robotics. The proposed method addresses limitations in existing cross-view localization methods. It is the first sparse visual-only method that enhances perception in dynamic environments.
  arXiv  Detail & Related papers  (2023-08-16T02:51:52Z)
• Boosting Few-shot Fine-grained Recognition with Background Suppression and Foreground Alignment [53.401889855278704]
  Few-shot fine-grained recognition (FS-FGR) aims to recognize novel fine-grained categories with the help of limited available samples. We propose a two-stage background suppression and foreground alignment framework, which is composed of a background activation suppression (BAS) module, a foreground object alignment (FOA) module, and a local to local (L2L) similarity metric. Experiments conducted on multiple popular fine-grained benchmarks demonstrate that our method outperforms the existing state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2022-10-04T07:54:40Z)
• Tackling Long-Tailed Category Distribution Under Domain Shifts [50.21255304847395]
  Existing approaches cannot handle the scenario where both issues exist. We designed three novel core functional blocks including Distribution Calibrated Classification Loss, Visual-Semantic Mapping and Semantic-Similarity Guided Augmentation. Two new datasets were proposed for this problem, named AWA2-LTS and ImageNet-LTS.
  arXiv  Detail & Related papers  (2022-07-20T19:07:46Z)
• Occlusion-Robust Object Pose Estimation with Holistic Representation [42.27081423489484]
  State-of-the-art (SOTA) object pose estimators take a two-stage approach. We develop a novel occlude-and-blackout batch augmentation technique. We also develop a multi-precision supervision architecture to encourage holistic pose representation learning.
  arXiv  Detail & Related papers  (2021-10-22T08:00:26Z)
• Learning Crisp Boundaries Using Deep Refinement Network and Adaptive Weighting Loss [15.867750740607864]
  Recent boundary detection models focus on "crisp" boundaries (precisely localized along the object's contour) We propose a novel network named deep refinement network (DRNet) that stacks multiple refinement modules to achieve richer feature representation and a novel loss function. Experimental results demonstrated that we achieve state-of-the-art performance for several available datasets.
  arXiv  Detail & Related papers  (2021-02-02T04:22:35Z)
• Learning Robust Representation for Clustering through Locality Preserving Variational Discriminative Network [16.259673823482665]
  Variational Deep Embedding achieves great success in various clustering tasks. VaDE suffers from two problems: 1) it is fragile to the input noise; 2) it ignores the locality information between the neighboring data points. We propose a joint learning framework that improves VaDE with a robust embedding discriminator and a local structure constraint.
  arXiv  Detail & Related papers  (2020-12-25T02:31:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.