Interactive Visual Feature Search

• URL: http://arxiv.org/abs/2211.15060v2
• Date: Fri, 15 Dec 2023 20:43:16 GMT
• Title: Interactive Visual Feature Search
• Authors: Devon Ulrich and Ruth Fong
• Abstract summary: We introduce Visual Feature Search, a novel interactive visualization that is adaptable to any CNN. Our tool allows a user to highlight an image region and search for images from a given dataset with the most similar model features. We demonstrate how our tool elucidates different aspects of model behavior by performing experiments on a range of applications, such as in medical imaging and wildlife classification.
• Score: 8.255656003475268
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many visualization techniques have been created to explain the behavior of computer vision models, but they largely consist of static diagrams that convey limited information. Interactive visualizations allow users to more easily interpret a model's behavior, but most are not easily reusable for new models. We introduce Visual Feature Search, a novel interactive visualization that is adaptable to any CNN and can easily be incorporated into a researcher's workflow. Our tool allows a user to highlight an image region and search for images from a given dataset with the most similar model features. We demonstrate how our tool elucidates different aspects of model behavior by performing experiments on a range of applications, such as in medical imaging and wildlife classification.

Related papers

• A Multimodal Automated Interpretability Agent [63.8551718480664]
  MAIA is a system that uses neural models to automate neural model understanding tasks. We first characterize MAIA's ability to describe (neuron-level) features in learned representations of images. We then show that MAIA can aid in two additional interpretability tasks: reducing sensitivity to spurious features, and automatically identifying inputs likely to be mis-classified.
  arXiv  Detail & Related papers  (2024-04-22T17:55:11Z)
• Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensors [49.99728312519117]
  The aim of this work is to establish how accurately a recent semantic-based active perception model is able to complete visual tasks that are regularly performed by humans. This model exploits the ability of current object detectors to localize and classify a large number of object classes and to update a semantic description of a scene across multiple fixations. In the task of scene exploration, the semantic-based method demonstrates superior performance compared to the traditional saliency-based model.
  arXiv  Detail & Related papers  (2024-04-16T18:15:57Z)
• An Image-based Typology for Visualization [23.716718517642878]
  We present and discuss the results of a qualitative analysis of visual representations from images. We derive a typology of 10 visualization types of defined groups. We provide a dataset of 6,833 tagged images and an online tool that can be used to explore and analyze the large set of labeled images.
  arXiv  Detail & Related papers  (2024-03-07T04:33:42Z)
• A Vision Check-up for Language Models [61.852026871772914]
  We show how a preliminary visual representation learning system can be trained using models of text. Experiments on self-supervised visual representation learning highlight the potential to train vision models capable of making semantic assessments of natural images.
  arXiv  Detail & Related papers  (2024-01-03T18:09:33Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes [1.76602679361245]
  We present two interactive visualizations in the context of Sprite, a system for creating Computer Vision (CV) models. We study how these visualizations help Sprite's users identify (evaluate) and select (plan) images where a model is struggling and can lead to improved performance.
  arXiv  Detail & Related papers  (2023-05-19T14:43:00Z)
• Benchmarking human visual search computational models in natural scenes: models comparison and reference datasets [0.0]
  We select publicly available state-of-the-art visual search models in natural scenes and evaluate them on different datasets. We propose an improvement to the Ideal Bayesian Searcher through a combination with a neural network-based visual search model.
  arXiv  Detail & Related papers  (2021-12-10T19:56:45Z)
• An Interactive Visualization Tool for Understanding Active Learning [12.345164513513671]
  We present an interactive visualization tool to elucidate the training process of active learning. The tool enables one to select a sample of interesting data points, view how their prediction values change at different querying stages, and thus better understand when and how active learning works.
  arXiv  Detail & Related papers  (2021-11-09T03:33:26Z)
• Self-Supervised Representation Learning from Flow Equivariance [97.13056332559526]
  We present a new self-supervised learning representation framework that can be directly deployed on a video stream of complex scenes. Our representations, learned from high-resolution raw video, can be readily used for downstream tasks on static images.
  arXiv  Detail & Related papers  (2021-01-16T23:44:09Z)
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions [50.435861435121915]
  We use human interaction and attention cues to investigate whether we can learn better representations compared to visual-only representations. Our experiments show that our "muscly-supervised" representation outperforms a visual-only state-of-the-art method MoCo.
  arXiv  Detail & Related papers  (2020-10-16T17:46:53Z)
• Interactive Visual Study of Multiple Attributes Learning Model of X-Ray Scattering Images [34.95218692917125]
  We present an interactive system for domain scientists to visually study the multiple attributes learning models applied to x-ray scattering images. The exploration is guided by the manifestation of model performance related to mutual relationships among attributes. The system thus supports domain scientists to improve the training dataset and model, find questionable attributes labels, and identify outlier images or spurious data clusters.
  arXiv  Detail & Related papers  (2020-09-03T00:38:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.