Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness

• URL: http://arxiv.org/abs/2407.20845v1
• Date: Tue, 30 Jul 2024 14:22:13 GMT
• Title: Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness
• Authors: Soohyun Lee, Minsuk Chang, Seokhyeon Park, Jinwook Seo,
• Abstract summary: We introduce a novel evaluation framework to assess the graphical perception of image embedding models. For chart comprehension, we examine two main aspects of channel effectiveness: accuracy and discriminability of various visual channels. Experiments with the CLIP model show that it perceives channel accuracy differently from humans and shows unique discriminability in channels like length, tilt, and curvature.
• Score: 20.269583912221734
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in vision models have greatly improved their ability to handle complex chart understanding tasks, like chart captioning and question answering. However, it remains challenging to assess how these models process charts. Existing benchmarks only roughly evaluate model performance without evaluating the underlying mechanisms, such as how models extract image embeddings. This limits our understanding of the model's ability to perceive fundamental graphical components. To address this, we introduce a novel evaluation framework to assess the graphical perception of image embedding models. For chart comprehension, we examine two main aspects of channel effectiveness: accuracy and discriminability of various visual channels. Channel accuracy is assessed through the linearity of embeddings, measuring how well the perceived magnitude aligns with the size of the stimulus. Discriminability is evaluated based on the distances between embeddings, indicating their distinctness. Our experiments with the CLIP model show that it perceives channel accuracy differently from humans and shows unique discriminability in channels like length, tilt, and curvature. We aim to develop this work into a broader benchmark for reliable visual encoders, enhancing models for precise chart comprehension and human-like perception in future applications.

Related papers

• COSE: A Consistency-Sensitivity Metric for Saliency on Image Classification [21.3855970055692]
  We present a set of metrics that utilize vision priors to assess the performance of saliency methods on image classification tasks. We show that although saliency methods are thought to be architecture-independent, most methods could better explain transformer-based models over convolutional-based models.
  arXiv  Detail & Related papers  (2023-09-20T01:06:44Z)
• Distance-Aware eXplanation Based Learning [5.578004730855819]
  We present a method to add a distance-aware explanation loss to categorical losses that trains a learner to focus on important regions of a training dataset. In addition to assessing our model using existing metrics, we propose an interpretability metric for evaluating visual feature-attribution based model explanations.
  arXiv  Detail & Related papers  (2023-09-11T15:33:00Z)
• CorrEmbed: Evaluating Pre-trained Model Image Similarity Efficacy with a Novel Metric [6.904776368895614]
  We evaluate the viability of the image embeddings from pre-trained computer vision models using a novel approach named CorrEmbed. Our approach computes the correlation between distances in image embeddings and distances in human-generated tag vectors. Our method also identifies deviations from this pattern, providing insights into how different models capture high-level image features.
  arXiv  Detail & Related papers  (2023-08-30T16:23:07Z)
• A Control-Centric Benchmark for Video Prediction [69.22614362800692]
  We propose a benchmark for action-conditioned video prediction in the form of a control benchmark. Our benchmark includes simulated environments with 11 task categories and 310 task instance definitions. We then leverage our benchmark to study the effects of scaling model size, quantity of training data, and model ensembling.
  arXiv  Detail & Related papers  (2023-04-26T17:59:45Z)
• Robustifying Deep Vision Models Through Shape Sensitization [19.118696557797957]
  We propose a simple, lightweight adversarial augmentation technique that explicitly incentivizes the network to learn holistic shapes. Our augmentations superpose edgemaps from one image onto another image with shuffled patches, using a randomly determined mixing proportion. We show that our augmentations significantly improve classification accuracy and robustness measures on a range of datasets and neural architectures.
  arXiv  Detail & Related papers  (2022-11-14T11:17:46Z)
• A Graph-Enhanced Click Model for Web Search [67.27218481132185]
  We propose a novel graph-enhanced click model (GraphCM) for web search. We exploit both intra-session and inter-session information for the sparsity and cold-start problems.
  arXiv  Detail & Related papers  (2022-06-17T08:32:43Z)
• Bayesian Graph Contrastive Learning [55.36652660268726]
  We propose a novel perspective of graph contrastive learning methods showing random augmentations leads to encoders. Our proposed method represents each node by a distribution in the latent space in contrast to existing techniques which embed each node to a deterministic vector. We show a considerable improvement in performance compared to existing state-of-the-art methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-12-15T01:45:32Z)
• A Multi-Level Attention Model for Evidence-Based Fact Checking [58.95413968110558]
  We present a simple model that can be trained on sequence structures. Results on a large-scale dataset for Fact Extraction and VERification show that our model outperforms the graph-based approaches.
  arXiv  Detail & Related papers  (2021-06-02T05:40:12Z)
• Visual Distant Supervision for Scene Graph Generation [66.10579690929623]
  Scene graph models usually require supervised learning on large quantities of labeled data with intensive human annotation. We propose visual distant supervision, a novel paradigm of visual relation learning, which can train scene graph models without any human-labeled data. Comprehensive experimental results show that our distantly supervised model outperforms strong weakly supervised and semi-supervised baselines.
  arXiv  Detail & Related papers  (2021-03-29T06:35:24Z)
• Model-Agnostic Graph Regularization for Few-Shot Learning [60.64531995451357]
  We present a comprehensive study on graph embedded few-shot learning. We introduce a graph regularization approach that allows a deeper understanding of the impact of incorporating graph information between labels. Our approach improves the performance of strong base learners by up to 2% on Mini-ImageNet and 6.7% on ImageNet-FS.
  arXiv  Detail & Related papers  (2021-02-14T05:28:13Z)
• An application of a pseudo-parabolic modeling to texture image recognition [0.0]
  We present a novel methodology for texture image recognition using a partial differential equation modeling. We employ the pseudo-parabolic Buckley-Leverett equation to provide a dynamics to the digital image representation and collect local descriptors from those images evolving in time.
  arXiv  Detail & Related papers  (2021-02-09T18:08:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.