VFA: Vision Frequency Analysis of Foundation Models and Human

• URL: http://arxiv.org/abs/2409.05817v1
• Date: Mon, 9 Sep 2024 17:23:39 GMT
• Title: VFA: Vision Frequency Analysis of Foundation Models and Human
• Authors: Mohammad-Javad Darvishi-Bayazi, Md Rifat Arefin, Jocelyn Faubert, Irina Rish,
• Abstract summary: Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. We investigate how various characteristics of large-scale computer vision models influence their alignment with human capabilities and robustness.
• Score: 10.112417527529868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. Models that better align with human perception may achieve higher out-of-distribution generalization. In this study, we investigate how various characteristics of large-scale computer vision models influence their alignment with human capabilities and robustness. Our findings indicate that increasing model and data size and incorporating rich semantic information and multiple modalities enhance models' alignment with human perception and their overall robustness. Our empirical analysis demonstrates a strong correlation between out-of-distribution accuracy and human alignment.

Related papers

• Linking Robustness and Generalization: A k* Distribution Analysis of Concept Clustering in Latent Space for Vision Models [56.89974470863207]
  This article uses the k* Distribution, a local neighborhood analysis method, to examine the learned latent space at the level of individual concepts. We introduce skewness-based true and approximate metrics for interpreting individual concepts to assess the overall quality of vision models' latent space.
  arXiv  Detail & Related papers  (2024-08-17T01:43:51Z)
• Learning Divergence Fields for Shift-Robust Graph Representations [73.11818515795761]
  In this work, we propose a geometric diffusion model with learnable divergence fields for the challenging problem with interdependent data. We derive a new learning objective through causal inference, which can guide the model to learn generalizable patterns of interdependence that are insensitive across domains.
  arXiv  Detail & Related papers  (2024-06-07T14:29:21Z)
• MONAL: Model Autophagy Analysis for Modeling Human-AI Interactions [11.972017738888825]
  We propose Model Autophagy Analysis (MONAL) for large models' self-consumption explanation. MONAL employs two distinct autophagous loops to elucidate the suppression of human-generated information in the exchange between human and AI systems. We evaluate the capacities of generated models as both creators and disseminators of information.
  arXiv  Detail & Related papers  (2024-02-17T13:02:54Z)
• Towards Human-like Perception: Learning Structural Causal Model in Heterogeneous Graph [26.361815957385417]
  This study introduces a novel solution, HG-SCM (Heterogeneous Graph as Structural Causal Model) It can mimic the human perception and decision process through two key steps: constructing intelligible variables based on semantics derived from the graph schema and automatically learning task-level causal relationships among these variables by incorporating advanced causal discovery techniques. HG-SCM achieved the highest average performance rank with minimal standard deviation, substantiating its effectiveness and superiority in terms of both predictive power and generalizability.
  arXiv  Detail & Related papers  (2023-12-10T04:34:35Z)
• Robust Computer Vision in an Ever-Changing World: A Survey of Techniques for Tackling Distribution Shifts [20.17397328893533]
  AI applications are becoming increasingly visible to the general public. There is a notable gap between the theoretical assumptions researchers make about computer vision models and the reality those models face when deployed in the real world. One of the critical reasons for this gap is a challenging problem known as distribution shift.
  arXiv  Detail & Related papers  (2023-12-03T23:40:12Z)
• Interpretable Computer Vision Models through Adversarial Training: Unveiling the Robustness-Interpretability Connection [0.0]
  Interpretability is as essential as robustness when we deploy the models to the real world. Standard models, compared to robust are more susceptible to adversarial attacks, and their learned representations are less meaningful to humans.
  arXiv  Detail & Related papers  (2023-07-04T13:51:55Z)
• Adaptive Contextual Perception: How to Generalize to New Backgrounds and Ambiguous Objects [75.15563723169234]
  We investigate how vision models adaptively use context for out-of-distribution generalization. We show that models that excel in one setting tend to struggle in the other. To replicate the generalization abilities of biological vision, computer vision models must have factorized object vs. background representations.
  arXiv  Detail & Related papers  (2023-06-09T15:29:54Z)
• Causal Reasoning Meets Visual Representation Learning: A Prospective Study [117.08431221482638]
  Lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models. Inspired by the strong inference ability of human-level agents, recent years have witnessed great effort in developing causal reasoning paradigms. This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods.
  arXiv  Detail & Related papers  (2022-04-26T02:22:28Z)
• DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations [119.1953397679783]
  We focus on advancing the state-of-the-art in interpreting multimodal models. Our proposed approach, DIME, enables accurate and fine-grained analysis of multimodal models.
  arXiv  Detail & Related papers  (2022-03-03T20:52:47Z)
• Exploring Alignment of Representations with Human Perception [47.53970721813083]
  We show that inputs that are mapped to similar representations by the model should be perceived similarly by humans. Our approach yields a measure of the extent to which a model is aligned with human perception. We find that various properties of a model like its architecture, training paradigm, training loss, and data augmentation play a significant role in learning representations that are aligned with human perception.
  arXiv  Detail & Related papers  (2021-11-29T17:26:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.