Learning Human-Aligned Representations with Contrastive Learning and Generative Similarity

• URL: http://arxiv.org/abs/2405.19420v3
• Date: Fri, 31 Jan 2025 16:19:24 GMT
• Title: Learning Human-Aligned Representations with Contrastive Learning and Generative Similarity
• Authors: Raja Marjieh, Sreejan Kumar, Declan Campbell, Liyi Zhang, Gianluca Bencomo, Jake Snell, Thomas L. Griffiths,
• Abstract summary: Humans rely on effective representations to learn from few examples and abstract useful information from sensory data. We use a Bayesian notion of generative similarity whereby two data points are considered similar if they are likely to have been sampled from the same distribution. We demonstrate the utility of our approach by showing that it can be used to capture human-like representations of shape regularity, abstract Euclidean geometric concepts, and semantic hierarchies for natural images.
• Score: 9.63129238638334
• License:
• Abstract: Humans rely on effective representations to learn from few examples and abstract useful information from sensory data. Inducing such representations in machine learning models has been shown to improve their performance on various benchmarks such as few-shot learning and robustness. However, finding effective training procedures to achieve that goal can be challenging as psychologically rich training data such as human similarity judgments are expensive to scale, and Bayesian models of human inductive biases are often intractable for complex, realistic domains. Here, we address this challenge by leveraging a Bayesian notion of generative similarity whereby two data points are considered similar if they are likely to have been sampled from the same distribution. This measure can be applied to complex generative processes, including probabilistic programs. We incorporate generative similarity into a contrastive learning objective to enable learning of embeddings that express human cognitive representations. We demonstrate the utility of our approach by showing that it can be used to capture human-like representations of shape regularity, abstract Euclidean geometric concepts, and semantic hierarchies for natural images.

Related papers

• Turing Representational Similarity Analysis (RSA): A Flexible Method for Measuring Alignment Between Human and Artificial Intelligence [0.62914438169038]
  We developed Turing Representational Similarity Analysis (RSA), a method that uses pairwise similarity ratings to quantify alignment between AIs and humans. We tested this approach on semantic alignment across text and image modalities, measuring how different Large Language and Vision Language Model (LLM and VLM) similarity judgments aligned with human responses at both group and individual levels.
  arXiv  Detail & Related papers  (2024-11-30T20:24:52Z)
• Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models [80.32412260877628]
  We study how to learn human-interpretable concepts from data. Weaving together ideas from both fields, we show that concepts can be provably recovered from diverse data.
  arXiv  Detail & Related papers  (2024-02-14T15:23:59Z)
• Evaluating alignment between humans and neural network representations in image-based learning tasks [5.657101730705275]
  We tested how well the representations of $86$ pretrained neural network models mapped to human learning trajectories. We found that while training dataset size was a core determinant of alignment with human choices, contrastive training with multi-modal data (text and imagery) was a common feature of currently publicly available models that predicted human generalisation. In conclusion, pretrained neural networks can serve to extract representations for cognitive models, as they appear to capture some fundamental aspects of cognition that are transferable across tasks.
  arXiv  Detail & Related papers  (2023-06-15T08:18:29Z)
• Predicting Human Similarity Judgments Using Large Language Models [13.33450619901885]
  We propose an efficient procedure for predicting similarity judgments based on text descriptions. The number of descriptions required grows only linearly with the number of stimuli, drastically reducing the amount of data required. We test this procedure on six datasets of naturalistic images and show that our models outperform previous approaches based on visual information.
  arXiv  Detail & Related papers  (2022-02-09T21:09:25Z)
• On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation [9.848635287149355]
  Most self-supervised methods encourage the system to learn an invariant representation of different transformations of the same image. In this paper, we attempt to reverse-engineer these augmentations to be more biologically or perceptually plausible. We find that random cropping can be substituted by cortical magnification, and saccade-like sampling of the image could also assist the representation learning.
  arXiv  Detail & Related papers  (2021-12-14T05:38:26Z)
• Exploring Alignment of Representations with Human Perception [47.53970721813083]
  We show that inputs that are mapped to similar representations by the model should be perceived similarly by humans. Our approach yields a measure of the extent to which a model is aligned with human perception. We find that various properties of a model like its architecture, training paradigm, training loss, and data augmentation play a significant role in learning representations that are aligned with human perception.
  arXiv  Detail & Related papers  (2021-11-29T17:26:50Z)
• Action similarity judgment based on kinematic primitives [48.99831733355487]
  We investigate to which extent a computational model based on kinematics can determine action similarity. The chosen model has its roots in developmental robotics and performs action classification based on learned kinematic primitives. The results show that both the model and human performance are highly accurate in an action similarity task based on kinematic-level features.
  arXiv  Detail & Related papers  (2020-08-30T13:58:47Z)
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning [141.2562447971]
  We propose to solve a few-shot (or low-shot) visual reasoning problem, by resorting to analogical reasoning. We extract structural relationships between elements in both domains, and enforce them to be as similar as possible with analogical learning. We validate our method on RAVEN dataset, on which it outperforms state-of-the-art method, with larger gains when the training data is scarce.
  arXiv  Detail & Related papers  (2020-07-23T14:00:34Z)
• Human Trajectory Forecasting in Crowds: A Deep Learning Perspective [89.4600982169]
  We present an in-depth analysis of existing deep learning-based methods for modelling social interactions. We propose two knowledge-based data-driven methods to effectively capture these social interactions. We develop a large scale interaction-centric benchmark TrajNet++, a significant yet missing component in the field of human trajectory forecasting.
  arXiv  Detail & Related papers  (2020-07-07T17:19:56Z)
• Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples [84.8370546614042]
  Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data. Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model. Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
  arXiv  Detail & Related papers  (2020-03-25T11:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.