Concept Representation Learning with Contrastive Self-Supervised Learning

• URL: http://arxiv.org/abs/2112.05677v1
• Date: Fri, 10 Dec 2021 17:16:23 GMT
• Title: Concept Representation Learning with Contrastive Self-Supervised Learning
• Authors: Daniel T. Chang
• Abstract summary: Concept-oriented deep learning (CODL) is a general approach to meet the future challenges for deep learning. We discuss major aspects of concept representation learning using Contrastive Self-supervised Learning (CSSL)
• Score: 0.6091702876917281
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept-oriented deep learning (CODL) is a general approach to meet the future challenges for deep learning: (1) learning with little or no external supervision, (2) coping with test examples that come from a different distribution than the training examples, and (3) integrating deep learning with symbolic AI. In CODL, as in human learning, concept representations are learned based on concept exemplars. Contrastive self-supervised learning (CSSL) provides a promising approach to do so, since it: (1) uses data-driven associations, to get away from semantic labels, (2) supports incremental and continual learning, to get away from (large) fixed datasets, and (3) accommodates emergent objectives, to get away from fixed objectives (tasks). We discuss major aspects of concept representation learning using CSSL. These include dual-level concept representations, CSSL for feature representations, exemplar similarity measures and self-supervised relational reasoning, incremental and continual CSSL, and contrastive self-supervised concept (class) incremental learning. The discussion leverages recent findings from cognitive neural science and CSSL.

Related papers

• Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking [154.2388970262703]
  Unified Vision-Language Models (UVLMs) aim to advance multimodal learning by supporting both understanding and generation within a single framework.<n>We introduce the interleaved Analyzing-Drafting problem-solving loop (AD-Loop), a new think paradigm that alternates between analytic and drafting operations.<n>By interleaving textual thoughts with visual thoughts, AD-Loop enables models to iteratively refine both comprehension and outputs, fostering genuine synergy.
  arXiv  Detail & Related papers  (2026-02-24T23:26:09Z)
• How Do Large Language Models Learn Concepts During Continual Pre-Training? [69.99800338599]
  We study how individual concepts are acquired and forgotten, as well as how multiple concepts interact through interference and synergy.<n>Our findings offer a circuit-level view of concept learning dynamics and inform the design of more interpretable and robust concept-aware training strategies.
  arXiv  Detail & Related papers  (2026-01-07T04:29:15Z)
• VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set [80.50996301430108]
  The alignment of vision-language representations endows current Vision-Language Models with strong multi-modal reasoning capabilities.<n>We propose VL-SAE, a sparse autoencoder that encodes vision-language representations into its hidden activations.<n>For interpretation, the alignment between vision and language representations can be understood by comparing their semantics with concepts.
  arXiv  Detail & Related papers  (2025-10-24T10:29:31Z)
• Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study [50.065744358362345]
  Large language models (LLMs) have shown impressive capabilities across tasks such as mathematics, coding, and reasoning.<n>Yet their learning ability, which is crucial for adapting to dynamic environments and acquiring new knowledge, remains underexplored.
  arXiv  Detail & Related papers  (2025-06-16T13:24:50Z)
• Language Guided Concept Bottleneck Models for Interpretable Continual Learning [62.09201360376577]
  Continual learning aims to enable learning systems to acquire new knowledge constantly without forgetting previously learned information. Most existing CL methods focus primarily on preserving learned knowledge to improve model performance. We introduce a novel framework that integrates language-guided Concept Bottleneck Models to address both challenges.
  arXiv  Detail & Related papers  (2025-03-30T02:41:55Z)
• Concept-Based Interpretable Reinforcement Learning with Limited to No Human Labels [38.05773318621547]
  We introduce a novel training scheme that enables RL algorithms to efficiently learn a concept-based policy. Our algorithm, LICORICE, involves three main contributions: interleaving concept learning and RL training, using a concept ensembles to actively select informative data points for labeling. We show how LICORICE reduces manual labeling efforts to to 500 or fewer concept labels in three environments.
  arXiv  Detail & Related papers  (2024-07-22T16:46:33Z)
• Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models [51.43538150982291]
  We study how to learn human-interpretable concepts from data. Weaving together ideas from both fields, we show that concepts can be provably recovered from diverse data.
  arXiv  Detail & Related papers  (2024-02-14T15:23:59Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Interpretable Neural-Symbolic Concept Reasoning [7.1904050674791185]
  Concept-based models aim to address this issue by learning tasks based on a set of human-understandable concepts. We propose the Deep Concept Reasoner (DCR), the first interpretable concept-based model that builds upon concept embeddings.
  arXiv  Detail & Related papers  (2023-04-27T09:58:15Z)
• Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Need [34.013568381942775]
  Self-Supervised Learning (SSL) has emerged as the solution of choice to learn transferable representations from unlabeled data. In this work, we formalize and generalize this principle through Positive Active Learning (PAL) where an oracle queries semantic relationships between samples. First, it unveils a theoretically grounded learning framework beyond SSL, based on similarity graphs, that can be extended to tackle supervised and semi-supervised learning depending on the employed oracle. Second, it provides a consistent algorithm to embed a priori knowledge, e.g. some observed labels, into any SSL losses without any change in the training pipeline.
  arXiv  Detail & Related papers  (2023-03-27T14:44:39Z)
• A Message Passing Perspective on Learning Dynamics of Contrastive Learning [60.217972614379065]
  We show that if we cast a contrastive objective equivalently into the feature space, then its learning dynamics admits an interpretable form. This perspective also establishes an intriguing connection between contrastive learning and Message Passing Graph Neural Networks (MP-GNNs)
  arXiv  Detail & Related papers  (2023-03-08T08:27:31Z)
• Exemplar-Based Contrastive Self-Supervised Learning with Few-Shot Class Incremental Learning [0.8722210937404288]
  In human learning, supervised learning of concepts based on exemplars takes place within the larger context of contrastive self-supervised learning (CSSL) based on unlabeled and labeled data. A major benefit of the extensions is that exemplar-based CSSL, with supervised finetuning, supports few-shot class incremental learning (CIL)
  arXiv  Detail & Related papers  (2022-02-05T17:14:07Z)
• A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics [131.93113552146195]
  We present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machines' capability of learning generalizable concepts. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images. We undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3.
  arXiv  Detail & Related papers  (2021-03-02T01:32:54Z)
• Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning [78.13740873213223]
  Bongard problems (BPs) were introduced as an inspirational challenge for visual cognition in intelligent systems. We propose a new benchmark Bongard-LOGO for human-level concept learning and reasoning.
  arXiv  Detail & Related papers  (2020-10-02T03:19:46Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.