Non-negative Contrastive Learning

• URL: http://arxiv.org/abs/2403.12459v3
• Date: Mon, 22 Apr 2024 21:28:17 GMT
• Title: Non-negative Contrastive Learning
• Abstract summary: Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant challenge. We propose Non-negative Contrastive Learning to deriving interpretable features.
• Score: 34.857412784865645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep representations have shown promising performance when transferred to downstream tasks in a black-box manner. Yet, their inherent lack of interpretability remains a significant challenge, as these features are often opaque to human understanding. In this paper, we propose Non-negative Contrastive Learning (NCL), a renaissance of Non-negative Matrix Factorization (NMF) aimed at deriving interpretable features. The power of NCL lies in its enforcement of non-negativity constraints on features, reminiscent of NMF's capability to extract features that align closely with sample clusters. NCL not only aligns mathematically well with an NMF objective but also preserves NMF's interpretability attributes, resulting in a more sparse and disentangled representation compared to standard contrastive learning (CL). Theoretically, we establish guarantees on the identifiability and downstream generalization of NCL. Empirically, we show that these advantages enable NCL to outperform CL significantly on feature disentanglement, feature selection, as well as downstream classification tasks. At last, we show that NCL can be easily extended to other learning scenarios and benefit supervised learning as well. Code is available at https://github.com/PKU-ML/non_neg.

Related papers

• Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization [1.5647246115099227]
  Interpretable Mesomorphic Neural Networks (IMNs) offer a promising framework that combines the predictive power of deep neural networks with the interpretability of linear models.<n>We introduce Local Fidelity Regularization (LFR), a novel penalty term that prevents degenerate weight collapse.<n>This structural constraint guarantees faithful explanations and substantially improves the reliability of model interpretations.
  arXiv  Detail & Related papers  (2026-06-29T08:26:20Z)
• On the Properties of Feature Attribution for Supervised Contrastive Learning [0.0]
  Most Neural Networks (NNs) for classification are trained using Cross-Entropy as a loss function.<n> Contrastive Learning (CL) has the NN produce an embedding space where projections of similar data are pulled together.<n>Supervised CL (SCL) provides crucial advantages over CE robustness with regard to adversarial and out-of-distribution detection.
  arXiv  Detail & Related papers  (2026-04-24T13:32:43Z)
• Negative Metric Learning for Graphs [3.67824041145419]
  Graph contrastive learning (GCL) often suffers from false negatives, which degrades the performance on downstream tasks.<n>We propose a novel Negative Metric Learning (NML) enhanced GCL (NML-GCL)<n>NML-GCL employs a learnable Negative Metric Network (NMN) to build a negative metric space, in which false negatives can be distinguished better from true negatives.
  arXiv  Detail & Related papers  (2025-05-15T13:53:48Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder. Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder. Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP [57.33324843049638]
  We introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions.<n>Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality.<n>Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately.
  arXiv  Detail & Related papers  (2025-01-19T01:17:05Z)
• CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMs [1.515687944002438]
  We propose Contrastive Semantic Similarity, a module to obtain similarity features for measuring uncertainty for text pairs. We conduct extensive experiments with three large language models (LLMs) on several benchmark question-answering datasets. Results show that our proposed method performs better in estimating reliable responses of LLMs than comparable baselines.
  arXiv  Detail & Related papers  (2024-06-05T11:35:44Z)
• Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations [43.484570564890866]
  Existing vision-language models (VLMs) treat text descriptions as a unit, confusing individual concepts in a prompt. We present CC-Neg, a dataset containing 228,246 images, true captions and their corresponding negated captions. Using CC-Neg along with modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework, has an improved understanding of negations.
  arXiv  Detail & Related papers  (2024-03-29T17:33:42Z)
• RecDCL: Dual Contrastive Learning for Recommendation [65.6236784430981]
  We propose a dual contrastive learning recommendation framework -- RecDCL. In RecDCL, the FCL objective is designed to eliminate redundant solutions on user-item positive pairs. The BCL objective is utilized to generate contrastive embeddings on output vectors for enhancing the robustness of the representations.
  arXiv  Detail & Related papers  (2024-01-28T11:51:09Z)
• An NMF-Based Building Block for Interpretable Neural Networks With Continual Learning [0.8158530638728501]
  Existing learning methods often struggle to balance interpretability and predictive performance. Our approach aims to strike a better balance between these two aspects through the use of a building block based on NMF.
  arXiv  Detail & Related papers  (2023-11-20T02:00:33Z)
• NPCL: Neural Processes for Uncertainty-Aware Continual Learning [26.642662729915234]
  Continual learning (CL) aims to train deep neural networks efficiently on streaming data while limiting the forgetting caused by new tasks. We propose handling CL tasks with neural processes (NPs), a class of meta-learners that encode different tasks into probabilistic distributions over functions.
  arXiv  Detail & Related papers  (2023-10-30T05:10:00Z)
• Understanding Emergent In-Context Learning from a Kernel Regression Perspective [55.95455089638838]
  Large language models (LLMs) have initiated a paradigm shift in transfer learning.<n>This paper proposes a kernel-regression perspective of understanding LLMs' ICL bahaviors when faced with in-context examples.<n>We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression.
  arXiv  Detail & Related papers  (2023-05-22T06:45:02Z)
• Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization [59.77647907277523]
  Adversarial contrastive learning (ACL) is a technique that enhances standard contrastive learning (SCL) In this paper, we propose adversarial invariant regularization (AIR) to enforce independence from style factors.
  arXiv  Detail & Related papers  (2023-04-30T03:12:21Z)
• Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods [61.49061000562676]
  We introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN trained to predict labels obtained by clustering the representations with K-means. We find that CL better correlates with in-distribution model performance than other competing recent evaluation schemes.
  arXiv  Detail & Related papers  (2022-06-02T19:05:13Z)
• Nested Collaborative Learning for Long-Tailed Visual Recognition [71.6074806468641]
  NCL consists of two core components, namely Nested Individual Learning (NIL) and Nested Balanced Online Distillation (NBOD) To learn representations more thoroughly, both NIL and NBOD are formulated in a nested way, in which the learning is conducted on not just all categories from a full perspective but some hard categories from a partial perspective. In the NCL, the learning from two perspectives is nested, highly related and complementary, and helps the network to capture not only global and robust features but also meticulous distinguishing ability.
  arXiv  Detail & Related papers  (2022-03-29T08:55:39Z)
• Structural and Semantic Contrastive Learning for Self-supervised Node Representation Learning [32.126228702554144]
  Graph Contrastive Learning (GCL) has drawn much research interest for learning generalizable, transferable, and robust node representations in a self-supervised fashion. In this work, we go beyond the existing unsupervised GCL counterparts and address their limitations by proposing a simple yet effective framework S$3$-CL. Our experiments demonstrate that the node representations learned by S$3$-CL achieve superior performance on different downstream tasks compared to the state-of-the-art GCL methods.
  arXiv  Detail & Related papers  (2022-02-17T07:20:09Z)
• Continual Learning in Recurrent Neural Networks [67.05499844830231]
  We evaluate the effectiveness of continual learning methods for processing sequential data with recurrent neural networks (RNNs) We shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs. We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements.
  arXiv  Detail & Related papers  (2020-06-22T10:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.