SCoRe: Submodular Combinatorial Representation Learning

• URL: http://arxiv.org/abs/2310.00165v2
• Date: Thu, 6 Jun 2024 06:33:47 GMT
• Title: SCoRe: Submodular Combinatorial Representation Learning
• Authors: Anay Majee, Suraj Kothawade, Krishnateja Killamsetty, Rishabh Iyer,
• Abstract summary: We introduce the SCoRe (Submodular Combinatorial Representation) framework, a novel approach in representation learning. SCoRe provides a new viewpoint to representation learning, by introducing a family of loss functions based on set-based submodular information measures.
• Score: 12.874523233023453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we introduce the SCoRe (Submodular Combinatorial Representation Learning) framework, a novel approach in representation learning that addresses inter-class bias and intra-class variance. SCoRe provides a new combinatorial viewpoint to representation learning, by introducing a family of loss functions based on set-based submodular information measures. We develop two novel combinatorial formulations for loss functions, using the Total Information and Total Correlation, that naturally minimize intra-class variance and inter-class bias. Several commonly used metric/contrastive learning loss functions like supervised contrastive loss, orthogonal projection loss, and N-pairs loss, are all instances of SCoRe, thereby underlining the versatility and applicability of SCoRe in a broad spectrum of learning scenarios. Novel objectives in SCoRe naturally model class-imbalance with up to 7.6\% improvement in classification on CIFAR-10-LT, CIFAR-100-LT, MedMNIST, 2.1% on ImageNet-LT, and 19.4% in object detection on IDD and LVIS (v1.0), demonstrating its effectiveness over existing approaches.

Related papers

• Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
  Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance. We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features. In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
  arXiv  Detail & Related papers  (2024-06-12T08:30:16Z)
• An Information Compensation Framework for Zero-Shot Skeleton-based Action Recognition [49.45660055499103]
  Zero-shot human skeleton-based action recognition aims to construct a model that can recognize actions outside the categories seen during training. Previous research has focused on aligning sequences' visual and semantic spatial distributions. We introduce a new loss function sampling method to obtain a tight and robust representation.
  arXiv  Detail & Related papers  (2024-06-02T06:53:01Z)
• Bayesian Learning-driven Prototypical Contrastive Loss for Class-Incremental Learning [42.14439854721613]
  We propose a prototypical network with a Bayesian learning-driven contrastive loss (BLCL) tailored specifically for class-incremental learning scenarios. Our approach dynamically adapts the balance between the cross-entropy and contrastive loss functions with a Bayesian learning technique.
  arXiv  Detail & Related papers  (2024-05-17T19:49:02Z)
• Class Anchor Margin Loss for Content-Based Image Retrieval [97.81742911657497]
  We propose a novel repeller-attractor loss that falls in the metric learning paradigm, yet directly optimize for the L2 metric without the need of generating pairs. We evaluate the proposed objective in the context of few-shot and full-set training on the CBIR task, by using both convolutional and transformer architectures.
  arXiv  Detail & Related papers  (2023-06-01T12:53:10Z)
• Continual Contrastive Finetuning Improves Low-Resource Relation Extraction [34.76128090845668]
  Relation extraction has been particularly challenging in low-resource scenarios and domains. Recent literature has tackled low-resource RE by self-supervised learning. We propose to pretrain and finetune the RE model using consistent objectives of contrastive learning.
  arXiv  Detail & Related papers  (2022-12-21T07:30:22Z)
• Self-Supervised Class Incremental Learning [51.62542103481908]
  Existing Class Incremental Learning (CIL) methods are based on a supervised classification framework sensitive to data labels. When updating them based on the new class data, they suffer from catastrophic forgetting: the model cannot discern old class data clearly from the new. In this paper, we explore the performance of Self-Supervised representation learning in Class Incremental Learning (SSCIL) for the first time.
  arXiv  Detail & Related papers  (2021-11-18T06:58:19Z)
• ScatSimCLR: self-supervised contrastive learning with pretext task regularization for small-scale datasets [5.2424255020469595]
  We consider a problem of self-supervised learning for small-scale datasets based on contrastive loss between multiple views of the data. We argue that the number of parameters of the whole system and the number of views can be considerably reduced while preserving the same classification accuracy.
  arXiv  Detail & Related papers  (2021-08-31T15:58:45Z)
• Learning by Minimizing the Sum of Ranked Range [58.24935359348289]
  We introduce the sum of ranked range (SoRR) as a general approach to form learning objectives. A ranked range is a consecutive sequence of sorted values of a set of real numbers. We explore two applications in machine learning of the minimization of the SoRR framework, namely the AoRR aggregate loss for binary classification and the TKML individual loss for multi-label/multi-class classification.
  arXiv  Detail & Related papers  (2020-10-05T01:58:32Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)
• Interpolation-based semi-supervised learning for object detection [44.37685664440632]
  We propose an Interpolation-based Semi-supervised learning method for object detection. The proposed losses dramatically improve the performance of semi-supervised learning as well as supervised learning.
  arXiv  Detail & Related papers  (2020-06-03T10:53:44Z)
• FairNN- Conjoint Learning of Fair Representations for Fair Decisions [40.05268461544044]
  We propose FairNN a neural network that performs joint feature representation and classification for fairness-aware learning. Our experiments on a variety of datasets demonstrate that such a joint approach is superior to separate treatment of unfairness in representation learning or supervised learning.
  arXiv  Detail & Related papers  (2020-04-05T12:08:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.