Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning

• URL: http://arxiv.org/abs/2509.04734v1
• Date: Fri, 05 Sep 2025 01:23:59 GMT
• Title: Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning
• Authors: Jasmine Shone, Shaden Alshammari, Mark Hamilton, Zhening Li, William Freeman,
• Abstract summary: We present Beyond I-Con, a framework that enables systematic discovery of novel loss functions.<n>Our results highlight the importance of considering divergence and similarity kernel choices in representation learning optimization.
• Score: 7.8851393122408515
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Information Contrastive (I-Con) framework revealed that over 23 representation learning methods implicitly minimize KL divergence between data and learned distributions that encode similarities between data points. However, a KL-based loss may be misaligned with the true objective, and properties of KL divergence such as asymmetry and unboundedness may create optimization challenges. We present Beyond I-Con, a framework that enables systematic discovery of novel loss functions by exploring alternative statistical divergences and similarity kernels. Key findings: (1) on unsupervised clustering of DINO-ViT embeddings, we achieve state-of-the-art results by modifying the PMI algorithm to use total variation (TV) distance; (2) on supervised contrastive learning, we outperform the standard approach by using TV and a distance-based similarity kernel instead of KL and an angular kernel; (3) on dimensionality reduction, we achieve superior qualitative results and better performance on downstream tasks than SNE by replacing KL with a bounded f-divergence. Our results highlight the importance of considering divergence and similarity kernel choices in representation learning optimization.

Related papers

• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• DIS2: Disentanglement Meets Distillation with Classwise Attention for Robust Remote Sensing Segmentation under Missing Modalities [28.992992584085787]
  DIS2 is a new paradigm shifting from modality-shared feature dependence to active, guided missing features compensation.<n> Compensatory features are explicitly captured which, when fused with the features of the available modality, approximate the ideal fused representation of the full-modality case.<n>Our proposed approach significantly outperforms state-of-the-art methods across benchmarks.
  arXiv  Detail & Related papers  (2026-01-20T01:33:54Z)
• Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning [4.946476970294861]
  Mutual Information is a fundamental measure of statistical dependence widely used in representation learning.<n>We derive a new, tight, and tractable lower bound on KLD as a function of JSD in the general case.<n>Our results provide new theoretical justifications and strong empirical evidence for using discriminative learning in MI-based representation learning.
  arXiv  Detail & Related papers  (2025-10-23T15:18:12Z)
• Understanding Catastrophic Interference: On the Identifibility of Latent Representations [67.05452287233122]
  Catastrophic interference, also known as catastrophic forgetting, is a fundamental challenge in machine learning.<n>We propose a novel theoretical framework that formulates catastrophic interference as an identification problem.<n>Our approach provides both theoretical guarantees and practical performance improvements across both synthetic and benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-27T00:53:32Z)
• FAIRWELL: Fair Multimodal Self-Supervised Learning for Wellbeing Prediction [16.913286294651236]
  We propose a novel subject-level loss function to learn fairer representations via the following three mechanisms.<n>We evaluate our method on three challenging real-world heterogeneous healthcare datasets.
  arXiv  Detail & Related papers  (2025-08-22T19:03:06Z)
• Generalized Kullback-Leibler Divergence Loss [105.66549870868971]
  We prove that the Kullback-Leibler (KL) Divergence loss is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss.<n>Thanks to the decoupled structure of DKL loss, we have identified two areas for improvement.
  arXiv  Detail & Related papers  (2025-03-11T04:43:33Z)
• Your contrastive learning problem is secretly a distribution alignment problem [11.75699373180322]
  We build connections between noise contrastive estimation losses widely used in vision and distribution alignment.<n>By using more information from the distribution of latents, our approach allows a more distribution-aware manipulation of the relationships within augmented sample sets.
  arXiv  Detail & Related papers  (2025-02-27T14:33:08Z)
• Revisiting Self-Supervised Heterogeneous Graph Learning from Spectral Clustering Perspective [52.662463893268225]
  Self-supervised heterogeneous graph learning (SHGL) has shown promising potential in diverse scenarios.<n>Existing SHGL methods encounter two significant limitations.<n>We introduce a novel framework enhanced by rank and dual consistency constraints.
  arXiv  Detail & Related papers  (2024-12-01T09:33:20Z)
• SimO Loss: Anchor-Free Contrastive Loss for Fine-Grained Supervised Contrastive Learning [0.0]
  We introduce a novel anchor-free contrastive learning (L) method leveraging our proposed Similarity-Orthogonality (SimO) loss. Our approach minimizes a semi-metric discriminative loss function that simultaneously optimize two key objectives. We provide visualizations that demonstrate the impact of SimO loss on the embedding space.
  arXiv  Detail & Related papers  (2024-10-07T17:41:10Z)
• Relaxed Contrastive Learning for Federated Learning [48.96253206661268]
  We propose a novel contrastive learning framework to address the challenges of data heterogeneity in federated learning. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks.
  arXiv  Detail & Related papers  (2024-01-10T04:55:24Z)
• Learning Invariant Representation via Contrastive Feature Alignment for Clutter Robust SAR Target Recognition [10.993101256393679]
  This letter proposes a solution called Contrastive Feature Alignment (CFA) to learn invariant representation for robust recognition. CFA combines both classification and CWMSE losses to train the model jointly. The proposed CFA combines both classification and CWMSE losses to train the model jointly, which allows for the progressive learning of invariant target representation.
  arXiv  Detail & Related papers  (2023-04-04T12:35:33Z)
• Mixed Graph Contrastive Network for Semi-Supervised Node Classification [63.924129159538076]
  We propose a novel graph contrastive learning method, termed Mixed Graph Contrastive Network (MGCN)<n>In our method, we improve the discriminative capability of the latent embeddings by an unperturbed augmentation strategy and a correlation reduction mechanism.<n>By combining the two settings, we extract rich supervision information from both the abundant nodes and the rare yet valuable labeled nodes for discriminative representation learning.
  arXiv  Detail & Related papers  (2022-06-06T14:26:34Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.