Variance & Greediness: A comparative study of metric-learning losses

• URL: http://arxiv.org/abs/2601.21450v1
• Date: Thu, 29 Jan 2026 09:28:30 GMT
• Title: Variance & Greediness: A comparative study of metric-learning losses
• Authors: Donghuo Zeng, Hao Niu, Zhi Li, Masato Taya,
• Abstract summary: Metric learning is central to retrieval, yet its effects on embedding geometry and optimization dynamics are not well understood.<n>We introduce a diagnostic framework, VARIANCE (intra-/inter-class variance) and GREEDINESS (active ratio and gradient norms) to compare seven representative losses.<n>Our analysis reveals that Triplet and SCL preserve higher within-class variance and clearer inter-class margins, leading to stronger top-1 retrieval in fine-grained settings.
• Score: 5.102429604787588
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metric learning is central to retrieval, yet its effects on embedding geometry and optimization dynamics are not well understood. We introduce a diagnostic framework, VARIANCE (intra-/inter-class variance) and GREEDINESS (active ratio and gradient norms), to compare seven representative losses, i.e., Contrastive, Triplet, N-pair, InfoNCE, ArcFace, SCL, and CCL, across five image-retrieval datasets. Our analysis reveals that Triplet and SCL preserve higher within-class variance and clearer inter-class margins, leading to stronger top-1 retrieval in fine-grained settings. In contrast, Contrastive and InfoNCE compact embeddings are achieved quickly through many small updates, accelerating convergence but potentially oversimplifying class structures. N-pair achieves a large mean separation but with uneven spacing. These insights reveal a form of efficiency-granularity trade-off and provide practical guidance: prefer Triplet/SCL when diversity preservation and hard-sample discrimination are critical, and Contrastive/InfoNCE when faster embedding compaction is desired.

Related papers

• Implicit Neural Representation-Based Continuous Single Image Super Resolution: An Empirical Study [50.15623093332659]
  Implicit neural representation (INR) has become the standard approach for arbitrary-scale image super-resolution (ASSR)<n>We compare existing techniques across diverse settings and present aggregated performance results on multiple image quality metrics.<n>We examine a new loss function that penalizes intensity variations while preserving edges, textures, and finer details during training.
  arXiv  Detail & Related papers  (2026-01-25T07:09:20Z)
• Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking [56.46309219272326]
  For large language models (LLMs), classification via supervised fine-tuning (SFT) predicts ''yes'' (resp. ''no'') token for relevant (resp. irrelevant) pairs.<n>This divergence raises a central question: which objective is intrinsically better suited to LLM-based reranking, and what mechanism underlies the difference?<n>We conduct a comprehensive comparison and analysis between CL and SFT for reranking, taking the universal multimodal retrieval (UMR) as the experimental playground.
  arXiv  Detail & Related papers  (2025-10-16T16:02:27Z)
• Comparing Contrastive and Triplet Loss: Variance Analysis and Optimization Behavior [2.608092703580602]
  We show that triplet loss preserves greater variance within and across classes, supporting finer-grained distinctions in the learned representations.<n>In contrast, contrastive loss tends to compact intra-class embeddings, which may obscure subtle semantic differences.<n>We find that contrastive loss drives many small updates early on, while triplet loss produces fewer but stronger updates that sustain learning on hard examples.
  arXiv  Detail & Related papers  (2025-10-02T16:11:46Z)
• Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning [48.11265601808718]
  We show that standard self-supervised contrastive learning objectives implicitly approximate a supervised variant we call the negatives-only supervised contrastive loss (NSCL)<n>We prove that the gap between the CL and NSCL losses vanishes as the number of semantic classes increases, under a bound that is both label-agnostic and architecture-independent.
  arXiv  Detail & Related papers  (2025-06-04T19:43:36Z)
• Relaxed Contrastive Learning for Federated Learning [48.96253206661268]
  We propose a novel contrastive learning framework to address the challenges of data heterogeneity in federated learning. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks.
  arXiv  Detail & Related papers  (2024-01-10T04:55:24Z)
• Center Contrastive Loss for Metric Learning [8.433000039153407]
  We propose a novel metric learning function called Center Contrastive Loss. It maintains a class-wise center bank and compares the category centers with the query data points using a contrastive loss. The proposed loss combines the advantages of both contrastive and classification methods.
  arXiv  Detail & Related papers  (2023-08-01T11:22:51Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• 2nd Place Solution for ICCV 2021 VIPriors Image Classification Challenge: An Attract-and-Repulse Learning Approach [41.346232387426944]
  Convolutional neural networks (CNNs) have achieved significant success in image classification by utilizing large-scale datasets. We propose Attract-and-Repulse, which consists of Contrastive Regularization (CR) to enrich the feature representations, Symmetric Cross Entropy (SCE) to balance the fitting for different classes. Specifically, SCE and CR learn discriminative representations while alleviating over-fitting by the adaptive trade-off between the information of classes (attract) and instances (repulse)
  arXiv  Detail & Related papers  (2022-06-13T13:54:33Z)
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE [104.37515476361405]
  We reveal mathematically why contrastive learners fail in the small-batch-size regime. We present a novel non-native contrastive objective named FlatNCE, which fixes this issue.
  arXiv  Detail & Related papers  (2021-07-02T15:50:43Z)
• Semi-supervised Contrastive Learning with Similarity Co-calibration [72.38187308270135]
  We propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL) SsCL combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning. We show that SsCL produces more discriminative representation and is beneficial to few shot learning.
  arXiv  Detail & Related papers  (2021-05-16T09:13:56Z)
• Supporting Clustering with Contrastive Learning [19.71262627336737]
  Unsupervised clustering aims at discovering semantic categories of data according to some distance measured in the representation space. Different categories often overlap with each other in the representation space at the beginning of the learning process. We propose Supporting Clustering with Contrastive Learning -- a novel framework to leverage contrastive learning to promote better separation.
  arXiv  Detail & Related papers  (2021-03-24T03:05:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.