KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models

• URL: http://arxiv.org/abs/2606.04180v1
• Date: Tue, 02 Jun 2026 19:47:41 GMT
• Title: KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
• Abstract summary: kernel-based framework for contrastive representation comparison and alignment.<n>We develop randomized low-dimensional approximations of joint kernels using random projections.<n> Empirically, KODA identifies consistent and interpretable discrepancy structures across vision-language representations.
• Score: 18.957478338649114
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language foundation models such as CLIP and SigLIP provide widely used representations for multimodal learning systems. While these models are typically compared through downstream performance, such evaluations often do not explain how their representations differ structurally. In this work, we study this problem through the task of Contrastive Embedding Clustering: identifying sample subsets that are weakly clustered under one representation but strongly clustered under another. We propose \emph{Kernel Optimization for Discrepancy Analysis (KODA)}, a kernel-based framework for contrastive representation comparison and alignment. KODA constructs unified multimodal kernels through modality-wise kernel composition and formulates discrepancy discovery as a constrained optimization problem that searches for coherent structures in one representation while suppressing coherence in a reference representation. This yields interpretable discrepancy directions associated with specific sample subsets and modality interactions. To scale KODA to large vision-language datasets, we develop randomized low-dimensional approximations of joint kernels using random projections, including Random Fourier Features for shift-invariant kernels. Empirically, KODA identifies consistent and interpretable discrepancy structures across vision-language representations and provides sample subsets for representation alignment. The code is available at https://github.com/yokiwuuu/KODA.

Related papers

• Semantic-based Distributed Learning for Diverse and Discriminative Representations [26.214742067718277]
  In large-scale distributed scenarios, increasingly complex tasks demand more intelligent collaboration across networks.<n>We propose a novel distributed learning framework that ensures both diverse and discriminative representations.<n>In both cases, the resulting optimal solutions are theoretically proven to maintain discriminative and diverse properties.
  arXiv  Detail & Related papers  (2026-04-20T13:22:58Z)
• Enhancing Clustering: An Explainable Approach via Filtered Patterns [4.576379639081977]
  explainable clustering is a knowledge-driven unsupervised learning paradigm that partitions data into $$ disjoint clusters.<n>Recent work improved clustering quality by introducing identical frequent patterns (k-relaxed patterns)<n>This approach suffers from a critical limitation: multiple distinct k-covers may induce identical k-covers, leading to redundant symbolic representations.<n>We propose an optimization strategy that removes redundant patterns by retaining a single representative pattern for each distinct k-cover.
  arXiv  Detail & Related papers  (2026-04-14T08:45:38Z)
• Weight-Informed Self-Explaining Clustering for Mixed-Type Tabular Data [63.62853416081748]
  WISE is a framework that unifies representation, feature weighting, clustering, and interpretation.<n>It produces faithful, human-interpretable explanations grounded in the same primitives that drive clustering.
  arXiv  Detail & Related papers  (2026-04-07T13:18:31Z)
• Towards an Explainable Comparison and Alignment of Feature Embeddings [8.056359341994941]
  We propose the SPEC framework to compare embeddings and identify their differences in clustering a reference dataset.<n>We present a scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size.<n>We provide numerical results demonstrating the SPEC's application to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO.
  arXiv  Detail & Related papers  (2025-06-06T16:50:37Z)
• MOKD: Cross-domain Finetuning for Few-shot Classification via Maximizing Optimized Kernel Dependence [97.93517982908007]
  In cross-domain few-shot classification, NCC aims to learn representations to construct a metric space where few-shot classification can be performed. In this paper, we find that there exist high similarities between NCC-learned representations of two samples from different classes. We propose a bi-level optimization framework, emphmaximizing optimized kernel dependence (MOKD) to learn a set of class-specific representations that match the cluster structures indicated by labeled data.
  arXiv  Detail & Related papers  (2024-05-29T05:59:52Z)
• Self Supervised Correlation-based Permutations for Multi-View Clustering [7.093692674858257]
  We propose an end-to-end deep learning-based multi-view clustering framework for general data types.<n>Our approach involves generating meaningful fused representations using a novel permutation-based canonical correlation objective.
  arXiv  Detail & Related papers  (2024-02-26T08:08:30Z)
• Multivariate Representation Learning for Information Retrieval [31.31440742912932]
  We propose a new representation learning framework for dense retrieval. Instead of learning a vector for each query and document, our framework learns a multivariate distribution. We show that it can be seamlessly integrated into the existing approximate nearest neighbor algorithms.
  arXiv  Detail & Related papers  (2023-04-27T20:30:46Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Adaptively-weighted Integral Space for Fast Multiview Clustering [54.177846260063966]
  We propose an Adaptively-weighted Integral Space for Fast Multiview Clustering (AIMC) with nearly linear complexity. Specifically, view generation models are designed to reconstruct the view observations from the latent integral space. Experiments conducted on several realworld datasets confirm the superiority of the proposed AIMC method.
  arXiv  Detail & Related papers  (2022-08-25T05:47:39Z)
• Out-of-distribution Generalization via Partial Feature Decorrelation [72.96261704851683]
  We present a novel Partial Feature Decorrelation Learning (PFDL) algorithm, which jointly optimize a feature decomposition network and the target image classification model. The experiments on real-world datasets demonstrate that our method can improve the backbone model's accuracy on OOD image classification datasets.
  arXiv  Detail & Related papers  (2020-07-30T05:48:48Z)
• Agglomerative Neural Networks for Multi-view Clustering [109.55325971050154]
  We propose the agglomerative analysis to approximate the optimal consensus view. We present Agglomerative Neural Network (ANN) based on Constrained Laplacian Rank to cluster multi-view data directly. Our evaluations against several state-of-the-art multi-view clustering approaches on four popular datasets show the promising view-consensus analysis ability of ANN.
  arXiv  Detail & Related papers  (2020-05-12T05:39:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.