Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning

• URL: http://arxiv.org/abs/2607.17673v1
• Date: Mon, 20 Jul 2026 08:25:45 GMT
• Title: Beyond Objective Expressivity: Geometry Preservation in Multimodal Contrastive Learning
• Abstract summary: We identify encoder Jacobian conditioning as a key factor in trimodal contrastive learning.<n>We introduce geometry-preserving encoders (GPEs) by directly conditioning the Jacobian through regularization.
• Score: 1.2472265402088736
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning is increasingly moving toward settings with three or more modalities instead of image-text pairs. Yet, extending models from pairwise to higher-order multimodal alignment can introduce optimization and representation challenges. We identify encoder Jacobian conditioning as a key factor in trimodal contrastive learning: poorly conditioned encoders exhibit collapsing or amplified singular-value spectra, leading to exploding Jacobian condition numbers and degraded multimodal alignment. We introduce geometry-preserving encoders (GPEs) by directly conditioning the Jacobian through regularization and demonstrating that simple modifications like LeakyReLU activations and residual paths recover these geometric benefits. Across a synthetic benchmark and four real-world datasets including missing modalities, improving Jacobian conditioning boosts retrieval and linear probe performance across multiple contrastive objectives, whereas expressive objectives yield little benefit in linear probes. More broadly, our results show that multimodal contrastive learning depends not only on objective expressivity, but also on the geometric and optimization properties of the underlying encoders.

Related papers

• CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning [79.07656918674684]
  CoME-VL is a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder.<n>Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin.
  arXiv  Detail & Related papers  (2026-04-03T17:59:51Z)
• Differentiable Zero-One Loss via Hypersimplex Projections [14.382224834970557]
  We introduce a novel differentiable approximation to the zero-one loss-long considered the gold standard for classification performance.<n>We show how its Jacobian can be efficiently computed and integrated into binary and multiclass learning systems.
  arXiv  Detail & Related papers  (2026-02-26T18:41:31Z)
• ECCO: Evidence-Driven Causal Reasoning for Compiler Optimization [9.85275171877854]
  We introduce ECCO, a framework that bridges interpretable reasoning with search.<n>We first propose a reverse engineering methodology to construct a Chain-of-Thought dataset.<n>We then design a collaborative inference mechanism where the Large Language Model functions as a strategist.
  arXiv  Detail & Related papers  (2026-01-23T01:23:20Z)
• Point Cloud Quantization through Multimodal Prompting for 3D Understanding [20.397232104616574]
  We propose a simple multimodal prompting-driven quantization framework for point cloud analysis.<n>Our methodology is built upon two core insights: 1) Text embeddings from pre-trained models inherently encode visual semantics.<n>We employ Gumbel-Softmax relaxation to achieve differentiable discretization while maintaining quantization sparsity.
  arXiv  Detail & Related papers  (2025-11-15T07:51:10Z)
• Iso-Riemannian Optimization on Learned Data Manifolds [6.345340156849189]
  We introduce a principled framework for optimization on learned data manifold using iso-Riemannian geometry.<n>We show that our approach yields interpretable barycentres, improved clustering, and provably efficient solutions to inverse problems.<n>These results establish that optimization under iso-Riemannian geometry can overcome distortions inherent to learned manifold mappings.
  arXiv  Detail & Related papers  (2025-10-23T22:34:55Z)
• Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval [11.20814404187967]
  We propose a framework that enables bi-directional learning between dense and sparse representations through Self-Knowledge Distillation.<n>This bi-directional learning is achieved using an integrated similarity score-a weighted sum of dense and sparse similarities-which serves as a shared teacher signal for both representations.<n>Experiments on MSCOCO and Flickr30k demonstrate that our sparse retriever not only outperforms existing sparse baselines, but also achieves performance comparable to-or even surpassing-its dense counterparts.
  arXiv  Detail & Related papers  (2025-08-22T13:25:58Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Optimizing Singular Spectrum for Large Language Model Compression [95.7621116637755]
  We introduce SoCo, a novel compression framework that learns to rescale the decomposed components of SVD in a data-driven manner.<n>Thanks to the learnable singular spectrum, SoCo adaptively prunes components according to the sparsified importance scores.<n> Experimental evaluations across multiple LLMs and benchmarks demonstrate that SoCo surpasses the state-of-the-art methods in model compression.
  arXiv  Detail & Related papers  (2025-02-20T23:18:39Z)
• Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning [52.70210390424605]
  In this work, we consider endowing a neural network autoencoder with three select inductive biases from the literature. In practice, however, naively combining existing techniques instantiating these inductive biases fails to yield significant benefits. We propose adaptations to the three techniques that simplify the learning problem, equip key regularization terms with stabilizing invariances, and quash degenerate incentives. The resulting model, Tripod, achieves state-of-the-art results on a suite of four image disentanglement benchmarks.
  arXiv  Detail & Related papers  (2024-04-16T04:52:41Z)
• Hodge-Aware Contrastive Learning [101.56637264703058]
  Simplicial complexes prove effective in modeling data with multiway dependencies. We develop a contrastive self-supervised learning approach for processing simplicial data.
  arXiv  Detail & Related papers  (2023-09-14T00:40:07Z)
• Self-Supervised Generative-Contrastive Learning of Multi-Modal Euclidean Input for 3D Shape Latent Representations: A Dynamic Switching Approach [53.376029341079054]
  We propose a combined generative and contrastive neural architecture for learning latent representations of 3D shapes.<n>The architecture uses two encoder branches for voxel grids and multi-view images from the same underlying shape.
  arXiv  Detail & Related papers  (2023-01-11T18:14:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.