MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining

• URL: http://arxiv.org/abs/2604.24374v1
• Date: Mon, 27 Apr 2026 12:07:40 GMT
• Title: MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining
• Abstract summary: Matryoshka Representation Learning (MRL) offers a flexible inference paradigm through nested embeddings.<n>We propose MIPIC (Matryoshka Representation Learning via Self-Distilled Intra-Relational Alignment and Progressive Information Chaining) to produce structurally coherent and semantically compact Matryoshka representations.
• Score: 16.316971846745528
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representation learning is fundamental to NLP, but building embeddings that work well at different computational budgets is challenging. Matryoshka Representation Learning (MRL) offers a flexible inference paradigm through nested embeddings; however, learning such structures requires explicit coordination of how information is arranged across embedding dimensionality and model depth. In this work, we propose MIPIC (Matryoshka Representation Learning via Self-Distilled Intra-Relational Alignment and Progressive Information Chaining), a unified training framework designed to produce structurally coherent and semantically compact Matryoshka representations. MIPIC promotes cross-dimensional structural consistency through Self-Distilled Intra-Relational Alignment (SIA), which aligns token-level geometric and attention-driven relations between full and truncated representations using top-k CKA self-distillation. Complementarily, it enables depth-wise semantic consolidation via Progressive Information Chaining (PIC), a scaffolded alignment strategy that incrementally transfers mature task semantics from deeper layers into earlier layers. Extensive experiments on STS, NLI, and classification benchmarks (spanning models from TinyBERT to BGEM3, Qwen3) demonstrate that MIPIC yields Matryoshka representations that are highly competitive across all capacities, with significant performance advantages observed under extreme low-dimensional.

Related papers

• Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System [90.64481646805855]
  unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model.<n>We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting.<n>We show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
  arXiv  Detail & Related papers  (2026-09-01T17:59:59Z)
• Understanding Submodular Information Measure Based Objectives for Representation Learning: A Variance and Separation Perspective [3.9890357781493595]
  Submodular Information Measures (SIMs) have emerged as a powerful framework for representation learning and multimodal learning.<n>We develop a unified theoretical framework connecting SIMs to classical concepts in representation learning and statistical pattern recognition.<n>Our results provide the first unified geometric and statistical understanding of submodular information measures.
  arXiv  Detail & Related papers  (2026-07-30T04:17:17Z)
• Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification [0.8029049649310211]
  Multi-view learning approaches typically rely on feature-level aggregation or single-stage cross-attention.<n>We propose a token-centric dual-view learning framework that unifies prompt-based adaptation and cross-view fusion within a frozen vision transformer backbone.<n>Experiments on VinDr-Mammo and CMMD datasets demonstrate consistent improvements over linear probing, prompt-only adaptation, and conventional fusion baselines.
  arXiv  Detail & Related papers  (2026-07-07T14:11:54Z)
• Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning [3.37216674402797]
  In educational deployments, individual modalities such as eye gaze exhibit inconsistent informativeness across student cohorts.<n>We propose the Affinity-Aligned Multimodal Learning Analytics framework, whose core contribution is the Cross-modal Affinity-guided Modality Alignment module.
  arXiv  Detail & Related papers  (2026-05-16T04:22:47Z)
• Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering [57.38215918201251]
  We propose HERL, a Hyperbolic Enhanced Representation Learning framework for Incomplete Multi-View Clustering.<n> operating within the Poincaré ball, HERL constructs a structure-aware latent space to enhance representation learning.<n>We show that HERL consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-04-18T10:50:46Z)
• Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning [3.055416693231887]
  We propose a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation.<n> Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines.
  arXiv  Detail & Related papers  (2026-04-05T19:08:51Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• TractoRC: A Unified Probabilistic Learning Framework for Joint Tractography Registration and Clustering [9.66539105471215]
  Two key tasks in tractography analysis include: 1) tractogram registration that aligns streamlines across individuals, and 2) streamline clustering that groups streamlines into compact fiber bundles.<n>We propose TractoRC, a unified probabilistic framework that jointly performs tractogram registration and streamline clustering within a single optimization scheme.
  arXiv  Detail & Related papers  (2026-03-11T05:04:53Z)
• Reconstructing Content via Collaborative Attention to Improve Multimodal Embedding Quality [59.651410243721045]
  CoCoA is a Content reconstruction pre-training paradigm based on Collaborative Attention for multimodal embedding optimization.<n>We introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding EOS> embeddings.<n>Experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves embedding quality.
  arXiv  Detail & Related papers  (2026-03-02T05:34:45Z)
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension [55.29309306566238]
  Current Large Language Models (LLMs) are confronted with overwhelming information volume when comprehending long-form documents.<n>This challenge raises the imperative of a cohesive memory module, which can elevate vanilla LLMs into autonomous reading agents.<n>We draw inspiration from Jean Piaget's Constructivist Theory, illuminating three traits of the agentic memory -- structured schemata, flexible assimilation, and dynamic accommodation.
  arXiv  Detail & Related papers  (2025-10-07T02:16:30Z)
• Patch-Level Kernel Alignment for Dense Self-Supervised Learning [7.5866326278176075]
  We introduce Patch-level Kernel Alignment (PaKA), a non-parametric, kernel-based approach that improves the dense representations of pretrained vision encoders with a post-(pre)training.<n>Our framework improves dense representations by conducting a lightweight post-training stage on top of a pretrained model.<n>With only 14 hours of additional training on a single GPU, our method achieves state-of-the-art performance across a range of dense vision benchmarks.
  arXiv  Detail & Related papers  (2025-09-06T05:42:32Z)
• SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillation [25.38025028623991]
  3D point cloud segmentation faces practical challenges due to the computational complexity and deployment limitations of large-scale transformer-based models.<n>We propose a novel Structure- and Relation-aware Knowledge Distillation framework, named SRKD, that transfers rich geometric and semantic knowledge from a large frozen teacher model to a lightweight student model.<n>Our method achieves significantly reduced model complexity, demonstrating its effectiveness and efficiency in real-world deployment scenarios.
  arXiv  Detail & Related papers  (2025-06-16T07:32:58Z)
• Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learning [81.02648336552421]
  We propose a Multi-Constraint Consistency Learning approach to facilitate the staged enhancement of the encoder and decoder.<n>Self-adaptive feature masking and noise injection are designed in an instance-specific manner to perturb the features for robust learning of the decoder.<n> Experimental results on Pascal VOC2012 and Cityscapes datasets demonstrate that our proposed MCCL achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-23T03:21:33Z)
• Interpretable deformable image registration: A geometric deep learning perspective [9.13809412085203]
  We present a theoretical foundation for designing an interpretable registration framework.<n>We formulate an end-to-end process that refines transformations in a coarse-to-fine fashion.<n>We conclude by showing significant improvement in performance metrics over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-12-17T19:47:10Z)
• GraphLearner: Graph Node Clustering with Fully Learnable Augmentation [76.63963385662426]
  Contrastive deep graph clustering (CDGC) leverages the power of contrastive learning to group nodes into different clusters. We propose a Graph Node Clustering with Fully Learnable Augmentation, termed GraphLearner. It introduces learnable augmentors to generate high-quality and task-specific augmented samples for CDGC.
  arXiv  Detail & Related papers  (2022-12-07T10:19:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.