Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning

• URL: http://arxiv.org/abs/2507.14137v2
• Date: Tue, 05 Aug 2025 16:34:09 GMT
• Title: Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learning
• Authors: Shashanka Venkataramanan, Valentinos Pariza, Mohammadreza Salehi, Lukas Knobel, Spyros Gidaris, Elias Ramzi, Andrei Bursuc, Yuki M. Asano,
• Abstract summary: Franca is the first fully open-source (data, code, weights) vision foundation model.<n>It matches and in many cases surpasses the performance of state-of-the-art proprietary models.<n>Our contributions establish a new standard for transparent, high-performance vision models.
• Score: 30.590869749117815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Franca (pronounced Fran-ka): free one; the first fully open-source (data, code, weights) vision foundation model that matches and in many cases surpasses the performance of state-of-the-art proprietary models, e.g., DINOv2, CLIP, SigLIPv2, etc. Our approach is grounded in a transparent training pipeline inspired by Web-SSL and uses publicly available data: ImageNet-21K and a subset of ReLAION-2B. Beyond model release, we tackle critical limitations in SSL clustering methods. While modern models rely on assigning image features to large codebooks via clustering algorithms like Sinkhorn-Knopp, they fail to account for the inherent ambiguity in clustering semantics. To address this, we introduce a parameter-efficient, multi-head clustering projector based on nested Matryoshka representations. This design progressively refines features into increasingly fine-grained clusters without increasing the model size, enabling both performance and memory efficiency. Additionally, we propose a novel positional disentanglement strategy that explicitly removes positional biases from dense representations, thereby improving the encoding of semantic content. This leads to consistent gains on several downstream benchmarks, demonstrating the utility of cleaner feature spaces. Our contributions establish a new standard for transparent, high-performance vision models and open a path toward more reproducible and generalizable foundation models for the broader AI community. The code and model checkpoints are available at https://github.com/valeoai/Franca.

Related papers

• Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency [57.961869351897384]
  We propose a framework based on cross-modal semantic consistency for efficient image clustering.<n>Our framework first builds a strong foundation via Cross-Modal Semantic Consistency.<n>In the first stage, we train lightweight clustering heads to align with the rich semantics of the pre-trained model.<n>In the second stage, we introduce a Self-Enhanced fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-08-02T08:12:57Z)
• An Enhanced Model-based Approach for Short Text Clustering [58.60681789677676]
  Short text clustering has become increasingly important with the popularity of social media like Twitter, Google+, and Facebook.<n>Existing methods can be broadly categorized into two paradigms: topic model-based approaches and deep representation learning-based approaches.<n>We propose a collapsed Gibbs Sampling algorithm for the Dirichlet Multinomial Mixture model (GSDMM), which effectively handles the sparsity and high dimensionality of short texts.<n>Based on several aspects of GSDMM that warrant further refinement, we propose an improved approach, GSDMM+, designed to further optimize its performance.
  arXiv  Detail & Related papers  (2025-07-18T10:07:42Z)
• Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction [52.09472099976885]
  IAR is an Improved AutoRegressive Visual Generation Method that enhances the training efficiency and generation quality of LLM-based visual generation models.<n>Our method consistently enhances the model training efficiency and performance from 100M to 1.4B, reducing the training time by half while achieving the same FID.
  arXiv  Detail & Related papers  (2025-01-01T15:58:51Z)
• SIGMA:Sinkhorn-Guided Masked Video Modeling [69.31715194419091]
  Sinkhorn-guided Masked Video Modelling ( SIGMA) is a novel video pretraining method. We distribute features of space-time tubes evenly across a limited number of learnable clusters. Experimental results on ten datasets validate the effectiveness of SIGMA in learning more performant, temporally-aware, and robust video representations.
  arXiv  Detail & Related papers  (2024-07-22T08:04:09Z)
• Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning [86.15009879251386]
  We propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBM) CBMs require an additional set of concepts to leverage. We show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models.
  arXiv  Detail & Related papers  (2024-04-04T09:43:43Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models [37.574691902971296]
  We propose a novel image clustering pipeline that leverages the powerful feature representation of large pre-trained models. We show that our pipeline works well on standard datasets such as CIFAR-10, CIFAR-100, and ImageNet-1k.
  arXiv  Detail & Related papers  (2023-06-08T15:20:27Z)
• Unicom: Universal and Compact Representation Learning for Image Retrieval [65.96296089560421]
  We cluster the large-scale LAION400M into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model. To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss. Our method significantly outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2023-04-12T14:25:52Z)
• PRANC: Pseudo RAndom Networks for Compacting deep models [22.793523211040682]
  PRANC enables significant compaction of a deep model. In this study, we employ PRANC to condense image classification models and compress images by compacting their associated implicit neural networks.
  arXiv  Detail & Related papers  (2022-06-16T22:03:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.