Hyperbolic Hierarchical Clustering for Visual Representation Learning

• URL: http://arxiv.org/abs/2608.22665v1
• Date: Mon, 24 Aug 2026 00:01:09 GMT
• Title: Hyperbolic Hierarchical Clustering for Visual Representation Learning
• Abstract summary: An effective token mixer is a fundamental component of modern vision backbones like Transformers.<n>We introduce ClusterMixer, a transparent token mixer grounded in a clustering paradigm and interpretable by design.<n>We present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies.
• Score: 62.38996677789715
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate the token mixer in vision backbones by revisiting clustering, one of the most classic approaches in machine learning. An effective token mixer is a fundamental component of modern vision backbones like vision Transformers, facilitating information exchange between image patches. Mainstream token mixers, which rely on convolution, attention, MLP, or their hybrids, primarily focus on navigating the trade-off between accuracy and computational cost. However, a significant drawback of these methods is their black-box nature; their encoding process is opaque and lacks interpretability. Diverging from these opaque designs, we introduce ClusterMixer, a transparent token mixer that is grounded in a clustering paradigm and interpretable by design. ClusterMixer explicitly formulates the token mixing process through a hierarchical clustering mechanism. To model the natural, tree-like relationships inherent in visual data, the clustering is performed in hyperbolic space, which is well-suited for embedding hierarchies with low distortion. Building on this innovation, we present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies to ensure robust performance across tasks. Extensive experiments demonstrate that HCFormer consistently outperforms its counterparts across diverse tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Considering its transparency and efficacy, we hope HCFormer can facilitate a paradigm shift toward interpretable backbones.

Related papers

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• Deep Clustering with Associative Memories [28.46602918457589]
  We propose a novel loss function utilizing energy-based dynamics via Associative Memories to formulate a new deep clustering method, DCAM.<n>Our experiments showcase the advantage of DCAM, producing improved clustering quality for various architecture choices.
  arXiv  Detail & Related papers  (2026-01-02T19:21:06Z)
• Hierarchical Identity Learning for Unsupervised Visible-Infrared Person Re-Identification [81.3063589622217]
  Unsupervised visible-infrared person re-identification (USVI-ReID) aims to learn modality-invariant image features from unlabeled cross-modal person datasets.
  arXiv  Detail & Related papers  (2025-09-15T05:10:43Z)
• Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency [57.961869351897384]
  We propose a framework based on cross-modal semantic consistency for efficient image clustering.<n>Our framework first builds a strong foundation via Cross-Modal Semantic Consistency.<n>In the first stage, we train lightweight clustering heads to align with the rich semantics of the pre-trained model.<n>In the second stage, we introduce a Self-Enhanced fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-08-02T08:12:57Z)
• Neural Clustering based Visual Representation Learning [61.72646814537163]
  Clustering is one of the most classic approaches in machine learning and data analysis. We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data. FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
  arXiv  Detail & Related papers  (2024-03-26T06:04:50Z)
• Grid Jigsaw Representation with CLIP: A New Perspective on Image Clustering [33.05984601411495]
  We propose a new perspective on image clustering, the pretrain-based Grid Jigsaw Representation (pGJR)<n>Inspired by human jigsaw puzzle processing, we modify the traditional jigsaw learning to gain a more sequential and incremental understanding of image structure.<n>Our experiments demonstrate that using the pretrained model as a feature extractor can accelerate the convergence of clustering.
  arXiv  Detail & Related papers  (2023-10-27T03:07:05Z)
• CoC-GAN: Employing Context Cluster for Unveiling a New Pathway in Image Generation [12.211795836214112]
  We propose a unique image generation process premised on the perspective of converting images into a set of point clouds. Our methodology leverages simple clustering methods named Context Clustering (CoC) to generate images from unordered point sets. We introduce this model with the novel structure as the Context Clustering Generative Adversarial Network (CoC-GAN)
  arXiv  Detail & Related papers  (2023-08-23T01:19:58Z)
• Graph Contrastive Clustering [131.67881457114316]
  We propose a novel graph contrastive learning framework, which is then applied to the clustering task and we come up with the Graph Constrastive Clustering(GCC) method. Specifically, on the one hand, the graph Laplacian based contrastive loss is proposed to learn more discriminative and clustering-friendly features. On the other hand, a novel graph-based contrastive learning strategy is proposed to learn more compact clustering assignments.
  arXiv  Detail & Related papers  (2021-04-03T15:32:49Z)
• Deep Transformation-Invariant Clustering [24.23117820167443]
  We present an approach that does not rely on abstract features but instead learns to predict image transformations. This learning process naturally fits in the gradient-based training of K-means and Gaussian mixture model. We demonstrate that our novel approach yields competitive and highly promising results on standard image clustering benchmarks.
  arXiv  Detail & Related papers  (2020-06-19T13:43:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.