Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning

• URL: http://arxiv.org/abs/2511.19515v1
• Date: Mon, 24 Nov 2025 03:10:12 GMT
• Title: Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning
• Abstract summary: We investigate the relationship between model capacity and the minimal number of visual tokens required to preserve image semantics.<n>Inspired by the Minimum Description Length principle, we reinterpret image tokens as vectors in a visual semantic space.<n>We propose Orthogonal Filtering, a lightweight module that adaptively clusters redundant tokens into a compact set of bases.
• Score: 7.379875124773055
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper investigates the fundamental relationship between model capacity and the minimal number of visual tokens required to preserve image semantics. Inspired by the Minimum Description Length principle, we reinterpret image tokens as vectors in a visual semantic space and define the intrinsic semantic complexity of an image as the smallest set of basis vectors needed to span this space. Building on this perspective, we propose Orthogonal Filtering, a lightweight module that adaptively clusters redundant tokens into a compact set of orthogonal bases. Through extensive experiments across a range of ViT models, we reveal a consistent token, model scaling law: larger models require significantly fewer tokens to span visual semantic space. Besides, we also contribute a visual long-context dataset.

Related papers

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• SNCE: Geometry-Aware Supervision for Scalable Discrete Image Generation [64.29376407025768]
  Neighbor Cross Entropy Minimization (SNCE) is a novel training objective designed to address the optimization challenges of large-codebook discrete image generators.<n>We conduct experiments across class-conditional ImageNet-256 generation, large-scale text-to-image synthesis, and image editing tasks.<n>Results show that SNCE significantly improves convergence speed and overall generation quality compared to standard cross-entropy objectives.
  arXiv  Detail & Related papers  (2026-03-16T11:44:11Z)
• Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness [16.541207762213272]
  Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations.<n>We introduce a novel finetuning framework that steers model reasoning toward concept-level semantics.<n>Our method improves robustness across multiple ViT-based models.
  arXiv  Detail & Related papers  (2026-03-09T12:31:14Z)
• Cross-Layer Discrete Concept Discovery for Interpreting Language Models [13.842670153893977]
  Cross-layer VQ-VAE is a framework that uses vector quantization to map representations across layers.<n>Our approach uniquely combines top-k temperature-based sampling during quantization with EMA codebook updates.
  arXiv  Detail & Related papers  (2025-06-24T22:43:36Z)
• Relative Drawing Identification Complexity is Invariant to Modality in Vision-Language Models [9.310396714303446]
  We evaluate the complexity of teaching vision-language models using two presentations: raw images as bitmaps and trace coordinates in TikZ format.<n>The results indicate that image-based representations generally require fewer segments and achieve higher accuracy than coordinate-based representations.<n>But, surprisingly, the teaching size usually ranks concepts similarly across both modalities, even when controlling for (a human proxy of) concept priors.
  arXiv  Detail & Related papers  (2025-05-14T09:41:38Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers [55.2480439325792]
  We propose FUSE, an approach to approximating an adapter layer that maps from one model's textual embedding space to another, even across different tokenizers. We show the efficacy of our approach via multi-objective optimization over vision-language and causal language models for image captioning and sentiment-based image captioning.
  arXiv  Detail & Related papers  (2024-08-09T02:16:37Z)
• Matryoshka Multimodal Models [92.41824727506751]
  We propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens. We find that COCO-style benchmarks only need around 9 visual tokens to obtain accuracy similar to that of using all 576 tokens.
  arXiv  Detail & Related papers  (2024-05-27T17:59:56Z)
• Subobject-level Image Tokenization [60.80949852899857]
  Patch-based image tokenization ignores the morphology of the visual world.<n>Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation.<n>We show that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.
  arXiv  Detail & Related papers  (2024-02-22T06:47:44Z)
• Scalable Visual Transformers with Hierarchical Pooling [61.05787583247392]
  We propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length. It brings a great benefit by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity. Our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2021-03-19T03:55:58Z)
• Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation [49.90178055521207]
  This work addresses weakly supervised semantic segmentation (WSSS), with the goal of bridging the gap between image-level annotations and pixel-level segmentation. We formulate WSSS as a novel group-wise learning task that explicitly models semantic dependencies in a group of images to estimate more reliable pseudo ground-truths. In particular, we devise a graph neural network (GNN) for group-wise semantic mining, wherein input images are represented as graph nodes.
  arXiv  Detail & Related papers  (2020-12-09T12:40:13Z)
• Multi-layer Feature Aggregation for Deep Scene Parsing Models [19.198074549944568]
  In this paper, we explore the effective use of multi-layer feature outputs of the deep parsing networks for spatial-semantic consistency. The proposed module can auto-select the intermediate visual features to correlate the spatial and semantic information. Experiments on four public scene parsing datasets prove that the deep parsing network equipped with the proposed feature aggregation module can achieve very promising results.
  arXiv  Detail & Related papers  (2020-11-04T23:07:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.