GViT: Representing Images as Gaussians for Visual Recognition

• URL: http://arxiv.org/abs/2506.23532v1
• Date: Mon, 30 Jun 2025 05:44:14 GMT
• Title: GViT: Representing Images as Gaussians for Visual Recognition
• Authors: Jefferson Hernandez, Ruozhen He, Guha Balakrishnan, Alexander C. Berg, Vicente Ordonez,
• Abstract summary: We introduce GVIT, a classification framework that abandons conventional pixel or patch grid input representations in favor of a compact set of learnable 2D Gaussians.<n>We demonstrate that by 2D Gaussian input representations coupled with our GVIT guidance, using a relatively standard ViT architecture, closely matches the performance of a traditional patch-based ViT.
• Score: 54.46109876668194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce GVIT, a classification framework that abandons conventional pixel or patch grid input representations in favor of a compact set of learnable 2D Gaussians. Each image is encoded as a few hundred Gaussians whose positions, scales, orientations, colors, and opacities are optimized jointly with a ViT classifier trained on top of these representations. We reuse the classifier gradients as constructive guidance, steering the Gaussians toward class-salient regions while a differentiable renderer optimizes an image reconstruction loss. We demonstrate that by 2D Gaussian input representations coupled with our GVIT guidance, using a relatively standard ViT architecture, closely matches the performance of a traditional patch-based ViT, reaching a 76.9% top-1 accuracy on Imagenet-1k using a ViT-B architecture.

Related papers

• How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings [106.3726679697804]
  We compare the two most common techniques for mitigating this spectral bias: Fourier feature encodings (FFE) and multigrid parametric encodings (MPE)<n>MPEs are seen as the standard for low dimensional mappings, but MPEs often outperform them and learn representations with higher resolution and finer detail.<n>We prove that MPEs improve a network's performance through the structure of their grid and not their learnable embedding.
  arXiv  Detail & Related papers  (2025-04-18T02:18:08Z)
• EigenGS Representation: From Eigenspace to Gaussian Image Space [20.454762899389358]
  EigenGS is an efficient transformation pipeline connecting eigenspace and image-space Gaussian representations.<n>We show that EigenGS achieves superior reconstruction quality compared to direct 2D Gaussian fitting.<n>The results highlight EigenGS's effectiveness and generalization ability across images with varying resolutions and diverse categories.
  arXiv  Detail & Related papers  (2025-03-10T15:27:03Z)
• Large Images are Gaussians: High-Quality Large Image Representation with Levels of 2D Gaussian Splatting [21.629316414488027]
  We present textbfLarge textbfImages are textbfGaussians (textbfLIG), which delves deeper into the application of 2DGS for image representations.
  arXiv  Detail & Related papers  (2025-02-13T07:48:56Z)
• Image-GS: Content-Adaptive Image Representation via 2D Gaussians [52.598772767324036]
  We introduce Image-GS, a content-adaptive image representation based on 2D Gaussians radiance.<n>It supports hardware-friendly rapid access for real-time usage, requiring only 0.3K MACs to decode a pixel.<n>We demonstrate its versatility with several applications, including texture compression, semantics-aware compression, and joint image compression and restoration.
  arXiv  Detail & Related papers  (2024-07-02T00:45:21Z)
• ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders [11.727612242016871]
  ViC-MAE is a model that combines Masked AutoEncoders (MAE) and contrastive learning. We show that visual representations learned under ViC-MAE generalize well to both video and image classification tasks.
  arXiv  Detail & Related papers  (2023-03-21T16:33:40Z)
• RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving [80.14669385741202]
  Vision transformers (ViTs) have achieved state-of-the-art results in many image-based benchmarks. ViTs are notoriously hard to train and require a lot of training data to learn powerful representations. We show that our method, called RangeViT, outperforms existing projection-based methods on nuScenes and Semantic KITTI.
  arXiv  Detail & Related papers  (2023-01-24T18:50:48Z)
• CyCLIP: Cyclic Contrastive Language-Image Pretraining [34.588147979731374]
  Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness. We demonstrate that the image and text representations learned via a standard contrastive objective are not interchangeable and can lead to inconsistent downstream predictions. We propose CyCLIP, a framework for contrastive representation learning that explicitly optimize for the learned representations to be geometrically consistent in the image and text space.
  arXiv  Detail & Related papers  (2022-05-28T15:31:17Z)
• Vector-quantized Image Modeling with Improved VQGAN [93.8443646643864]
  We propose a Vector-quantized Image Modeling approach that involves pretraining a Transformer to predict image tokens autoregressively. We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. When trained on ImageNet at 256x256 resolution, we achieve Inception Score (IS) of 175.1 and Frechet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN.
  arXiv  Detail & Related papers  (2021-10-09T18:36:00Z)
• DFM: A Performance Baseline for Deep Feature Matching [10.014010310188821]
  The proposed method uses pre-trained VGG architecture as a feature extractor and does not require any additional training specific to improve matching. Our algorithm achieves 0.57 and 0.80 overall scores in terms of Mean Matching Accuracy (MMA) for 1 pixel and 2 pixels thresholds respectively on Hpatches dataset.
  arXiv  Detail & Related papers  (2021-06-14T22:55:06Z)
• Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning [116.91819311885166]
  We propose a hierarchical semantic alignment strategy via expanding the views generated by a single image to textbfCross-samples and Multi-level representation. Our method, termed as CsMl, has the ability to integrate multi-level visual representations across samples in a robust way.
  arXiv  Detail & Related papers  (2020-12-04T17:26:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.