Is an Image Also Worth 16x16=256 Superpixels? A Framework for Attentional Image Classification

• URL: http://arxiv.org/abs/2605.27144v1
• Date: Tue, 26 May 2026 15:09:46 GMT
• Title: Is an Image Also Worth 16x16=256 Superpixels? A Framework for Attentional Image Classification
• Abstract summary: We propose Superpixel Transformers (SPT), a novel framework that unifies superpixel-based image classification and Vision Transformers (ViTs)<n>SPT generalizes the Superpixel Image Classification with Graph Attention Networks (SICGAT) model and ViT to support arbitrary superpixel-based chunking strategies, connectivity graphs, and positional encodings.<n>We show that SPT achieves superior performance compared to previous superpixel-based GNN methods and remains competitive with ViTs.
• Score: 0.11470070927586014
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Superpixel-based image classification has traditionally leveraged graph neural networks (GNNs) for processing irregular image representations. Recent advances in computer vision, driven by Vision Transformers (ViTs), have introduced new paradigms in self-attentional models, surpassing convolutional neural networks (CNNs) in various tasks. However, a synergistic connection between GNNs, superpixels, and transformers remains unexplored. In this work, we propose Superpixel Transformers (SPT), a novel framework that unifies superpixel-based image classification and ViTs. SPT generalizes the Superpixel Image Classification with Graph Attention Networks (SICGAT) model and ViT to support arbitrary superpixel-based chunking strategies, connectivity graphs, and positional encodings. We introduce refinements including a multidimensional sine-cosine positional encoding and an enriched patch data structure that fully incorporates superpixel shape and color information. By testing SPT across datasets such as CIFAR10, FashionMNIST, and Imagenette, with various superpixel generation and graph connectivity strategies, we demonstrate that SPT achieves superior performance compared to previous superpixel-based GNN methods and remains competitive with ViTs. Notably, our approach addresses the limitations of SICGAT, such as information loss during pixel aggregation, and shows how constrained graph connectivity can enhance ViT performance. SPT bridges the gap between superpixel-based and transformer models, opening avenues for cross-domain generalization and future innovations in hybrid attentional frameworks, and showing that an image can also be worth $16\times16$ superpixels.

Related papers

• On Preserving Geometrical Invariance for Superpixel Image Classification using Graph Transformer [0.19116784879310025]
  Convolutional Neural Network (CNN) and Vision Transformer (ViT) for image classification exploit a dense grid of pixels containing redundant information.<n>For a larger image dataset, CNNs and ViTs face deployability challenges due to high computational complexity.<n>This paper proposes SuperGT, a Graph Transformer-based framework for image classification, which captures the long range dependencies.
  arXiv  Detail & Related papers  (2026-07-05T12:16:23Z)
• Square Superpixel Generation and Representation Learning via Granular Ball Computing [97.99956628286243]
  Superpixels provide a compact region-based representation that preserves object boundaries and local structures.<n>Most existing superpixel algorithms produce irregularly shaped regions, which are not well aligned with regular operators such as convolutions.<n>Motivated by the adaptive representation and coverage property of granular-ball computing, we develop a square superpixel generation approach.
  arXiv  Detail & Related papers  (2026-03-31T09:06:58Z)
• SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers [0.0]
  Vision Transformers (ViTs) have redefined image classification by leveraging self-attention to capture complex patterns and long-range dependencies between image patches.<n>A key challenge for ViTs is efficiently incorporating multi-scale feature representations, which is inherent in convolutional neural networks (CNNs) through their hierarchical structure.<n>We propose SAG-ViT, a Scale-Aware Graph Attention ViT that integrates multi-scale feature capabilities of CNNs, representational power of ViTs, graph-attended patching to enable richer contextual representation.
  arXiv  Detail & Related papers  (2024-11-14T13:15:27Z)
• T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
  A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields. In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting. Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
  arXiv  Detail & Related papers  (2023-05-12T04:10:42Z)
• Restormer: Efficient Transformer for High-Resolution Image Restoration [118.9617735769827]
  convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data. Transformers have shown significant performance gains on natural language and high-level vision tasks. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks.
  arXiv  Detail & Related papers  (2021-11-18T18:59:10Z)
• Less is More: Pay Less Attention in Vision Transformers [61.05787583247392]
  Less attention vIsion Transformer builds upon the fact that convolutions, fully-connected layers, and self-attentions have almost equivalent mathematical expressions for processing image patch sequences. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation.
  arXiv  Detail & Related papers  (2021-05-29T05:26:07Z)
• Superpixel-based Domain-Knowledge Infusion in Computer Vision [0.7349727826230862]
  Superpixels are higher-order perceptual groups of pixels in an image, often carrying much more information than raw pixels. There is an inherent relational structure to the relationship among different superpixels of an image. This relational information can convey some form of domain information about the image, e.g. relationship between superpixels representing two eyes in a cat image.
  arXiv  Detail & Related papers  (2021-05-20T01:25:42Z)
• Implicit Integration of Superpixel Segmentation into Fully Convolutional Networks [11.696069523681178]
  We propose a way to implicitly integrate a superpixel scheme into CNNs. Our proposed method hierarchically groups pixels at downsampling layers and generates superpixels. We evaluate our method on several tasks such as semantic segmentation, superpixel segmentation, and monocular depth estimation.
  arXiv  Detail & Related papers  (2021-03-05T02:20:26Z)
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale [112.94212299087653]
  Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.
  arXiv  Detail & Related papers  (2020-10-22T17:55:59Z)
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision [67.55770209540306]
  Visual Transformer ( VT) operates in a semantic token space, judiciously attending to different image parts based on context. Using an advanced training recipe, our VTs significantly outperform their convolutional counterparts. For semantic segmentation on LIP and COCO-stuff, VT-based feature pyramid networks (FPN) achieve 0.35 points higher mIoU while reducing the FPN module's FLOPs by 6.5x.
  arXiv  Detail & Related papers  (2020-06-05T20:49:49Z)
• Superpixel Segmentation with Fully Convolutional Networks [32.878045921919714]
  We present a novel method that employs a simple fully convolutional network to predict superpixels on a regular image grid. Experimental results on benchmark datasets show that our method achieves state-of-the-art superpixel segmentation performance. We modify a popular network architecture for stereo matching to simultaneously predict superpixels and disparities.
  arXiv  Detail & Related papers  (2020-03-29T02:42:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.