Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment

• URL: http://arxiv.org/abs/2504.02522v2
• Date: Thu, 15 May 2025 09:48:06 GMT
• Title: Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment
• Authors: Fatemeh Behrad, Tinne Tuytelaars, Johan Wagemans,
• Abstract summary: Vision transformers (ViTs) are typically trained on small, fixed-size images obtained through downscaling or cropping.<n>We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously.<n>Charm improves ViT performance and generalizability for image aesthetic assessment.
• Score: 36.633379840639314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone. Code and pre-trained models are available at https://github.com/FBehrad/Charm.

Related papers

• Accelerating Vision Transformers with Adaptive Patch Sizes [58.48800204993534]
  Vision Transformers partition input images into uniformly sized patches regardless of their content.<n>We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image.<n>APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H.
  arXiv  Detail & Related papers  (2025-10-20T20:37:11Z)
• You Only Need Less Attention at Each Stage in Vision Transformers [19.660385306028047]
  Vision Transformers (ViTs) capture the global information of images through self-attention modules. We propose the Less-Attention Vision Transformer (LaViT), which computes only a few attention operations at each stage. Our architecture demonstrates exceptional performance across various vision tasks including classification, detection and segmentation.
  arXiv  Detail & Related papers  (2024-06-01T12:49:16Z)
• No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling [55.203866875294516]
  Vision Transformers (ViTs) have demonstrated outstanding performance in computer vision tasks. Various token pruning techniques have been introduced to alleviate the high computational burden of ViTs. We propose IdleViT, a dynamic token-idle-based method that achieves an excellent trade-off between performance and efficiency.
  arXiv  Detail & Related papers  (2023-10-09T12:10:41Z)
• MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers [14.787864686489032]
  We introduce a conditional gating mechanism that selects the optimal token scale for every image region. We show that our gating module is able to learn meaningful semantics despite operating locally at the coarse patch-level. In contrast to token pruning, MSViT does not lose information about the input, thus can be readily applied for dense tasks.
  arXiv  Detail & Related papers  (2023-07-05T14:22:31Z)
• ViT-FOD: A Vision Transformer based Fine-grained Object Discriminator [21.351034332423374]
  We propose a novel ViT based fine-grained object discriminator for Fine-Grained Visual Classification (FGVC) tasks. Besides a ViT backbone, it introduces three novel components, i.e. Attention Patch Combination (APC), Critical Regions Filter (CRF) and Complementary Tokens Integration (CTI) We conduct comprehensive experiments on widely used datasets and the results demonstrate that ViT-FOD is able to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-03-24T02:34:57Z)
• Coarse-to-Fine Vision Transformer [83.45020063642235]
  We propose a coarse-to-fine vision transformer (CF-ViT) to relieve computational burden while retaining performance. Our proposed CF-ViT is motivated by two important observations in modern ViT models. Our CF-ViT reduces 53% FLOPs of LV-ViT, and also achieves 2.01x throughput.
  arXiv  Detail & Related papers  (2022-03-08T02:57:49Z)
• When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations [111.44860506703307]
  Vision Transformers (ViTs) and existing VisionNets signal efforts on replacing hand-wired features or inductive throughputs with general-purpose neural architectures. This paper investigates ViTs and Res-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and inference. We show that the improved robustness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform Nets of similar size and smoothness when trained from scratch on ImageNet without large-scale pretraining or strong data augmentations.
  arXiv  Detail & Related papers  (2021-06-03T02:08:03Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Scalable Visual Transformers with Hierarchical Pooling [61.05787583247392]
  We propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length. It brings a great benefit by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity. Our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2021-03-19T03:55:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.