Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting

• URL: http://arxiv.org/abs/2511.19021v1
• Date: Mon, 24 Nov 2025 11:55:22 GMT
• Title: Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting
• Authors: Qiyang Yu, Yu Fang, Tianrui Li, Xuemei Cao, Yan Chen, Jianghao Li, Fan Min,
• Abstract summary: Vision Transformers (ViTs) have demonstrated strong capabilities in capturing global dependencies but often struggle to efficiently represent fine-grained local details.<n>We propose Granularity-driven Vision Transformer (Grc-ViT), a dynamic coarse-to-fine framework that adaptively adjusts visual granularity based on image complexity.<n>Two learnable parameters, and beta, are optimized end-to-end to balance global reasoning and local perception.
• Score: 15.751224470424786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Transformers (ViTs) have demonstrated strong capabilities in capturing global dependencies but often struggle to efficiently represent fine-grained local details. Existing multi-scale approaches alleviate this issue by integrating hierarchical or hybrid features; however, they rely on fixed patch sizes and introduce redundant computation. To address these limitations, we propose Granularity-driven Vision Transformer (Grc-ViT), a dynamic coarse-to-fine framework that adaptively adjusts visual granularity based on image complexity. It comprises two key stages: (1) Coarse Granularity Evaluation module, which assesses visual complexity using edge density, entropy, and frequency-domain cues to estimate suitable patch and window sizes; (2) Fine-grained Refinement module, which refines attention computation according to the selected granularity, enabling efficient and precise feature learning. Two learnable parameters, α and \b{eta}, are optimized end-to-end to balance global reasoning and local perception. Comprehensive evaluations demonstrate that Grc-ViT enhances fine-grained discrimination while achieving a superior trade-off between accuracy and computational efficiency.

Related papers

• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models [45.12546316524245]
  We introduce DART, a fully differentiable Dynamic Region Adaptive Tokenizer.<n>DART employs learnable region scores and quantile-based partitioning to create content-aware patches of varying sizes.<n>The impact of this approach is profound, where a DART-Small matches the performance of a DeiT-Base86 with nearly double the inference speed.
  arXiv  Detail & Related papers  (2025-06-12T06:25:37Z)
• Breaking Complexity Barriers: High-Resolution Image Restoration with Rank Enhanced Linear Attention [54.42902794496325]
  Linear attention, a variant of softmax attention, demonstrates promise in global context modeling.<n>We propose Rank Enhanced Linear Attention (RELA), a simple yet effective method that enriches feature representations by integrating a lightweight depthwise convolution.<n>Building upon RELA, we propose an efficient and effective image restoration Transformer, named LAformer.
  arXiv  Detail & Related papers  (2025-05-22T02:57:23Z)
• ContextFormer: Redefining Efficiency in Semantic Segmentation [48.81126061219231]
  Convolutional methods, although capturing local dependencies well, struggle with long-range relationships.<n>Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands.<n>We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation.
  arXiv  Detail & Related papers  (2025-01-31T16:11:04Z)
• Revisiting the Integration of Convolution and Attention for Vision Backbone [59.50256661158862]
  Convolutions and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. We propose in this work to use MSHAs and Convs in parallel textbfat different granularity levels instead. We empirically verify the potential of the proposed integration scheme, named textitGLMix: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few semantic slots.
  arXiv  Detail & Related papers  (2024-11-21T18:59:08Z)
• Efficient High-Resolution Visual Representation Learning with State Space Model for Human Pose Estimation [60.80423207808076]
  Capturing long-range dependencies while preserving high-resolution visual representations is crucial for dense prediction tasks such as human pose estimation.<n>We propose the Dynamic Visual State Space (DVSS) block, which augments visual state space models with multi-scale convolutional operations.<n>We build HRVMamba, a novel model for efficient high-resolution representation learning.
  arXiv  Detail & Related papers  (2024-10-04T06:19:29Z)
• Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention [34.26177289099421]
  Self-attention mechanism has been a key factor in the recent progress of Vision Transformer (ViT) We propose a novel local attention module, which leverages common convolution operations to achieve high efficiency, flexibility and generalizability. Our module realizes the local attention paradigm in both efficient and flexible manner.
  arXiv  Detail & Related papers  (2023-04-09T13:37:59Z)
• Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection [77.50110439560152]
  Current object detectors typically have a feature pyramid (FP) module for multi-level feature fusion (MFF) We propose a novel and efficient context modeling mechanism that can help existing FPs deliver better MFF results. In particular, we introduce a novel insight that comprehensive contexts can be decomposed and condensed into two types of representations for higher efficiency.
  arXiv  Detail & Related papers  (2022-07-14T01:45:03Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• MIA-Former: Efficient and Robust Vision Transformers via Multi-grained Input-Adaptation [14.866949449862226]
  Vision Transformer (ViT) models are too computationally expensive to be fitted onto real-world resource-constrained devices. We propose a Multi-grained Input-adaptive Vision Transformer framework dubbed MIA-Former that can input-adaptively adjust the structure of ViTs. Experiments and ablation studies validate that the proposed MIA-Former framework can effectively allocate budgets adaptive to the difficulty of input images.
  arXiv  Detail & Related papers  (2021-12-21T22:06:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.