Attention Via Convolutional Nearest Neighbors

• URL: http://arxiv.org/abs/2511.14137v2
• Date: Fri, 21 Nov 2025 12:17:52 GMT
• Title: Attention Via Convolutional Nearest Neighbors
• Abstract summary: We argue that convolution and self-attention can be unified within a single k-nearest neighbor aggregation framework.<n>We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection.<n>We validate the framework's coherence on CIFAR-10 and CIFAR-100 classification tasks.
• Score: 0.5729426778193399
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The shift from Convolutional Neural Networks to Transformers has reshaped computer vision, yet these two architectural families are typically viewed as fundamentally distinct. We argue that convolution and self-attention, despite their apparent differences, can be unified within a single k-nearest neighbor aggregation framework. The critical insight is that both operations are special cases of neighbor selection and aggregation; convolution selects neighbors by spatial proximity, while attention selects by feature similarity, revealing they exist on a continuous spectrum. We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection. Crucially, ConvNN serves as a drop-in replacement for convolutional and attention layers, enabling systematic exploration of the intermediate spectrum between these two extremes. We validate the framework's coherence on CIFAR-10 and CIFAR-100 classification tasks across two complementary architectures: (1) Hybrid branching in VGG improves accuracy on both CIFAR datasets by combining spatial-proximity and feature-similarity selection; and (2) ConvNN in ViT outperforms standard attention and other attention variants on both datasets. Extensive ablations on $k$ values and architectural variants reveal that interpolating along this spectrum provides regularization benefits by balancing local and global receptive fields. Our work provides a unifying framework that dissolves the apparent distinction between convolution and attention, with implications for designing more principled and interpretable vision architectures.

Related papers

• Structure-Feature Aligned Graph Learning via Alternating Constrained Optimization [7.881713230406407]
  We introduce a constrained two-view framework for node prediction that aligns structure-conditioned GNN embeddings with a structure-free feature prior learned by an anchor model.<n>We also propose a Channel-Split Adaptive Gated GNN (CSAG-GNN) that dynamically routes representations between global spectral smoothing and local spatial discrimination through a node-wise gating mechanism.
  arXiv  Detail & Related papers  (2026-07-13T13:58:52Z)
• Interpolation between Convolution and Attention via K-Nearest Neighbors [0.0]
  We argue that convolution and self-attention can be unified within a single k-nearest neighbor aggregation framework.<n>We introduce Convolutional Nearest Neighbors (ConvNN), a unified framework that formalizes this connection.
  arXiv  Detail & Related papers  (2026-05-31T23:23:42Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• Hierarchical Graph Feature Enhancement with Adaptive Frequency Modulation for Visual Recognition [6.580655899524989]
  Convolutional neural networks (CNNs) have demonstrated strong performance in visual recognition tasks.<n>We propose a novel framework that integrates graph-based rea soning into CNNs to enhance both structural awareness and feature representation.<n>The proposed HGFE module is lightweight, end-to-end trainable, and can be seamlessly integrated into standard CNN backbone networks.
  arXiv  Detail & Related papers  (2025-08-15T14:19:50Z)
• Cross-domain Hyperspectral Image Classification based on Bi-directional Domain Adaptation [9.274122822335586]
  We propose a Bi-directional Domain Adaptation (BiDA) framework for cross-domain hyperspectral image (HSI) classification.<n>BiDA focuses on extracting both domain-invariant features and domain-specific information in the independent adaptive space.<n> Experimental results on cross-temporal/scene airborne and satellite datasets demonstrate that the proposed BiDA performs significantly better than some state-of-the-art domain adaptation approaches.
  arXiv  Detail & Related papers  (2025-07-03T03:22:28Z)
• Cross Paradigm Representation and Alignment Transformer for Image Deraining [40.66823807648992]
  We propose a novel Cross Paradigm Representation and Alignment Transformer (CPRAformer)<n>Its core idea is the hierarchical representation and alignment, leveraging the strengths of both paradigms to aid image reconstruction.<n>We use two types of self-attention in the Transformer blocks: sparse prompt channel self-attention (SPC-SA) and spatial pixel refinement self-attention (SPR-SA)
  arXiv  Detail & Related papers  (2025-04-23T06:44:46Z)
• Revisiting the Integration of Convolution and Attention for Vision Backbone [59.50256661158862]
  Convolutions and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. We propose in this work to use MSHAs and Convs in parallel textbfat different granularity levels instead. We empirically verify the potential of the proposed integration scheme, named textitGLMix: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few semantic slots.
  arXiv  Detail & Related papers  (2024-11-21T18:59:08Z)
• Augmenting Convolutional networks with attention-based aggregation [55.97184767391253]
  We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth) It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption.
  arXiv  Detail & Related papers  (2021-12-27T14:05:41Z)
• An attention-driven hierarchical multi-scale representation for visual recognition [3.3302293148249125]
  Convolutional Neural Networks (CNNs) have revolutionized the understanding of visual content. We propose a method to capture high-level long-range dependencies by exploring Graph Convolutional Networks (GCNs) Our approach is simple yet extremely effective in solving both the fine-grained and generic visual classification problems.
  arXiv  Detail & Related papers  (2021-10-23T09:22:22Z)
• HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning [74.76431541169342]
  Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. We propose a novel hierarchical semantic-visual adaptation (HSVA) framework to align semantic and visual domains. Experiments on four benchmark datasets demonstrate HSVA achieves superior performance on both conventional and generalized ZSL.
  arXiv  Detail & Related papers  (2021-09-30T14:27:50Z)
• X-volution: On the unification of convolution and self-attention [52.80459687846842]
  We propose a multi-branch elementary module composed of both convolution and self-attention operation. The proposed X-volution achieves highly competitive visual understanding improvements.
  arXiv  Detail & Related papers  (2021-06-04T04:32:02Z)
• Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning [85.6386289476598]
  We develop a novel adversarial graph representation adaptation (AGRA) framework for cross-domain holistic-local feature co-adaptation. We conduct extensive and fair evaluations on several popular benchmarks and show that the proposed AGRA framework outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-08-03T15:00:31Z)
• Contradictory Structure Learning for Semi-supervised Domain Adaptation [67.89665267469053]
  Current adversarial adaptation methods attempt to align the cross-domain features. Two challenges remain unsolved: 1) the conditional distribution mismatch and 2) the bias of the decision boundary towards the source domain. We propose a novel framework for semi-supervised domain adaptation by unifying the learning of opposite structures.
  arXiv  Detail & Related papers  (2020-02-06T22:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.