Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models

• URL: http://arxiv.org/abs/2507.15798v1
• Date: Mon, 21 Jul 2025 16:57:25 GMT
• Title: Exploring Superposition and Interference in State-of-the-Art Low-Parameter Vision Models
• Authors: Lilian Hollard, Lucas Mohimont, Nathalie Gaveau, Luiz-Angelo Steffenel,
• Abstract summary: We address interference in feature maps, a phenomenon associated with superposition, where neurons simultaneously encode multiple characteristics.<n>Our research suggests that limiting interference can enhance scaling and accuracy in very low-scaled networks (under 1.5M parameters)<n>We propose a proof-of-concept architecture named NoDepth Bottleneck built on mechanistic insights from our experiments, demonstrating robust scaling accuracy on the ImageNet dataset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The paper investigates the performance of state-of-the-art low-parameter deep neural networks for computer vision, focusing on bottleneck architectures and their behavior using superlinear activation functions. We address interference in feature maps, a phenomenon associated with superposition, where neurons simultaneously encode multiple characteristics. Our research suggests that limiting interference can enhance scaling and accuracy in very low-scaled networks (under 1.5M parameters). We identify key design elements that reduce interference by examining various bottleneck architectures, leading to a more efficient neural network. Consequently, we propose a proof-of-concept architecture named NoDepth Bottleneck built on mechanistic insights from our experiments, demonstrating robust scaling accuracy on the ImageNet dataset. These findings contribute to more efficient and scalable neural networks for the low-parameter range and advance the understanding of bottlenecks in computer vision. https://caiac.pubpub.org/pub/3dh6rsel

Related papers

• Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens [0.5745241788717261]
  We empirically analyze how neural networks behave under data and model scaling through the lens of the neural tangent kernel (NTK)<n>Our findings of standard vision tasks show that similar performance scaling exponents can occur even though the internal model dynamics show opposite behavior.<n>We also address a previously unresolved issue in neural scaling: how convergence to the infinite-width limit affects scaling behavior in finite-width models.
  arXiv  Detail & Related papers  (2025-07-07T14:17:44Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Impact of spiking neurons leakages and network recurrences on event-based spatio-temporal pattern recognition [0.0]
  Spiking neural networks coupled with neuromorphic hardware and event-based sensors are getting increased interest for low-latency and low-power inference at the edge. We explore the impact of synaptic and membrane leakages in spiking neurons.
  arXiv  Detail & Related papers  (2022-11-14T21:34:02Z)
• FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks [77.34726150561087]
  We show how to explore high-dimensional landscape characteristics of neural networks. We generalize observations on small neural networks to more complex systems. An interactive dashboard opens up a number of possible application networks.
  arXiv  Detail & Related papers  (2022-04-09T16:41:53Z)
• Vision Transformer with Convolutions Architecture Search [72.70461709267497]
  We propose an architecture search method-Vision Transformer with Convolutions Architecture Search (VTCAS) The high-performance backbone network searched by VTCAS introduces the desirable features of convolutional neural networks into the Transformer architecture. It enhances the robustness of the neural network for object recognition, especially in the low illumination indoor scene.
  arXiv  Detail & Related papers  (2022-03-20T02:59:51Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Max and Coincidence Neurons in Neural Networks [0.07614628596146598]
  We optimize networks containing models of the max and coincidence neurons using neural architecture search. We analyze the structure, operations, and neurons of optimized networks to develop a signal-processing ResNet. The developed network achieves an average of 2% improvement in accuracy and a 25% improvement in network size across a variety of datasets.
  arXiv  Detail & Related papers  (2021-10-04T07:13:50Z)
• Neural network relief: a pruning algorithm based on neural activity [47.57448823030151]
  We propose a simple importance-score metric that deactivates unimportant connections. We achieve comparable performance for LeNet architectures on MNIST. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations.
  arXiv  Detail & Related papers  (2021-09-22T15:33:49Z)
• Information contraction in noisy binary neural networks and its implications [11.742803725197506]
  We consider noisy binary neural networks, where each neuron has a non-zero probability of producing an incorrect output. Our key finding is a lower bound for the required number of neurons in noisy neural networks, which is first of its kind. This paper offers new understanding of noisy information processing systems through the lens of information theory.
  arXiv  Detail & Related papers  (2021-01-28T00:01:45Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• AttendNets: Tiny Deep Image Recognition Neural Networks for the Edge via Visual Attention Condensers [81.17461895644003]
  We introduce AttendNets, low-precision, highly compact deep neural networks tailored for on-device image recognition. AttendNets possess deep self-attention architectures based on visual attention condensers. Results show AttendNets have significantly lower architectural and computational complexity when compared to several deep neural networks.
  arXiv  Detail & Related papers  (2020-09-30T01:53:17Z)
• HyNNA: Improved Performance for Neuromorphic Vision Sensor based Surveillance using Hybrid Neural Network Architecture [7.293414498855147]
  We improve on a recently proposed hybrid event-frame approach by using morphological image processing algorithms for region proposal. We also address the low-power requirement for object detection and classification by exploring various convolutional neural network (CNN) architectures. Specifically, we compare the results obtained from our object detection framework against the state-of-the-art low-power NVS surveillance system and show an improved accuracy of 82.16% from 63.1%.
  arXiv  Detail & Related papers  (2020-03-19T07:18:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.