Scalar Invariant Networks with Zero Bias
- URL: http://arxiv.org/abs/2211.08486v4
- Date: Mon, 29 May 2023 12:20:11 GMT
- Title: Scalar Invariant Networks with Zero Bias
- Authors: Chuqin Geng, Xiaojie Xu, Haolin Ye, Xujie Si
- Abstract summary: We show that zero-bias neural networks can perform comparably to biased networks for practical image classification tasks.
We prove that zero-bias neural networks are fair in predicting the zero image.
The robustness and fairness advantages of zero-bias neural networks may also indicate a promising path towards trustworthy and ethical AI.
- Score: 3.428731916567677
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Just like weights, bias terms are the learnable parameters of many popular
machine learning models, including neural networks. Biases are thought to
enhance the representational power of neural networks, enabling them to solve a
variety of tasks in computer vision. However, we argue that biases can be
disregarded for some image-related tasks such as image classification, by
considering the intrinsic distribution of images in the input space and desired
model properties from first principles. Our findings suggest that zero-bias
neural networks can perform comparably to biased networks for practical image
classification tasks. We demonstrate that zero-bias neural networks possess a
valuable property called scalar (multiplication) invariance. This means that
the prediction of the network remains unchanged when the contrast of the input
image is altered. We extend scalar invariance to more general cases, enabling
formal verification of certain convex regions of the input space. Additionally,
we prove that zero-bias neural networks are fair in predicting the zero image.
Unlike state-of-the-art models that may exhibit bias toward certain labels,
zero-bias networks have uniform belief in all labels. We believe dropping bias
terms can be considered as a geometric prior in designing neural network
architecture for image classification, which shares the spirit of adapting
convolutions as the transnational invariance prior. The robustness and fairness
advantages of zero-bias neural networks may also indicate a promising path
towards trustworthy and ethical AI.
Related papers
- An experimental comparative study of backpropagation and alternatives for training binary neural networks for image classification [1.0749601922718608]
Binary neural networks promise to reduce the size of deep neural network models.
They may allow the deployment of more powerful models on edge devices.
However, binary neural networks are still proven to be difficult to train using the backpropagation-based gradient descent scheme.
arXiv Detail & Related papers (2024-08-08T13:39:09Z) - Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
We propose to represent neural networks as computational graphs of parameters.
Our approach enables a single model to encode neural computational graphs with diverse architectures.
We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
arXiv Detail & Related papers (2024-03-18T18:01:01Z) - Closed-Form Interpretation of Neural Network Classifiers with Symbolic Gradients [0.7832189413179361]
I introduce a unified framework for finding a closed-form interpretation of any single neuron in an artificial neural network.
I demonstrate how to interpret neural network classifiers to reveal closed-form expressions of the concepts encoded in their decision boundaries.
arXiv Detail & Related papers (2024-01-10T07:47:42Z) - Emergence of Shape Bias in Convolutional Neural Networks through
Activation Sparsity [8.54598311798543]
Current deep-learning models for object recognition are heavily biased toward texture.
In contrast, human visual systems are known to be biased toward shape and structure.
We show that sparse coding, a ubiquitous principle in the brain, can in itself introduce shape bias into the network.
arXiv Detail & Related papers (2023-10-29T04:07:52Z) - Why do CNNs excel at feature extraction? A mathematical explanation [53.807657273043446]
We introduce a novel model for image classification, based on feature extraction, that can be used to generate images resembling real-world datasets.
In our proof, we construct piecewise linear functions that detect the presence of features, and show that they can be realized by a convolutional network.
arXiv Detail & Related papers (2023-07-03T10:41:34Z) - A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree
Spectral Bias of Neural Networks [79.28094304325116]
Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions.
We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets.
We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
arXiv Detail & Related papers (2023-05-16T20:06:01Z) - Neural networks trained with SGD learn distributions of increasing
complexity [78.30235086565388]
We show that neural networks trained using gradient descent initially classify their inputs using lower-order input statistics.
We then exploit higher-order statistics only later during training.
We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of universality in learning.
arXiv Detail & Related papers (2022-11-21T15:27:22Z) - Interpreting Bias in the Neural Networks: A Peek Into Representational
Similarity [0.0]
We investigate the performance and internal representational structure of convolution-based neural networks trained on biased data.
We specifically study similarities in representations, using Centered Kernel Alignment (CKA) for different objective functions.
We note that without progressive representational similarities among the layers of a neural network, the performance is less likely to be robust.
arXiv Detail & Related papers (2022-11-14T22:17:14Z) - Learning from Failure: Training Debiased Classifier from Biased
Classifier [76.52804102765931]
We show that neural networks learn to rely on spurious correlation only when it is "easier" to learn than the desired knowledge.
We propose a failure-based debiasing scheme by training a pair of neural networks simultaneously.
Our method significantly improves the training of the network against various types of biases in both synthetic and real-world datasets.
arXiv Detail & Related papers (2020-07-06T07:20:29Z) - Towards Understanding Hierarchical Learning: Benefits of Neural
Representations [160.33479656108926]
In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks.
We show that neural representation can achieve improved sample complexities compared with the raw input.
Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.
arXiv Detail & Related papers (2020-06-24T02:44:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.