Related papers: Learning distinct features helps, provably

Learning distinct features helps, provably

URL: http://arxiv.org/abs/2106.06012v3
Date: Tue, 13 Jun 2023 07:34:21 GMT
Title: Learning distinct features helps, provably
Authors: Firas Laakom, Jenni Raitoharju, Alexandros Iosifidis, Moncef Gabbouj
Abstract summary: We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average $L$-distance between the hidden-layer features.
Score: 98.78384185493624
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: We study the diversity of the features learned by a two-layer neural network trained with the least squares loss. We measure the diversity by the average $L_2$-distance between the hidden-layer features and theoretically investigate how learning non-redundant distinct features affects the performance of the network. To do so, we derive novel generalization bounds depending on feature diversity based on Rademacher complexity for such networks. Our analysis proves that more distinct features at the network's units within the hidden layer lead to better generalization. We also show how to extend our results to deeper networks and different losses.

Related papers

Asymptotics of Learning with Deep Structured (Random) Features [9.366617422860543]
For a large class of feature maps we provide a tight characterisation of the test error associated with learning the readout layer. In some cases our results can capture feature maps learned by deep, finite-width neural networks trained under gradient descent.
arXiv Detail & Related papers (2024-02-21T18:35:27Z)
Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory [64.06519549649495]
We provide the definition of what we call functionally equivalent features. These features produce equivalent output under certain transformations. We propose an efficient algorithm named Iterative Feature Merging.
arXiv Detail & Related papers (2023-10-10T16:27:12Z)
Understanding Deep Neural Networks via Linear Separability of Hidden Layers [68.23950220548417]
We first propose Minkowski difference based linear separability measures (MD-LSMs) to evaluate the linear separability degree of two points sets. We demonstrate that there is a synchronicity between the linear separability degree of hidden layer outputs and the network training performance.
arXiv Detail & Related papers (2023-07-26T05:29:29Z)
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks [49.808194368781095]
We show that three-layer neural networks have provably richer feature learning capabilities than two-layer networks. This work makes progress towards understanding the provable benefit of three-layer neural networks over two-layer networks in the feature learning regime.
arXiv Detail & Related papers (2023-05-11T17:19:30Z)
Universal Representations: A Unified Look at Multiple Task and Domain Learning [37.27708297562079]
We propose a unified look at jointly learning multiple vision tasks and visual domains through universal representations. We show that universal representations achieve state-of-the-art performances in learning of multiple dense prediction problems. We also conduct multiple analysis through ablation and qualitative studies.
arXiv Detail & Related papers (2022-04-06T11:40:01Z)
Multi-scale Matching Networks for Semantic Correspondence [38.904735120815346]
The proposed method achieves state-of-the-art performance on three popular benchmarks with high computational efficiency. Our multi-scale matching network can be trained end-to-end easily with few additional learnable parameters.
arXiv Detail & Related papers (2021-07-31T10:57:24Z)
A neural anisotropic view of underspecification in deep learning [60.119023683371736]
We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
arXiv Detail & Related papers (2021-04-29T14:31:09Z)
ReMarNet: Conjoint Relation and Margin Learning for Small-Sample Image Classification [49.87503122462432]
We introduce a novel neural network termed Relation-and-Margin learning Network (ReMarNet) Our method assembles two networks of different backbones so as to learn the features that can perform excellently in both of the aforementioned two classification mechanisms. Experiments on four image datasets demonstrate that our approach is effective in learning discriminative features from a small set of labeled samples.
arXiv Detail & Related papers (2020-06-27T13:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.