Recent Advances in Large Margin Learning

• URL: http://arxiv.org/abs/2103.13598v1
• Date: Thu, 25 Mar 2021 04:12:00 GMT
• Title: Recent Advances in Large Margin Learning
• Authors: Yiwen Guo, Changshui Zhang
• Abstract summary: This paper serves as a survey of recent advances in large margin training and its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs) We generalize the formulation of classification margins from classical research to latest DNNs, summarize theoretical connections between the margin, network generalization, and robustness, and introduce recent efforts in enlarging the margins for DNNs comprehensively.
• Score: 63.982279380483526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper serves as a survey of recent advances in large margin training and its theoretical foundations, mostly for (nonlinear) deep neural networks (DNNs) that are probably the most prominent machine learning models for large-scale data in the community over the past decade. We generalize the formulation of classification margins from classical research to latest DNNs, summarize theoretical connections between the margin, network generalization, and robustness, and introduce recent efforts in enlarging the margins for DNNs comprehensively. Since the viewpoint of different methods is discrepant, we categorize them into groups for ease of comparison and discussion in the paper. Hopefully, our discussions and overview inspire new research work in the community that aim to improve the performance of DNNs, and we also point to directions where the large margin principle can be verified to provide theoretical evidence why certain regularizations for DNNs function well in practice. We managed to shorten the paper such that the crucial spirit of large margin learning and related methods are better emphasized.

Related papers

• Piecewise Linear Neural Networks and Deep Learning [27.02556725989978]
  PieceWise Linear Neural Networks (PWLNNs) have proven successful in various fields, most recently in deep learning. In 1977, the canonical representation pioneered the works of shallow PWLNNs learned by incremental designs. In 2010, the Rectified Linear Unit (ReLU) advocated the prevalence of PWLNNs in deep learning.
  arXiv  Detail & Related papers  (2022-06-18T08:41:42Z)
• Knowledge Enhanced Neural Networks for relational domains [83.9217787335878]
  We focus on a specific method, KENN, a Neural-Symbolic architecture that injects prior logical knowledge into a neural network. In this paper, we propose an extension of KENN for relational data.
  arXiv  Detail & Related papers  (2022-05-31T13:00:34Z)
• Subgroup Generalization and Fairness of Graph Neural Networks [12.88476464580968]
  We present a novel PAC-Bayesian analysis for GNNs under a non-IID semi-supervised learning setup. We further study an accuracy-(dis)parity-style (un)fairness of GNNs from a theoretical perspective.
  arXiv  Detail & Related papers  (2021-06-29T16:13:41Z)
• Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth [57.10183643449905]
  Graph Neural Networks (GNNs) have been studied from the lens of expressive power and generalization. We study the dynamics of GNNs by studying deep skip optimization. Our results provide first theoretical support for the success of GNNs.
  arXiv  Detail & Related papers  (2021-05-10T17:59:01Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• Deep Neural Networks Are Congestion Games: From Loss Landscape to Wardrop Equilibrium and Beyond [12.622643370707328]
  We argue that our work provides a very promising novel tool for analyzing the deep neural networks (DNNs) We show how one can benefit from the classic readily available results from the latter when analyzing the former.
  arXiv  Detail & Related papers  (2020-10-21T14:11:40Z)
• A Survey of Active Learning for Text Classification using Deep Neural Networks [1.2310316230437004]
  Natural language processing (NLP) and neural networks (NNs) have both undergone significant changes in recent years. For active learning (AL) purposes, NNs are, however, less commonly used -- despite their current popularity.
  arXiv  Detail & Related papers  (2020-08-17T12:53:20Z)
• On Connections between Regularizations for Improving DNN Robustness [67.28077776415724]
  This paper analyzes regularization terms proposed recently for improving the adversarial robustness of deep neural networks (DNNs) We study possible connections between several effective methods, including input-gradient regularization, Jacobian regularization, curvature regularization, and a cross-Lipschitz functional.
  arXiv  Detail & Related papers  (2020-07-04T23:43:32Z)
• A Chain Graph Interpretation of Real-World Neural Networks [58.78692706974121]
  We propose an alternative interpretation that identifies NNs as chain graphs (CGs) and feed-forward as an approximate inference procedure. The CG interpretation specifies the nature of each NN component within the rich theoretical framework of probabilistic graphical models. We demonstrate with concrete examples that the CG interpretation can provide novel theoretical support and insights for various NN techniques.
  arXiv  Detail & Related papers  (2020-06-30T14:46:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.