Max-Margin is Dead, Long Live Max-Margin!
- URL: http://arxiv.org/abs/2105.15069v1
- Date: Mon, 31 May 2021 15:55:52 GMT
- Title: Max-Margin is Dead, Long Live Max-Margin!
- Authors: Alex Nowak-Vila, Alessandro Rudi, Francis Bach
- Abstract summary: We show that Max-Margin loss can only be consistent to the classification task under highly restrictive assumptions on the discrete loss measuring the error between distances.
We introduce Restricted-Max-Margin, where the loss-augmented scores is maintained, but performed over a subset of the original domain.
- Score: 87.90853526750716
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The foundational concept of Max-Margin in machine learning is ill-posed for
output spaces with more than two labels such as in structured prediction. In
this paper, we show that the Max-Margin loss can only be consistent to the
classification task under highly restrictive assumptions on the discrete loss
measuring the error between outputs. These conditions are satisfied by
distances defined in tree graphs, for which we prove consistency, thus being
the first losses shown to be consistent for Max-Margin beyond the binary
setting. We finally address these limitations by correcting the concept of
Max-Margin and introducing the Restricted-Max-Margin, where the maximization of
the loss-augmented scores is maintained, but performed over a subset of the
original domain. The resulting loss is also a generalization of the binary
support vector machine and it is consistent under milder conditions on the
discrete loss.
Related papers
- Spectral Aware Softmax for Visible-Infrared Person Re-Identification [123.69049942659285]
Visible-infrared person re-identification (VI-ReID) aims to match specific pedestrian images from different modalities.
Existing methods still follow the softmax loss training paradigm, which is widely used in single-modality classification tasks.
We propose the spectral-aware softmax (SA-Softmax) loss, which can fully explore the embedding space with the modality information.
arXiv Detail & Related papers (2023-02-03T02:57:18Z) - Joint Discriminative and Metric Embedding Learning for Person
Re-Identification [8.137833258504381]
Person re-identification is a challenging task because of the high intra-class variance induced by the unrestricted nuisance factors of variations.
Recent approaches postulate that powerful architectures have the capacity to learn feature representations invariant to nuisance factors.
arXiv Detail & Related papers (2022-12-28T22:08:42Z) - A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized
Linear Models [33.36787620121057]
We prove a new generalization bound that shows for any class of linear predictors in Gaussian space.
We use our finite-sample bound to directly recover the "optimistic rate" of Zhou et al. (2021)
We show that application of our bound generalization using localized Gaussian width will generally be sharp for empirical risk minimizers.
arXiv Detail & Related papers (2022-10-21T16:16:55Z) - Calibrating Segmentation Networks with Margin-based Label Smoothing [19.669173092632]
We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses.
These losses could be viewed as approximations of a linear penalty imposing equality constraints on logit distances.
We propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.
arXiv Detail & Related papers (2022-09-09T20:21:03Z) - Learning Towards the Largest Margins [83.7763875464011]
Loss function should promote the largest possible margins for both classes and samples.
Not only does this principled framework offer new perspectives to understand and interpret existing margin-based losses, but it can guide the design of new tools.
arXiv Detail & Related papers (2022-06-23T10:03:03Z) - On the Equity of Nuclear Norm Maximization in Unsupervised Domain
Adaptation [53.29437277730871]
Nuclear norm has shown the power to enhance the transferability of unsupervised domain adaptation model.
Two new losses are proposed to maximize both predictive discriminability and equity, from the class level and the sample level.
arXiv Detail & Related papers (2022-04-12T07:55:47Z) - The Devil is in the Margin: Margin-based Label Smoothing for Network
Calibration [21.63888208442176]
In spite of the dominant performances of deep neural networks, recent works have shown that they are poorly calibrated.
We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses.
We propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.
arXiv Detail & Related papers (2021-11-30T14:21:47Z) - Real Additive Margin Softmax for Speaker Verification [14.226089039985151]
We show that AM-Softmax loss does not implement real max-margin training.
We present a Real AM-Softmax loss which involves a true margin function in the softmax training.
arXiv Detail & Related papers (2021-10-18T09:11:14Z) - Distribution of Classification Margins: Are All Data Equal? [61.16681488656473]
We motivate theoretically and show empirically that the area under the curve of the margin distribution on the training set is in fact a good measure of generalization.
The resulting subset of "high capacity" features is not consistent across different training runs.
arXiv Detail & Related papers (2021-07-21T16:41:57Z) - Consistent Structured Prediction with Max-Min Margin Markov Networks [84.60515484036239]
Max-margin methods for binary classification have been extended to the structured prediction setting under the name of max-margin Markov networks ($M3N$)
We overcome such limitations by defining the learning problem in terms of a "max-min" margin formulation, naming the resulting method max-min margin Markov networks ($M4N$)
Experiments on multi-class classification, ordinal regression, sequence prediction and ranking demonstrate the effectiveness of the proposed method.
arXiv Detail & Related papers (2020-07-02T10:48:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.