Counterexample-Guided Learning of Monotonic Neural Networks
- URL: http://arxiv.org/abs/2006.08852v1
- Date: Tue, 16 Jun 2020 01:04:26 GMT
- Title: Counterexample-Guided Learning of Monotonic Neural Networks
- Authors: Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, Guy Van den
Broeck
- Abstract summary: We focus on monotonicity constraints, which are common and require that the function's output increases with increasing values of specific input features.
We develop a counterexample-guided technique to provably enforce monotonicity constraints at prediction time.
We also propose a technique to use monotonicity as an inductive bias for deep learning.
- Score: 32.73558242733049
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The widespread adoption of deep learning is often attributed to its automatic
feature construction with minimal inductive bias. However, in many real-world
tasks, the learned function is intended to satisfy domain-specific constraints.
We focus on monotonicity constraints, which are common and require that the
function's output increases with increasing values of specific input features.
We develop a counterexample-guided technique to provably enforce monotonicity
constraints at prediction time. Additionally, we propose a technique to use
monotonicity as an inductive bias for deep learning. It works by iteratively
incorporating monotonicity counterexamples in the learning process. Contrary to
prior work in monotonic learning, we target general ReLU neural networks and do
not further restrict the hypothesis space. We have implemented these techniques
in a tool called COMET. Experiments on real-world datasets demonstrate that our
approach achieves state-of-the-art results compared to existing monotonic
learners, and can improve the model quality compared to those that were trained
without taking monotonicity constraints into account.
Related papers
- Expressive Monotonic Neural Networks [1.0128808054306184]
The monotonic dependence of the outputs of a neural network on some of its inputs is a crucial inductive bias in many scenarios where domain knowledge dictates such behavior.
We propose a weight-constrained architecture with a single residual connection to achieve exact monotonic dependence in any subset of the inputs.
We show how the algorithm is used to train powerful, robust, and interpretable discriminators that achieve competitive performance.
arXiv Detail & Related papers (2023-07-14T17:59:53Z) - How to address monotonicity for model risk management? [1.0878040851638]
This paper studies transparent neural networks in the presence of three types of monotonicity: individual monotonicity, weak pairwise monotonicity, and strong pairwise monotonicity.
As a means of achieving monotonicity while maintaining transparency, we propose the monotonic groves of neural additive models.
arXiv Detail & Related papers (2023-04-28T04:21:02Z) - Theoretical Characterization of the Generalization Performance of
Overfitted Meta-Learning [70.52689048213398]
This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features.
We find new and interesting properties that do not exist in single-task linear regression.
Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
arXiv Detail & Related papers (2023-04-09T20:36:13Z) - Generalization Analysis for Contrastive Representation Learning [80.89690821916653]
Existing generalization error bounds depend linearly on the number $k$ of negative examples.
We establish novel generalization bounds for contrastive learning which do not depend on $k$, up to logarithmic terms.
arXiv Detail & Related papers (2023-02-24T01:03:56Z) - Constrained Monotonic Neural Networks [0.685316573653194]
Wider adoption of neural networks in many critical domains such as finance and healthcare is being hindered by the need to explain their predictions.
Monotonicity constraint is one of the most requested properties in real-world scenarios.
We show it can approximate any continuous monotone function on a compact subset of $mathbbRn$.
arXiv Detail & Related papers (2022-05-24T04:26:10Z) - Characterizing and overcoming the greedy nature of learning in
multi-modal deep neural networks [62.48782506095565]
We show that due to the greedy nature of learning in deep neural networks, models tend to rely on just one modality while under-fitting the other modalities.
We propose an algorithm to balance the conditional learning speeds between modalities during training and demonstrate that it indeed addresses the issue of greedy learning.
arXiv Detail & Related papers (2022-02-10T20:11:21Z) - Generalization of Neural Combinatorial Solvers Through the Lens of
Adversarial Robustness [68.97830259849086]
Most datasets only capture a simpler subproblem and likely suffer from spurious features.
We study adversarial robustness - a local generalization property - to reveal hard, model-specific instances and spurious features.
Unlike in other applications, where perturbation models are designed around subjective notions of imperceptibility, our perturbation models are efficient and sound.
Surprisingly, with such perturbations, a sufficiently expressive neural solver does not suffer from the limitations of the accuracy-robustness trade-off common in supervised learning.
arXiv Detail & Related papers (2021-10-21T07:28:11Z) - Certified Monotonic Neural Networks [15.537695725617576]
We propose to certify the monotonicity of the general piece-wise linear neural networks by solving a mixed integer linear programming problem.
Our approach does not require human-designed constraints on the weight space and also yields more accurate approximation.
arXiv Detail & Related papers (2020-11-20T04:58:13Z) - Learning What Makes a Difference from Counterfactual Examples and
Gradient Supervision [57.14468881854616]
We propose an auxiliary training objective that improves the generalization capabilities of neural networks.
We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task.
Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
arXiv Detail & Related papers (2020-04-20T02:47:49Z) - Exact Hard Monotonic Attention for Character-Level Transduction [76.66797368985453]
We show that neural sequence-to-sequence models that use non-monotonic soft attention often outperform popular monotonic models.
We develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns a latent alignment jointly while learning to transduce.
arXiv Detail & Related papers (2019-05-15T17:51:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.