Expressive Monotonic Neural Networks

• URL: http://arxiv.org/abs/2307.07512v1
• Date: Fri, 14 Jul 2023 17:59:53 GMT
• Title: Expressive Monotonic Neural Networks
• Authors: Ouail Kitouni, Niklas Nolte, Michael Williams
• Abstract summary: The monotonic dependence of the outputs of a neural network on some of its inputs is a crucial inductive bias in many scenarios where domain knowledge dictates such behavior. We propose a weight-constrained architecture with a single residual connection to achieve exact monotonic dependence in any subset of the inputs. We show how the algorithm is used to train powerful, robust, and interpretable discriminators that achieve competitive performance.
• Score: 1.0128808054306184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The monotonic dependence of the outputs of a neural network on some of its inputs is a crucial inductive bias in many scenarios where domain knowledge dictates such behavior. This is especially important for interpretability and fairness considerations. In a broader context, scenarios in which monotonicity is important can be found in finance, medicine, physics, and other disciplines. It is thus desirable to build neural network architectures that implement this inductive bias provably. In this work, we propose a weight-constrained architecture with a single residual connection to achieve exact monotonic dependence in any subset of the inputs. The weight constraint scheme directly controls the Lipschitz constant of the neural network and thus provides the additional benefit of robustness. Compared to currently existing techniques used for monotonicity, our method is simpler in implementation and in theory foundations, has negligible computational overhead, is guaranteed to produce monotonic dependence, and is highly expressive. We show how the algorithm is used to train powerful, robust, and interpretable discriminators that achieve competitive performance compared to current state-of-the-art methods across various benchmarks, from social applications to the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider.

Related papers

• MonoKAN: Certified Monotonic Kolmogorov-Arnold Network [48.623199394622546]
  In certain applications, model predictions must align with expert-imposed requirements, sometimes exemplified by partial monotonicity constraints. We introduce a novel ANN architecture called MonoKAN, based on the KAN architecture and achieves certified partial monotonicity while enhancing interpretability. Our experiments demonstrate that MonoKAN not only enhances interpretability but also improves predictive performance across the majority of benchmarks, outperforming state-of-the-art monotonic approaches.
  arXiv  Detail & Related papers  (2024-09-17T11:10:59Z)
• Towards Practical Control of Singular Values of Convolutional Layers [65.25070864775793]
  Convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties. We offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity.
  arXiv  Detail & Related papers  (2022-11-24T19:09:44Z)
• Neural Dependencies Emerging from Learning Massive Categories [94.77992221690742]
  This work presents two astonishing findings on neural networks learned for large-scale image classification. 1) Given a well-trained model, the logits predicted for some category can be directly obtained by linearly combining the predictions of a few other categories. 2) Neural dependencies exist not only within a single model, but even between two independently learned models.
  arXiv  Detail & Related papers  (2022-11-21T09:42:15Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Interpretable part-whole hierarchies and conceptual-semantic relationships in neural networks [4.153804257347222]
  We present Agglomerator, a framework capable of providing a representation of part-whole hierarchies from visual cues. We evaluate our method on common datasets, such as SmallNORB, MNIST, FashionMNIST, CIFAR-10, and CIFAR-100.
  arXiv  Detail & Related papers  (2022-03-07T10:56:13Z)
• Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach [60.67748036747221]
  Implicit neural networks offer competitive performance and reduced memory consumption. They can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and computational framework for robustness verification of implicit neural networks.
  arXiv  Detail & Related papers  (2021-12-10T03:08:55Z)
• Robust and Provably Monotonic Networks [0.0]
  We present a new method to constrain the Lipschitz constant of dense deep learning models. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor decays in the LHCb realtime data-processing system.
  arXiv  Detail & Related papers  (2021-11-30T19:01:32Z)
• Interpretable Social Anchors for Human Trajectory Forecasting in Crowds [84.20437268671733]
  We propose a neural network-based system to predict human trajectory in crowds. We learn interpretable rule-based intents, and then utilise the expressibility of neural networks to model scene-specific residual. Our architecture is tested on the interaction-centric benchmark TrajNet++.
  arXiv  Detail & Related papers  (2021-05-07T09:22:34Z)
• Certified Monotonic Neural Networks [15.537695725617576]
  We propose to certify the monotonicity of the general piece-wise linear neural networks by solving a mixed integer linear programming problem. Our approach does not require human-designed constraints on the weight space and also yields more accurate approximation.
  arXiv  Detail & Related papers  (2020-11-20T04:58:13Z)
• The Curious Case of Convex Neural Networks [12.56278477726461]
  We show that the convexity constraints can be enforced on both fully connected and convolutional layers. We draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize and reduce the problem of overfitting; (b) Although heavily constrained, they outperform the base multi layer perceptrons and achieve similar performance as compared to base convolutional architectures.
  arXiv  Detail & Related papers  (2020-06-09T08:16:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.