Optimisation in Neurosymbolic Learning Systems
- URL: http://arxiv.org/abs/2401.10819v1
- Date: Fri, 19 Jan 2024 17:09:32 GMT
- Title: Optimisation in Neurosymbolic Learning Systems
- Authors: Emile van Krieken
- Abstract summary: We study neurosymbolic learning, where we have both data and background knowledge expressed using symbolic languages.
Probabilistic reasoning is a natural fit for neural networks, which we usually train to be probabilistic.
Our insight is to train a neural network with synthetic data to predict the result of probabilistic reasoning.
- Score: 1.450405446885067
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neurosymbolic AI aims to integrate deep learning with symbolic AI. This
integration has many promises, such as decreasing the amount of data required
to train a neural network, improving the explainability and interpretability of
answers given by models and verifying the correctness of trained systems. We
study neurosymbolic learning, where we have both data and background knowledge
expressed using symbolic languages. How do we connect the symbolic and neural
components to communicate this knowledge? One option is fuzzy reasoning, which
studies degrees of truth. For example, being tall is not a binary concept.
Instead, probabilistic reasoning studies the probability that something is true
or will happen. Our first research question studies how different forms of
fuzzy reasoning combine with learning. We find surprising results like a
connection to the Raven paradox stating we confirm "ravens are black" when we
observe a green apple. In this study, we did not use the background knowledge
when we deployed our models after training. In our second research question, we
studied how to use background knowledge in deployed models. We developed a new
neural network layer based on fuzzy reasoning. Probabilistic reasoning is a
natural fit for neural networks, which we usually train to be probabilistic.
However, they are expensive to compute and do not scale well to large tasks. In
our third research question, we study how to connect probabilistic reasoning
with neural networks by sampling to estimate averages, while in the final
research question, we study scaling probabilistic neurosymbolic learning to
much larger problems than before. Our insight is to train a neural network with
synthetic data to predict the result of probabilistic reasoning.
Related papers
- EXPLAIN, AGREE, LEARN: Scaling Learning for Neural Probabilistic Logic [14.618208661185365]
We propose a sampling based objective to scale learning to more complex systems.
We prove that the objective has a bounded error with respect to the likelihood, which vanishes when increasing the sample count.
We then develop the EXPLAIN, AGREE, LEARN (EXAL) method that uses this objective.
In contrast to previous NeSy methods, EXAL can scale to larger problem sizes while retaining theoretical guarantees on the error.
arXiv Detail & Related papers (2024-08-15T13:07:51Z) - Hebbian Learning based Orthogonal Projection for Continual Learning of
Spiking Neural Networks [74.3099028063756]
We develop a new method with neuronal operations based on lateral connections and Hebbian learning.
We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities.
Our method consistently solves for spiking neural networks with nearly zero forgetting.
arXiv Detail & Related papers (2024-02-19T09:29:37Z) - Identifying Interpretable Visual Features in Artificial and Biological
Neural Systems [3.604033202771937]
Single neurons in neural networks are often interpretable in that they represent individual, intuitively meaningful features.
Many neurons exhibit $textitmixed selectivity$, i.e., they represent multiple unrelated features.
We propose an automated method for quantifying visual interpretability and an approach for finding meaningful directions in network activation space.
arXiv Detail & Related papers (2023-10-17T17:41:28Z) - Utility-Probability Duality of Neural Networks [4.871730595406078]
We propose an alternative utility-based explanation to the standard supervised learning procedure in deep learning.
The basic idea is to interpret the learned neural network not as a probability model but as an ordinal utility function.
We show that for all neural networks with softmax outputs, the SGD learning dynamic of maximum likelihood estimation can be seen as an iteration process.
arXiv Detail & Related papers (2023-05-24T08:09:07Z) - Brain-inspired Graph Spiking Neural Networks for Commonsense Knowledge
Representation and Reasoning [11.048601659933249]
How neural networks in the human brain represent commonsense knowledge is an important research topic in neuroscience, cognitive science, psychology, and artificial intelligence.
This work investigates how population encoding and spiking timing-dependent plasticity (STDP) mechanisms can be integrated into the learning of spiking neural networks.
The neuron populations of different communities together constitute the entire commonsense knowledge graph, forming a giant graph spiking neural network.
arXiv Detail & Related papers (2022-07-11T05:22:38Z) - The mathematics of adversarial attacks in AI -- Why deep learning is
unstable despite the existence of stable neural networks [69.33657875725747]
We prove that any training procedure based on training neural networks for classification problems with a fixed architecture will yield neural networks that are either inaccurate or unstable (if accurate)
The key is that the stable and accurate neural networks must have variable dimensions depending on the input, in particular, variable dimensions is a necessary condition for stability.
Our result points towards the paradox that accurate and stable neural networks exist, however, modern algorithms do not compute them.
arXiv Detail & Related papers (2021-09-13T16:19:25Z) - The Causal Neural Connection: Expressiveness, Learnability, and
Inference [125.57815987218756]
An object called structural causal model (SCM) represents a collection of mechanisms and sources of random variation of the system under investigation.
In this paper, we show that the causal hierarchy theorem (Thm. 1, Bareinboim et al., 2020) still holds for neural models.
We introduce a special type of SCM called a neural causal model (NCM), and formalize a new type of inductive bias to encode structural constraints necessary for performing causal inferences.
arXiv Detail & Related papers (2021-07-02T01:55:18Z) - The Neural Coding Framework for Learning Generative Models [91.0357317238509]
We propose a novel neural generative model inspired by the theory of predictive processing in the brain.
In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
arXiv Detail & Related papers (2020-12-07T01:20:38Z) - Extending Answer Set Programs with Neural Networks [2.512827436728378]
We propose NeurASP -- a simple extension of answer set programs by embracing neural networks.
We show that NeurASP can not only improve the perception accuracy of a pre-trained neural network, but also help to train a neural network better by giving restrictions through logic rules.
arXiv Detail & Related papers (2020-09-22T00:52:30Z) - Towards Understanding Hierarchical Learning: Benefits of Neural
Representations [160.33479656108926]
In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks.
We show that neural representation can achieve improved sample complexities compared with the raw input.
Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.
arXiv Detail & Related papers (2020-06-24T02:44:54Z) - A Heterogeneous Graph with Factual, Temporal and Logical Knowledge for
Question Answering Over Dynamic Contexts [81.4757750425247]
We study question answering over a dynamic textual environment.
We develop a graph neural network over the constructed graph, and train the model in an end-to-end manner.
arXiv Detail & Related papers (2020-04-25T04:53:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.