Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations

• URL: http://arxiv.org/abs/2312.15310v1
• Date: Sat, 23 Dec 2023 17:54:03 GMT
• Title: Towards Generalization in Subitizing with Neuro-Symbolic Loss using Holographic Reduced Representations
• Authors: Mohammad Mahmudul Alam, Edward Raff, Tim Oates
• Abstract summary: We show that adapting tools used in CogSci research can improve the subitizing generalization of CNNs and ViTs. We investigate how this neuro-symbolic approach to learning affects the subitizing capability of CNNs and ViTs. We find that ViTs perform considerably worse compared to CNNs in most respects on subitizing, except on one axis where an HRR-based loss provides improvement.
• Score: 49.22640185566807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While deep learning has enjoyed significant success in computer vision tasks over the past decade, many shortcomings still exist from a Cognitive Science (CogSci) perspective. In particular, the ability to subitize, i.e., quickly and accurately identify the small (less than 6) count of items, is not well learned by current Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs) when using a standard cross-entropy (CE) loss. In this paper, we demonstrate that adapting tools used in CogSci research can improve the subitizing generalization of CNNs and ViTs by developing an alternative loss function using Holographic Reduced Representations (HRRs). We investigate how this neuro-symbolic approach to learning affects the subitizing capability of CNNs and ViTs, and so we focus on specially crafted problems that isolate generalization to specific aspects of subitizing. Via saliency maps and out-of-distribution performance, we are able to empirically observe that the proposed HRR loss improves subitizing generalization though it does not completely solve the problem. In addition, we find that ViTs perform considerably worse compared to CNNs in most respects on subitizing, except on one axis where an HRR-based loss provides improvement.

Related papers

• Dynamical loss functions shape landscape topography and improve learning in artificial neural networks [0.9208007322096533]
  We show how to transform cross-entropy and mean squared error into dynamical loss functions. We show how they significantly improve validation accuracy for networks of varying sizes.
  arXiv  Detail & Related papers  (2024-10-14T16:27:03Z)
• Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networks [0.0]
  For classification, neural networks learn by minimizing cross-entropy, but are evaluated and compared using accuracy. This disparity suggests neural loss function search (NLFS), the search for a drop-in replacement loss function of cross-entropy for neural networks. We propose a new search space for NLFS that encourages more diverse loss functions to be explored.
  arXiv  Detail & Related papers  (2024-01-30T17:21:28Z)
• Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning [42.674679049746175]
  We propose to infuse human experts' intelligence and domain knowledge into the training of deep neural networks. We propose a novel eye-gaze-guided vision transformer (EG-ViT) for diagnosis with limited medical image data.
  arXiv  Detail & Related papers  (2022-05-25T03:29:10Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks [59.692017490560275]
  Adversarial training has been widely demonstrated to improve model's robustness against adversarial attacks. It remains unclear how the adversarial training could improve the generalization abilities of GNNs in the graph analytics problem. We construct the co-adversarial perturbation (CAP) optimization problem in terms of weights and features, and design the alternating adversarial perturbation algorithm to flatten the weight and feature loss landscapes alternately.
  arXiv  Detail & Related papers  (2021-10-28T02:28:13Z)
• Backpropagation with Biologically Plausible Spatio-Temporal Adjustment For Training Deep Spiking Neural Networks [5.484391472233163]
  The success of deep learning is inseparable from backpropagation. We propose a biological plausible spatial adjustment, which rethinks the relationship between membrane potential and spikes. Secondly, we propose a biologically plausible temporal adjustment making the error propagate across the spikes in the temporal dimension.
  arXiv  Detail & Related papers  (2021-10-17T15:55:51Z)
• Universal Adversarial Perturbations Through the Lens of Deep Steganography: Towards A Fourier Perspective [78.05383266222285]
  A human imperceptible perturbation can be generated to fool a deep neural network (DNN) for most images. A similar phenomenon has been observed in the deep steganography task, where a decoder network can retrieve a secret image back from a slightly perturbed cover image. We propose two new variants of universal perturbations: (1) Universal Secret Adversarial Perturbation (USAP) that simultaneously achieves attack and hiding; (2) high-pass UAP (HP-UAP) that is less visible to the human eye.
  arXiv  Detail & Related papers  (2021-02-12T12:26:39Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting [135.0863818867184]
  artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks. ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
  arXiv  Detail & Related papers  (2020-11-12T06:06:33Z)
• Anomalous diffusion dynamics of learning in deep neural networks [0.0]
  Learning in deep neural networks (DNNs) is implemented through minimizing a highly non-equilibrium loss function. We present a novel account of how such effective deep learning emerges through the interactions of the fractal-like structure of the loss landscape.
  arXiv  Detail & Related papers  (2020-09-22T14:57:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.