PReLU: Yet Another Single-Layer Solution to the XOR Problem

• URL: http://arxiv.org/abs/2409.10821v1
• Date: Tue, 17 Sep 2024 01:28:40 GMT
• Title: PReLU: Yet Another Single-Layer Solution to the XOR Problem
• Abstract summary: We show that a single-layer neural network using Parametric Rectified Linear Unit (PReLU) activation can solve the XOR problem. Our results show that the single-layer PReLU network can achieve 100% success rate in a wider range of learning rates.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper demonstrates that a single-layer neural network using Parametric Rectified Linear Unit (PReLU) activation can solve the XOR problem, a simple fact that has been overlooked so far. We compare this solution to the multi-layer perceptron (MLP) and the Growing Cosine Unit (GCU) activation function and explain why PReLU enables this capability. Our results show that the single-layer PReLU network can achieve 100\% success rate in a wider range of learning rates while using only three learnable parameters.

Related papers

• DeepShare: Sharing ReLU Across Channels and Layers for Efficient Private Inference [16.43629306994231]
  Private Inference (PI) uses cryptographic primitives to perform privacy preserving machine learning.<n>A major computational bottleneck of PI is the calculation of the gate (i.e., ReLU)<n>We propose a new activation module where the DReLU operation is only performed on a subset of the channels.<n>We show that this formulation can drastically reduce the number of DReLU operations in resnet type network.
  arXiv  Detail & Related papers  (2025-12-19T09:50:23Z)
• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving [37.94354699202412]
  We show that higher solution divergence is positively related to better problem-solving abilities across various models.<n>We propose solution divergence as a novel metric that can support both SFT and RL strategies.
  arXiv  Detail & Related papers  (2025-09-26T15:27:50Z)
• Universal Model Routing for Efficient LLM Inference [72.65083061619752]
  We consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. We prove that these strategies are estimates of a theoretically optimal routing rule, and provide an excess risk bound to quantify their errors.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• TeLU Activation Function for Fast and Stable Deep Learning [1.9116784879310025]
  Hyperbolic Tangent Exponential Linear Unit (TeLU) is a neural network hidden activation function defined as TeLU(x)=xtanh(exp(x)) TeLU's design is grounded in the core principles of key activation functions, achieving strong convergence. Our results highlight TeLU's potential to set a new standard in activation functions, driving more efficient and stable learning in deep neural networks.
  arXiv  Detail & Related papers  (2024-12-28T20:50:08Z)
• Activation function optimization method: Learnable series linear units (LSLUs) [12.089173508371246]
  We propose a series-based learnable ac-tivation function called LSLU (Learnable Series Linear Units) This method simplifies deep learning networks while im-proving accuracy. We evaluate LSLU's performance on CIFAR10, CIFAR100, and specific task datasets (e.g., Silkworm)
  arXiv  Detail & Related papers  (2024-08-28T11:12:27Z)
• ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs [91.31204876440765]
  We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold. To find the most efficient activation function for sparse computation, we propose a systematic framework. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$2$.
  arXiv  Detail & Related papers  (2024-02-06T08:45:51Z)
• Robustly Learning a Single Neuron via Sharpness [45.40045240987339]
  We study the problem of learning a single neuron with respect to the $L2$-loss in the presence of adversarial label noise. We give an efficient algorithm that, for a broad family of activations including ReLUs, approximates the optimal $L2$-error within a constant factor.
  arXiv  Detail & Related papers  (2023-06-13T16:34:02Z)
• Delaunay-Triangulation-Based Learning with Hessian Total-Variation Regularization [21.69222364939501]
  In this paper, we propose an alternative method that leverages the expressivity of CPWL functions. Our approach relies on the partitioning of the domain of the CPWL function by a Delaunay triangulation. Our experiments validate the usage of our method in low-dimensional scenarios.
  arXiv  Detail & Related papers  (2022-08-16T14:55:05Z)
• AMS-Net: Adaptive Multiscale Sparse Neural Network with Interpretable Basis Expansion for Multiphase Flow Problems [8.991619150027267]
  We propose an adaptive sparse learning algorithm that can be applied to learn the physical processes and obtain a sparse representation of the solution given a large snapshot space. The information of the basis functions are incorporated in the loss function, which minimizes the differences between the downscaled reduced order solutions and reference solutions at multiple time steps. More numerical tests are performed on two-phase multiscale flow problems to show the capability and interpretability of the proposed method on complicated applications.
  arXiv  Detail & Related papers  (2022-07-24T13:12:43Z)
• Deep Unsupervised Learning for Generalized Assignment Problems: A Case-Study of User-Association in Wireless Networks [11.42707683459227]
  We propose a novel deep unsupervised learning (DUL) approach to solve the generalized assignment problems (GAP) in a time-efficient manner. In particular, we propose a new approach that facilitates to train a deep neural network (DNN) using a customized loss function. Numerical results demonstrate that the proposed DUL approach provides near-optimal results with significantly lower time-complexity.
  arXiv  Detail & Related papers  (2021-03-26T16:07:02Z)
• Self Sparse Generative Adversarial Networks [73.590634413751]
  Generative Adversarial Networks (GANs) are an unsupervised generative model that learns data distribution through adversarial training. We propose a Self Sparse Generative Adversarial Network (Self-Sparse GAN) that reduces the parameter space and alleviates the zero gradient problem.
  arXiv  Detail & Related papers  (2021-01-26T04:49:12Z)
• Learning by Fixing: Solving Math Word Problems with Weak Supervision [70.62896781438694]
  Previous neural solvers of math word problems (MWPs) are learned with full supervision and fail to generate diverse solutions. We introduce a textitweakly-supervised paradigm for learning MWPs. Our method only requires the annotations of the final answers and can generate various solutions for a single problem.
  arXiv  Detail & Related papers  (2020-12-19T03:10:21Z)
• Comparisons among different stochastic selection of activation layers for convolutional neural networks for healthcare [77.99636165307996]
  We classify biomedical images using ensembles of neural networks. We select our activations among the following ones: ReLU, leaky ReLU, Parametric ReLU, ELU, Adaptive Piecewice Linear Unit, S-Shaped ReLU, Swish, Mish, Mexican Linear Unit, Parametric Deformable Linear Unit, Soft Root Sign.
  arXiv  Detail & Related papers  (2020-11-24T01:53:39Z)
• Physarum Powered Differentiable Linear Programming Layers and Applications [48.77235931652611]
  We propose an efficient and differentiable solver for general linear programming problems. We show the use of our solver in a video segmentation task and meta-learning for few-shot learning.
  arXiv  Detail & Related papers  (2020-04-30T01:50:37Z)
• Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units [68.30422112784355]
  We propose a new gating mechanism within general gated recurrent neural networks to handle this issue. The proposed gates directly short connect the extracted input features to the outputs of vanilla gates. We verify the proposed gating mechanism on three popular types of gated RNNs including LSTM, GRU and MGU.
  arXiv  Detail & Related papers  (2020-02-26T07:51:38Z)
• Gaussian Error Linear Units (GELUs) [58.195342948092964]
  We propose a neural network activation function that weights inputs by their value, rather than gates by their sign. We find performance improvements across all considered computer vision, natural language processing, and speech tasks.
  arXiv  Detail & Related papers  (2016-06-27T19:20:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.