Optical Phase Dropout in Diffractive Deep Neural Network

• URL: http://arxiv.org/abs/2011.14130v1
• Date: Sat, 28 Nov 2020 13:33:23 GMT
• Title: Optical Phase Dropout in Diffractive Deep Neural Network
• Abstract summary: Unitary learning serves to update unitary weights update in deep complex-valued neural network with full connections. The square matrix property of unitary weights induces that the function signal has a limited dimension that could not generalize well. Phase dropout in unitary space that is evolved from a complex dropout and has a statistical inference is formulated for the first time. The equivalence between digital and diffractive model determines compound modulations that could successfully circumvent the nonlinear activations physically implemented in [DN]2.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unitary learning is a backpropagation that serves to unitary weights update in deep complex-valued neural network with full connections, meeting a physical unitary prior in diffractive deep neural network ([DN]2). However, the square matrix property of unitary weights induces that the function signal has a limited dimension that could not generalize well. To address the overfitting problem that comes from the small samples loaded to [DN]2, an optical phase dropout trick is implemented. Phase dropout in unitary space that is evolved from a complex dropout and has a statistical inference is formulated for the first time. A synthetic mask recreated from random point apertures with random phase-shifting and its smothered modulation tailors the redundant links through incompletely sampling the input optical field at each diffractive layer. The physical features about the synthetic mask using different nonlinear activations are elucidated in detail. The equivalence between digital and diffractive model determines compound modulations that could successfully circumvent the nonlinear activations physically implemented in [DN]2. The numerical experiments verify the superiority of optical phase dropout in [DN]2 to enhance accuracy in 2D classification and recognition tasks-oriented.

Related papers

• Optical Neural Networks from Coherent Transient Dynamics in Waveguide QED [0.0]
  Optical neural networks promise ultrafast, low-energy information processing by performing computation directly with photons.<n>We propose an all-optical fully connected neural network architecture in which the basic neuronal functions are realized by coherent quantum dynamics.<n>Full-physics simulations demonstrate high classification accuracy on MNIST and colored-object recognition tasks.
  arXiv  Detail & Related papers  (2026-05-18T02:11:34Z)
• ShiftLIF: Efficient Multi-Level Spiking Neurons with Power-of-Two Quantization [17.71721533103354]
  ShiftLIF is a multi-level spiking neuron that maps membrane potentials to a logarithmically spaced power-of-two spike set.<n>We evaluate ShiftLIF on 10 datasets spanning wireless, acoustic, motion, and visual sensing tasks.
  arXiv  Detail & Related papers  (2026-05-03T13:20:53Z)
• Global stability of a Hebbian/anti-Hebbian network for principal subspace learning [8.27902169448946]
  How modifications at the synaptic level give to such computations at the network level remains an open question.<n>Global stability of the continuum limit of the synaptic dynamics show that the dynamics evolve in two phases.
  arXiv  Detail & Related papers  (2026-01-19T15:53:31Z)
• Phase Transitions between Accuracy Regimes in L2 regularized Deep Neural Networks [0.0]
  Increasing the L2 regularization of Deep Neural Networks (DNNs) causes a first-order phase transition into the under-parametrized phase.<n>We predict new transition points as the data complexity is increased and, in accordance with the theory of phase transitions, the existence of effects.
  arXiv  Detail & Related papers  (2025-05-10T11:02:30Z)
• Mask-PINNs: Regulating Feature Distributions in Physics-Informed Neural Networks [1.6984490081106065]
  Mask-PINNs regulates internal feature distributions through a smooth, learnable mask function applied pointwise across hidden layers.<n>We show consistent improvements in prediction accuracy, convergence stability, and robustness, with relative L2 errors reduced by up to two orders of magnitude over baseline models.
  arXiv  Detail & Related papers  (2025-05-09T15:38:52Z)
• Training Large-Scale Optical Neural Networks with Two-Pass Forward Propagation [0.0]
  This paper addresses the limitations in Optical Neural Networks (ONNs) related to training efficiency, nonlinear function implementation, and large input data processing. We introduce Two-Pass Forward Propagation, a novel training method that avoids specific nonlinear activation functions by modulating and re-entering error with random noise. We propose a new way to implement convolutional neural networks using simple neural networks in integrated optical systems.
  arXiv  Detail & Related papers  (2024-08-15T11:27:01Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition [67.95996816744251]
  State-of-the-art language models (LMs) represented by long-short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming increasingly complex and expensive for practical applications. Current quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different parts of LMs to quantization errors. Novel mixed precision neural network LM quantization methods are proposed in this paper.
  arXiv  Detail & Related papers  (2021-11-29T12:24:02Z)
• LocalDrop: A Hybrid Regularization for Deep Neural Networks [98.30782118441158]
  We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs) has been developed based on the proposed upper bound of the local Rademacher complexity.
  arXiv  Detail & Related papers  (2021-03-01T03:10:11Z)
• DIFFnet: Diffusion parameter mapping network generalized for input diffusion gradient schemes and bvalues [6.7487278071108525]
  A new deep neural network, referred to as DIFFnet, is developed to function as a generalized reconstruction tool of the diffusion-weighted signals. DIFFnet is evaluated for diffusion tensor imaging (DIFFnetDTI) and for neurite orientation dispersion and density imaging (DIFFnetNODDI) The results demonstrate accurate reconstruction of the diffusion parameters at substantially reduced processing time.
  arXiv  Detail & Related papers  (2021-02-04T07:45:36Z)
• Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samples [0.0]
  We introduce the so-called entangled weights, which compose weights of successive layers intertwined with suitable diagonal and invertible matrices depending on the activation functions and their shifts. We prove that entangled weights are completely and stably approximated by an efficient and robust algorithm. In terms of practical impact, our study shows that we can relate input-output information uniquely and stably to network parameters, providing a form of explainability.
  arXiv  Detail & Related papers  (2021-01-18T16:31:19Z)
• Quantal synaptic dilution enhances sparse encoding and dropout regularisation in deep networks [0.0]
  Dropout is a technique that silences the activity of unitsally while training deep networks to reduce overfitting. Here we introduce Quantal Synaptic Dilution (QSD), a biologically plausible model of dropout regularisation based on the quantal properties of neuronal synapses.
  arXiv  Detail & Related papers  (2020-09-28T09:29:49Z)
• Unitary Learning for Deep Diffractive Neural Network [0.0]
  We present a unitary learning protocol on deep diffractive neural network. The temporal-space evolution characteristic in unitary learning is formulated and elucidated. As a preliminary application, deep diffractive neural network with unitary learning is tentatively implemented on the 2D classification and verification tasks.
  arXiv  Detail & Related papers  (2020-08-17T07:16:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.