Emergence of Latent Binary Encoding in Deep Neural Network Classifiers

• URL: http://arxiv.org/abs/2310.08224v4
• Date: Tue, 28 May 2024 06:55:23 GMT
• Title: Emergence of Latent Binary Encoding in Deep Neural Network Classifiers
• Authors: Luigi Sbailò, Luca Ghiringhelli,
• Abstract summary: We investigate the emergence of binary encoding within the latent space of deep-neural-network classifiers. By analyzing several datasets of increasing complexity, we provide empirical evidence that the emergence of binary encoding dramatically enhances robustness.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate the emergence of binary encoding within the latent space of deep-neural-network classifiers. Such binary encoding is induced by the introduction of a linear penultimate layer, which employs during training a loss function specifically designed to compress the latent representations. As a result of a trade-off between compression and information retention, the network learns to assume only one of two possible values for each dimension in the latent space. The binary encoding is provoked by the collapse of all representations of the same class to the same point, which corresponds to the vertex of a hypercube. By analyzing several datasets of increasing complexity, we provide empirical evidence that the emergence of binary encoding dramatically enhances robustness while also significantly improving the reliability and generalization of the network.

Related papers

• Reversible Decoupling Network for Single Image Reflection Removal [15.763420129991255]
  High-level semantic clues tend to be compressed or discarded during layer-by-layer propagation. We propose a novel architecture called Reversible Decoupling Network (RDNet) RDNet employs a reversible encoder to secure valuable information while flexibly decoupling transmission- and reflection-relevant features during the forward pass.
  arXiv  Detail & Related papers  (2024-10-10T15:58:27Z)
• Regressions on quantum neural networks at maximal expressivity [0.0]
  We analyze the expressivity of a universal deep neural network that can be organized as a series of nested qubit rotations. The maximal expressive power increases with the depth of the network and the number of qubits, but is fundamentally bounded by the data encoding mechanism.
  arXiv  Detail & Related papers  (2023-11-10T14:43:24Z)
• BiBench: Benchmarking and Analyzing Network Binarization [72.59760752906757]
  Network binarization emerges as one of the most promising compression approaches offering extraordinary computation and memory savings. Common challenges of binarization, such as accuracy degradation and efficiency limitation, suggest that its attributes are not fully understood. We present BiBench, a rigorously designed benchmark with in-depth analysis for network binarization.
  arXiv  Detail & Related papers  (2023-01-26T17:17:16Z)
• BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance [54.214426436283134]
  Deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. We present a strong yet efficient binary neural network for KWS, namely BiFSMNv2, pushing it to the real-network accuracy performance. We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1x speedup and 20.2x storage-saving on edge hardware.
  arXiv  Detail & Related papers  (2022-11-13T18:31:45Z)
• Neural network is heterogeneous: Phase matters more [10.812772606528172]
  In complex-valued neural networks, we show that among different types of pruning, the weight matrix with only phase information preserved achieves the best accuracy. The conclusion can be generalized to real-valued neural networks, where signs take the place of phases.
  arXiv  Detail & Related papers  (2021-11-03T04:30:20Z)
• High-Capacity Expert Binary Networks [56.87581500474093]
  Network binarization is a promising hardware-aware direction for creating efficient deep models. Despite its memory and computational advantages, reducing the accuracy gap between binary models and their real-valued counterparts remains an unsolved challenging research problem. We propose Expert Binary Convolution, which, for the first time, tailors conditional computing to binary networks by learning to select one data-specific expert binary filter at a time conditioned on input features.
  arXiv  Detail & Related papers  (2020-10-07T17:58:10Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding [59.48857453699463]
  In sequence-to-sequence learning, the decoder relies on the attention mechanism to efficiently extract information from the encoder. Recent work has proposed to use representations from different encoder layers for diversified levels of information. We propose layer-wise multi-view decoding, where for each decoder layer, together with the representations from the last encoder layer, which serve as a global view, those from other encoder layers are supplemented for a stereoscopic view of the source sequences.
  arXiv  Detail & Related papers  (2020-05-16T20:00:39Z)
• BiDet: An Efficient Binarized Object Detector [96.19708396510894]
  We propose a binarized neural network learning method called BiDet for efficient object detection. Our BiDet fully utilizes the representational capacity of the binary neural networks for object detection by redundancy removal. Our method outperforms the state-of-the-art binary neural networks by a sizable margin.
  arXiv  Detail & Related papers  (2020-03-09T08:16:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.