Classification and Reconstruction Processes in Deep Predictive Coding Networks: Antagonists or Allies?

• URL: http://arxiv.org/abs/2401.09237v1
• Date: Wed, 17 Jan 2024 14:34:32 GMT
• Title: Classification and Reconstruction Processes in Deep Predictive Coding Networks: Antagonists or Allies?
• Authors: Jan Rathjens and Laurenz Wiskott
• Abstract summary: Predictive coding-inspired deep networks for visual computing integrate classification and reconstruction processes in shared intermediate layers. We take a critical look at how classifying and reconstructing interact in deep learning architectures. Our findings underscore a significant challenge: Classification-driven information diminishes reconstruction-driven information in intermediate layers' shared representations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predictive coding-inspired deep networks for visual computing integrate classification and reconstruction processes in shared intermediate layers. Although synergy between these processes is commonly assumed, it has yet to be convincingly demonstrated. In this study, we take a critical look at how classifying and reconstructing interact in deep learning architectures. Our approach utilizes a purposefully designed family of model architectures reminiscent of autoencoders, each equipped with an encoder, a decoder, and a classification head featuring varying modules and complexities. We meticulously analyze the extent to which classification- and reconstruction-driven information can seamlessly coexist within the shared latent layer of the model architectures. Our findings underscore a significant challenge: Classification-driven information diminishes reconstruction-driven information in intermediate layers' shared representations and vice versa. While expanding the shared representation's dimensions or increasing the network's complexity can alleviate this trade-off effect, our results challenge prevailing assumptions in predictive coding and offer guidance for future iterations of predictive coding concepts in deep networks.

Related papers

• Visualization Of Class Activation Maps To Explain AI Classification Of Network Packet Captures [0.0]
  The number of connections and the addition of new applications in our networks causes a vast amount of log data. Deep learning methods provide both feature extraction and classification from data in a single system. We present a visual interactive tool that combines the classification of network data with an explanation technique to form an interface between experts, algorithms, and data.
  arXiv  Detail & Related papers  (2022-09-05T16:34:43Z)
• The Neural Race Reduction: Dynamics of Abstraction in Gated Networks [12.130628846129973]
  We introduce the Gated Deep Linear Network framework that schematizes how pathways of information flow impact learning dynamics. We derive an exact reduction and, for certain cases, exact solutions to the dynamics of learning. Our work gives rise to general hypotheses relating neural architecture to learning and provides a mathematical approach towards understanding the design of more complex architectures.
  arXiv  Detail & Related papers  (2022-07-21T12:01:03Z)
• Rank Diminishing in Deep Neural Networks [71.03777954670323]
  Rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear.
  arXiv  Detail & Related papers  (2022-06-13T12:03:32Z)
• AIGenC: An AI generalisation model via creativity [1.933681537640272]
  Inspired by cognitive theories of creativity, this paper introduces a computational model (AIGenC) It lays down the necessary components to enable artificial agents to learn, use and generate transferable representations. We discuss the model's capability to yield better out-of-distribution generalisation in artificial agents.
  arXiv  Detail & Related papers  (2022-05-19T17:43:31Z)
• Augmenting Convolutional networks with attention-based aggregation [55.97184767391253]
  We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth) It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption.
  arXiv  Detail & Related papers  (2021-12-27T14:05:41Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• SIRe-Networks: Skip Connections over Interlaced Multi-Task Learning and Residual Connections for Structure Preserving Object Classification [28.02302915971059]
  In this paper, we introduce an interlaced multi-task learning strategy, defined SIRe, to reduce the vanishing gradient in relation to the object classification task. The presented methodology directly improves a convolutional neural network (CNN) by enforcing the input image structure preservation through auto-encoders. To validate the presented methodology, a simple CNN and various implementations of famous networks are extended via the SIRe strategy and extensively tested on the CIFAR100 dataset.
  arXiv  Detail & Related papers  (2021-10-06T13:54:49Z)
• Denoise and Contrast for Category Agnostic Shape Completion [48.66519783934386]
  We present a deep learning model that exploits the power of self-supervision to perform 3D point cloud completion. A denoising pretext task provides the network with the needed local cues, decoupled from the high-level semantics. contrastive learning maximizes the agreement between variants of the same shape with different missing portions.
  arXiv  Detail & Related papers  (2021-03-30T20:33:24Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Visual Concept Reasoning Networks [93.99840807973546]
  A split-transform-merge strategy has been broadly used as an architectural constraint in convolutional neural networks for visual recognition tasks. We propose to exploit this strategy and combine it with our Visual Concept Reasoning Networks (VCRNet) to enable reasoning between high-level visual concepts. Our proposed model, VCRNet, consistently improves the performance by increasing the number of parameters by less than 1%.
  arXiv  Detail & Related papers  (2020-08-26T20:02:40Z)
• Learn Class Hierarchy using Convolutional Neural Networks [0.9569316316728905]
  We propose a new architecture for hierarchical classification of images, introducing a stack of deep linear layers with cross-entropy loss functions and center loss combined. We experimentally show that our hierarchical classifier presents advantages to the traditional classification approaches finding application in computer vision tasks.
  arXiv  Detail & Related papers  (2020-05-18T12:06:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.