Capsule Networks Do Not Need to Model Everything

• URL: http://arxiv.org/abs/2204.01298v2
• Date: Sat, 12 Jul 2025 15:35:47 GMT
• Title: Capsule Networks Do Not Need to Model Everything
• Authors: Riccardo Renzulli, Enzo Tartaglione, Marco Grangetto,
• Abstract summary: We present REM (Routing Entropy Minimization), a technique that minimizes the entropy of the parse tree-like structure.<n> REM drives the model parameters distribution towards low entropy configurations through a pruning mechanism.<n>This empowers capsules to learn more stable and succinct representations with fewer parameters and negligible performance loss.
• Score: 10.778550494961996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Capsule networks are biologically inspired neural networks that group neurons into vectors called capsules, each explicitly representing an object or one of its parts. The routing mechanism connects capsules in consecutive layers, forming a hierarchical structure between parts and objects, also known as a parse tree. Capsule networks often attempt to model all elements in an image, requiring large network sizes to handle complexities such as intricate backgrounds or irrelevant objects. However, this comprehensive modeling leads to increased parameter counts and computational inefficiencies. Our goal is to enable capsule networks to focus only on the object of interest, reducing the number of parse trees. We accomplish this with REM (Routing Entropy Minimization), a technique that minimizes the entropy of the parse tree-like structure. REM drives the model parameters distribution towards low entropy configurations through a pruning mechanism, significantly reducing the generation of intra-class parse trees. This empowers capsules to learn more stable and succinct representations with fewer parameters and negligible performance loss.

Related papers

• Weight transport through spike timing for robust local gradients [0.5236468296934584]
  plasticity in functional neural networks is frequently expressed as gradient descent on a cost. This imposes symmetry constraints that are difficult to reconcile with local computation. We introduce spike-based alignment learning, which uses spike timing statistics to extract and correct the asymmetry between effective reciprocal connections.
  arXiv  Detail & Related papers  (2025-03-04T14:05:39Z)
• ParseCaps: An Interpretable Parsing Capsule Network for Medical Image Diagnosis [6.273401483558281]
  This paper introduces a novel capsule network, ParseCaps, which utilizes the sparse axial attention routing and parse convolutional capsule layer to form a parse-tree-like structure. Experimental results on CE-MRI, PH$2$, and Derm7pt datasets show that ParseCaps not only outperforms other capsule network variants in classification accuracy, redundancy reduction and robustness, but also provides interpretable explanations.
  arXiv  Detail & Related papers  (2024-11-03T13:34:31Z)
• Hierarchical Object-Centric Learning with Capsule Networks [0.0]
  Capsule networks (CapsNets) were introduced to address convolutional neural networks limitations. This thesis investigates the intriguing aspects of CapsNets and focuses on three key questions to unlock their full potential.
  arXiv  Detail & Related papers  (2024-05-30T09:10:33Z)
• Semantic Loss Functions for Neuro-Symbolic Structured Prediction [74.18322585177832]
  We discuss the semantic loss, which injects knowledge about such structure, defined symbolically, into training. It is agnostic to the arrangement of the symbols, and depends only on the semantics expressed thereby. It can be combined with both discriminative and generative neural models.
  arXiv  Detail & Related papers  (2024-05-12T22:18:25Z)
• Leveraging Low-Rank and Sparse Recurrent Connectivity for Robust Closed-Loop Control [63.310780486820796]
  We show how a parameterization of recurrent connectivity influences robustness in closed-loop settings. We find that closed-form continuous-time neural networks (CfCs) with fewer parameters can outperform their full-rank, fully-connected counterparts.
  arXiv  Detail & Related papers  (2023-10-05T21:44:18Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Why Capsule Neural Networks Do Not Scale: Challenging the Dynamic Parse-Tree Assumption [16.223322939363033]
  Capsule neural networks replace simple, scalar-valued neurons with vector-valued capsules. CapsNet is the first actual implementation of the conceptual idea of capsule neural networks. No work was able to scale the CapsNet architecture to more reasonable-sized datasets.
  arXiv  Detail & Related papers  (2023-01-04T12:59:51Z)
• Parameter-Efficient Masking Networks [61.43995077575439]
  Advanced network designs often contain a large number of repetitive structures (e.g., Transformer) In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning masks. It leads to a new paradigm for model compression to diminish the model size.
  arXiv  Detail & Related papers  (2022-10-13T03:39:03Z)
• Towards Efficient Capsule Networks [7.1577508803778045]
  Capsule Networks were introduced to enhance explainability of a model, where each capsule is an explicit representation of an object or its parts. We show how pruning with Capsule Network achieves high generalization with less memory requirements, computational effort, and inference and training time.
  arXiv  Detail & Related papers  (2022-08-19T08:03:25Z)
• Simple and Efficient Heterogeneous Graph Neural Network [55.56564522532328]
  Heterogeneous graph neural networks (HGNNs) have powerful capability to embed rich structural and semantic information of a heterogeneous graph into node representations. Existing HGNNs inherit many mechanisms from graph neural networks (GNNs) over homogeneous graphs, especially the attention mechanism and the multi-layer structure. This paper conducts an in-depth and detailed study of these mechanisms and proposes Simple and Efficient Heterogeneous Graph Neural Network (SeHGNN)
  arXiv  Detail & Related papers  (2022-07-06T10:01:46Z)
• Receding Neuron Importances for Structured Pruning [11.375436522599133]
  Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. We introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration.
  arXiv  Detail & Related papers  (2022-04-13T14:08:27Z)
• BScNets: Block Simplicial Complex Neural Networks [79.81654213581977]
  Simplicial neural networks (SNN) have recently emerged as the newest direction in graph learning. We present Block Simplicial Complex Neural Networks (BScNets) model for link prediction. BScNets outperforms state-of-the-art models by a significant margin while maintaining low costs.
  arXiv  Detail & Related papers  (2021-12-13T17:35:54Z)
• Auto-Parsing Network for Image Captioning and Visual Question Answering [101.77688388554097]
  We propose an Auto-Parsing Network (APN) to discover and exploit the input data's hidden tree structures. Specifically, we impose a Probabilistic Graphical Model (PGM) parameterized by the attention operations on each self-attention layer to incorporate sparse assumption.
  arXiv  Detail & Related papers  (2021-08-24T08:14:35Z)
• Deformable Capsules for Object Detection [3.702343116848637]
  We introduce a new family of capsule networks, deformable capsules (textitDeformCaps), to address a very important problem in computer vision: object detection. We demonstrate that the proposed methods efficiently scale up to create the first-ever capsule network for object detection in the literature.
  arXiv  Detail & Related papers  (2021-04-11T15:36:30Z)
• Attentive Tree-structured Network for Monotonicity Reasoning [2.4366811507669124]
  We develop an attentive tree-structured neural network for monotonicity reasoning. It is designed to model the syntactic parse tree information from the sentence pair of a reasoning task. A self-attentive aggregator is used for aligning the representations of the premise and the hypothesis.
  arXiv  Detail & Related papers  (2021-01-03T01:29:48Z)
• Growing Deep Forests Efficiently with Soft Routing and Learned Connectivity [79.83903179393164]
  This paper further extends the deep forest idea in several important aspects. We employ a probabilistic tree whose nodes make probabilistic routing decisions, a.k.a., soft routing, rather than hard binary decisions. Experiments on the MNIST dataset demonstrate that our empowered deep forests can achieve better or comparable performance than [1],[3].
  arXiv  Detail & Related papers  (2020-12-29T18:05:05Z)
• Learning Compositional Structures for Deep Learning: Why Routing-by-agreement is Necessary [4.10184810111551]
  We present a formal grammar description of convolutional neural networks and capsule networks. We show that routing is an important part of capsule networks -- effectively answering recent work that has questioned its necessity.
  arXiv  Detail & Related papers  (2020-10-04T05:50:51Z)
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning [153.76012414126643]
  We propose a hierarchical compositional reasoning model called the "Linguistically driven Graph Capsule Network" The compositional process is guided by the linguistic parse tree. Specifically, we bind each capsule in the lowest layer to bridge the linguistic embedding of a single word in the original question with visual evidence. Experiments on the CLEVR dataset, CLEVR compositional generation test, and FigureQA dataset demonstrate the effectiveness and composition generalization ability of our end-to-end model.
  arXiv  Detail & Related papers  (2020-03-23T03:34:25Z)
• Subspace Capsule Network [85.69796543499021]
  SubSpace Capsule Network (SCN) exploits the idea of capsule networks to model possible variations in the appearance or implicitly defined properties of an entity. SCN can be applied to both discriminative and generative models without incurring computational overhead compared to CNN during test time.
  arXiv  Detail & Related papers  (2020-02-07T17:51:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.