Effective and Interpretable Information Aggregation with Capacity Networks

• URL: http://arxiv.org/abs/2207.12013v1
• Date: Mon, 25 Jul 2022 09:45:16 GMT
• Title: Effective and Interpretable Information Aggregation with Capacity Networks
• Authors: Markus Zopf
• Abstract summary: Capacity networks generate multiple interpretable intermediate results which can be aggregated in a semantically meaningful space. Our experiments show that implementing this simple inductive bias leads to improvements over different encoder-decoder architectures.
• Score: 3.4012007729454807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How to aggregate information from multiple instances is a key question multiple instance learning. Prior neural models implement different variants of the well-known encoder-decoder strategy according to which all input features are encoded a single, high-dimensional embedding which is then decoded to generate an output. In this work, inspired by Choquet capacities, we propose Capacity networks. Unlike encoder-decoders, Capacity networks generate multiple interpretable intermediate results which can be aggregated in a semantically meaningful space to obtain the final output. Our experiments show that implementing this simple inductive bias leads to improvements over different encoder-decoder architectures in a wide range of experiments. Moreover, the interpretable intermediate results make Capacity networks interpretable by design, which allows a semantically meaningful inspection, evaluation, and regularization of the network internals.

Related papers

• Towards Diverse Binary Segmentation via A Simple yet General Gated Network [71.19503376629083]
  We propose a simple yet general gated network (GateNet) to tackle binary segmentation tasks. With the help of multi-level gate units, the valuable context information from the encoder can be selectively transmitted to the decoder. We introduce a "Fold" operation to improve the atrous convolution and form a novel folded atrous convolution.
  arXiv  Detail & Related papers  (2023-03-18T11:26:36Z)
• Complexity of Representations in Deep Learning [2.0219767626075438]
  We analyze the effectiveness of the learned representations in separating the classes from a data complexity perspective. We show how the data complexity evolves through the network, how it changes during training, and how it is impacted by the network design and the availability of training samples.
  arXiv  Detail & Related papers  (2022-09-01T15:20:21Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Exploring and Exploiting Multi-Granularity Representations for Machine Reading Comprehension [13.191437539419681]
  We propose a novel approach called Adaptive Bidirectional Attention-Capsule Network (ABA-Net) ABA-Net adaptively exploits the source representations of different levels to the predictor. We set the new state-of-the-art performance on the SQuAD 1.0 dataset.
  arXiv  Detail & Related papers  (2022-08-18T10:14:32Z)
• Multi-scale and Cross-scale Contrastive Learning for Semantic Segmentation [5.281694565226513]
  We apply contrastive learning to enhance the discriminative power of the multi-scale features extracted by semantic segmentation networks. By first mapping the encoder's multi-scale representations to a common feature space, we instantiate a novel form of supervised local-global constraint.
  arXiv  Detail & Related papers  (2022-03-25T01:24:24Z)
• Latent Code-Based Fusion: A Volterra Neural Network Approach [21.25021807184103]
  We propose a deep structure encoder using the recently introduced Volterra Neural Networks (VNNs) We show that the proposed approach demonstrates a much-improved sample complexity over CNN-based auto-encoder with a superb robust classification performance.
  arXiv  Detail & Related papers  (2021-04-10T18:29:01Z)
• On the Sub-Layer Functionalities of Transformer Decoder [74.83087937309266]
  We study how Transformer-based decoders leverage information from the source and target languages. Based on these insights, we demonstrate that the residual feed-forward module in each Transformer decoder layer can be dropped with minimal loss of performance.
  arXiv  Detail & Related papers  (2020-10-06T11:50:54Z)
• Beyond Single Stage Encoder-Decoder Networks: Deep Decoders for Semantic Image Segmentation [56.44853893149365]
  Single encoder-decoder methodologies for semantic segmentation are reaching their peak in terms of segmentation quality and efficiency per number of layers. We propose a new architecture based on a decoder which uses a set of shallow networks for capturing more information content. In order to further improve the architecture we introduce a weight function which aims to re-balance classes to increase the attention of the networks to under-represented objects.
  arXiv  Detail & Related papers  (2020-07-19T18:44:34Z)
• Suppress and Balance: A Simple Gated Network for Salient Object Detection [89.88222217065858]
  We propose a simple gated network (GateNet) to solve both issues at once. With the help of multilevel gate units, the valuable context information from the encoder can be optimally transmitted to the decoder. In addition, we adopt the atrous spatial pyramid pooling based on the proposed "Fold" operation (Fold-ASPP) to accurately localize salient objects of various scales.
  arXiv  Detail & Related papers  (2020-07-16T02:00:53Z)
• Rethinking and Improving Natural Language Generation with Layer-Wise Multi-View Decoding [59.48857453699463]
  In sequence-to-sequence learning, the decoder relies on the attention mechanism to efficiently extract information from the encoder. Recent work has proposed to use representations from different encoder layers for diversified levels of information. We propose layer-wise multi-view decoding, where for each decoder layer, together with the representations from the last encoder layer, which serve as a global view, those from other encoder layers are supplemented for a stereoscopic view of the source sequences.
  arXiv  Detail & Related papers  (2020-05-16T20:00:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.