Revisiting Pooling through the Lens of Optimal Transport

• URL: http://arxiv.org/abs/2201.09191v1
• Date: Sun, 23 Jan 2022 06:20:39 GMT
• Title: Revisiting Pooling through the Lens of Optimal Transport
• Authors: Minjie Cheng and Hongteng Xu
• Abstract summary: We develop a novel and solid algorithmic pooling framework through the lens of optimal transport. We make the parameters of the UOT problem learnable, and accordingly, propose a generalized pooling layer called textitUOT-Pooling for neural networks. We test our UOT-Pooling layers in two application scenarios, including multi-instance learning (MIL) and graph embedding.
• Score: 25.309212446782684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pooling is one of the most significant operations in many machine learning models and tasks, whose implementation, however, is often empirical in practice. In this paper, we develop a novel and solid algorithmic pooling framework through the lens of optimal transport. In particular, we demonstrate that most existing pooling methods are equivalent to solving some specializations of an unbalanced optimal transport (UOT) problem. Making the parameters of the UOT problem learnable, we unify most existing pooling methods in the same framework, and accordingly, propose a generalized pooling layer called \textit{UOT-Pooling} for neural networks. Moreover, we implement the UOT-Pooling with two different architectures, based on the Sinkhorn scaling algorithm and the Bregman ADMM algorithm, respectively, and study their stability and efficiency quantitatively. We test our UOT-Pooling layers in two application scenarios, including multi-instance learning (MIL) and graph embedding. For state-of-the-art models of these two tasks, we can improve their performance by replacing conventional pooling layers with our UOT-Pooling layers.

Related papers

• Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion [53.33473557562837]
  Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost. We propose a practical and scalable approach to solve this problem via mixture of experts (MoE) based model fusion. By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives.
  arXiv  Detail & Related papers  (2024-06-14T07:16:18Z)
• Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion [86.6191592951269]
  Merging models fine-tuned from common extensively pretrained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multitask model that performs well across diverse tasks. We propose the CONtinuous relaxation dis (Concrete) subspace learning method to identify a common lowdimensional subspace and utilize its shared information track interference problem without sacrificing performance.
  arXiv  Detail & Related papers  (2023-12-11T07:24:54Z)
• Multi-Objective Optimization for Sparse Deep Multi-Task Learning [0.0]
  We present a Multi-Objective Optimization algorithm using a modified Weighted Chebyshev scalarization for training Deep Neural Networks (DNNs) Our work aims to address the (economical and also ecological) sustainability issue of DNN models, with particular focus on Deep Multi-Task models.
  arXiv  Detail & Related papers  (2023-08-23T16:42:27Z)
• Regularized Optimal Transport Layers for Generalized Global Pooling Operations [25.309212446782684]
  We develop a novel and generalized global pooling framework through the lens of optimal transport. Our framework is interpretable from the perspective of expectation-maximization. Experimental results show that applying our ROTP layers can reduce the difficulty of the design and selection of global pooling.
  arXiv  Detail & Related papers  (2022-12-13T02:46:36Z)
• Entropic Neural Optimal Transport via Diffusion Processes [105.34822201378763]
  We propose a novel neural algorithm for the fundamental problem of computing the entropic optimal transport (EOT) plan between continuous probability distributions. Our algorithm is based on the saddle point reformulation of the dynamic version of EOT which is known as the Schr"odinger Bridge problem. In contrast to the prior methods for large-scale EOT, our algorithm is end-to-end and consists of a single learning step.
  arXiv  Detail & Related papers  (2022-11-02T14:35:13Z)
• Scaling adaptive quantum simulation algorithms via operator pool tiling [0.0]
  We present a technique called operator pool tiling that facilitates the construction of problem-tailored pools for arbitrarily large problem instances. We demonstrate the method here on strongly correlated quantum spin models in one and two dimensions, finding that ADAPT automatically finds a highly effective ansatz for these systems.
  arXiv  Detail & Related papers  (2022-06-28T18:00:06Z)
• Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
  Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers. Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters. We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
  arXiv  Detail & Related papers  (2022-03-30T23:16:07Z)
• AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling [82.08631594071656]
  Pooling layers are essential building blocks of Convolutional Neural Networks (CNNs) We propose an adaptive and exponentially weighted pooling method named adaPool. We demonstrate how adaPool improves the preservation of detail through a range of tasks including image and video classification and object detection.
  arXiv  Detail & Related papers  (2021-11-01T08:50:37Z)
• Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
  We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC. We provide a practical parametrization of the core optimization problem.
  arXiv  Detail & Related papers  (2021-07-08T18:01:02Z)
• Meta-learning of Pooling Layers for Character Recognition [3.708656266586146]
  We propose a meta-learning framework for pooling layers in convolutional neural network-based character recognition. A parameterized pooling layer is proposed in which the kernel shape and pooling operation are trainable using two parameters. We also propose a meta-learning algorithm for the parameterized pooling layer, which allows us to acquire a suitable pooling layer across multiple tasks.
  arXiv  Detail & Related papers  (2021-03-17T09:25:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.