Understanding MLP-Mixer as a Wide and Sparse MLP

• URL: http://arxiv.org/abs/2306.01470v2
• Date: Mon, 6 May 2024 20:03:17 GMT
• Title: Understanding MLP-Mixer as a Wide and Sparse MLP
• Authors: Tomohiro Hayase, Ryo Karakida,
• Abstract summary: Multi-layer perceptron (MLP) is a fundamental component of deep learning. Recent-based architectures, especially the Mixer-Mixer, have achieved significant empirical success. We show that sparseness is a key mechanism underlying the Mixer-Mixers.
• Score: 7.734726150561087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-layer perceptron (MLP) is a fundamental component of deep learning, and recent MLP-based architectures, especially the MLP-Mixer, have achieved significant empirical success. Nevertheless, our understanding of why and how the MLP-Mixer outperforms conventional MLPs remains largely unexplored. In this work, we reveal that sparseness is a key mechanism underlying the MLP-Mixers. First, the Mixers have an effective expression as a wider MLP with Kronecker-product weights, clarifying that the Mixers efficiently embody several sparseness properties explored in deep learning. In the case of linear layers, the effective expression elucidates an implicit sparse regularization caused by the model architecture and a hidden relation to Monarch matrices, which is also known as another form of sparse parameterization. Next, for general cases, we empirically demonstrate quantitative similarities between the Mixer and the unstructured sparse-weight MLPs. Following a guiding principle proposed by Golubeva, Neyshabur and Gur-Ari (2021), which fixes the number of connections and increases the width and sparsity, the Mixers can demonstrate improved performance.

Related papers

• Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking [6.9366619419210656]
  Transformers have established themselves as the leading neural network model in natural language processing. Recent research has explored replacing attention modules with other mechanisms, including those described by MetaFormers. This paper integrates Krotov's hierarchical associative memory with MetaFormers, enabling a comprehensive representation of the Transformer block.
  arXiv  Detail & Related papers  (2024-06-18T02:42:19Z)
• SCHEME: Scalable Channel Mixer for Vision Transformers [52.605868919281086]
  Vision Transformers have achieved impressive performance in many vision tasks. Much less research has been devoted to the channel mixer or feature mixing block (FFN or) We show that the dense connections can be replaced with a diagonal block structure that supports larger expansion ratios.
  arXiv  Detail & Related papers  (2023-12-01T08:22:34Z)
• NTK-approximating MLP Fusion for Efficient Language Model Fine-tuning [40.994306592119266]
  Fine-tuning a pre-trained language model (PLM) emerges as the predominant strategy in many natural language processing applications. Some general approaches (e.g. quantization and distillation) have been widely studied to reduce the compute/memory of PLM fine-tuning. We propose to coin a lightweight PLM through NTK-approximating modules in fusion.
  arXiv  Detail & Related papers  (2023-07-18T03:12:51Z)
• SplitMixer: Fat Trimmed From MLP-like Models [53.12472550578278]
  We present SplitMixer, a simple and lightweight isotropic-like architecture, for visual recognition. It contains two types of interleaving convolutional operations to mix information across locations (spatial mixing) and channels (channel mixing)
  arXiv  Detail & Related papers  (2022-07-21T01:37:07Z)
• Boosting Adversarial Transferability of MLP-Mixer [9.957957463532738]
  We propose an adversarial attack method against the Dense-Mixer called Maxwell's demon Attack (MA) Our method can be easily combined with existing methods and can improve the transferability by up to 38.0% on ResMLP. To the best of our knowledge, we are the first work to study adversarial transferability of Dense-Mixer.
  arXiv  Detail & Related papers  (2022-04-26T10:18:59Z)
• Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs [84.3235981545673]
  Token-mixing multi-layer perceptron (MLP) models have shown competitive performance in computer vision tasks. We present Mix-Shift-MLP which makes the size of the local receptive field used for mixing increase with respect to the amount of spatial shifting. MS-MLP achieves competitive performance in multiple vision benchmarks.
  arXiv  Detail & Related papers  (2022-02-14T06:53:48Z)
• PointMixer: MLP-Mixer for Point Cloud Understanding [74.694733918351]
  The concept of channel-mixings and token-mixings achieves noticeable performance in visual recognition tasks. Unlike images, point clouds are inherently sparse, unordered and irregular, which limits the direct use of universal-Mixer for point cloud understanding. We propose PointMixer, a universal point set operator that facilitates information sharing among unstructured 3D points.
  arXiv  Detail & Related papers  (2021-11-22T13:25:54Z)
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone [34.47616917228978]
  We propose an improved structure as termed Circulant Channel-Specific (CCS) token-mixing benchmark, which is spatial-invariant and channel-specific. It takes fewer parameters but achieves higher classification accuracy on ImageNet1K.
  arXiv  Detail & Related papers  (2021-06-28T17:59:57Z)
• MLP-Mixer: An all-MLP Architecture for Vision [93.16118698071993]
  We present-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). Mixer attains competitive scores on image classification benchmarks, with pre-training and inference comparable to state-of-the-art models.
  arXiv  Detail & Related papers  (2021-05-04T16:17:21Z)
• Modal Regression based Structured Low-rank Matrix Recovery for Multi-view Learning [70.57193072829288]
  Low-rank Multi-view Subspace Learning has shown great potential in cross-view classification in recent years. Existing LMvSL based methods are incapable of well handling view discrepancy and discriminancy simultaneously. We propose Structured Low-rank Matrix Recovery (SLMR), a unique method of effectively removing view discrepancy and improving discriminancy.
  arXiv  Detail & Related papers  (2020-03-22T03:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.