MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing

• URL: http://arxiv.org/abs/2206.06292v1
• Date: Mon, 13 Jun 2022 16:21:33 GMT
• Title: MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing
• Authors: Zhaofan Qiu and Ting Yao and Chong-Wah Ngo and Tao Mei
• Abstract summary: We present a novel-like 3D architecture for video recognition. The results are comparable to state-of-the-art widely-used 3D CNNs and video.
• Score: 123.43419144051703
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Convolutional Neural Networks (CNNs) have been regarded as the go-to models for visual recognition. More recently, convolution-free networks, based on multi-head self-attention (MSA) or multi-layer perceptrons (MLPs), become more and more popular. Nevertheless, it is not trivial when utilizing these newly-minted networks for video recognition due to the large variations and complexities in video data. In this paper, we present MLP-3D networks, a novel MLP-like 3D architecture for video recognition. Specifically, the architecture consists of MLP-3D blocks, where each block contains one MLP applied across tokens (i.e., token-mixing MLP) and one MLP applied independently to each token (i.e., channel MLP). By deriving the novel grouped time mixing (GTM) operations, we equip the basic token-mixing MLP with the ability of temporal modeling. GTM divides the input tokens into several temporal groups and linearly maps the tokens in each group with the shared projection matrix. Furthermore, we devise several variants of GTM with different grouping strategies, and compose each variant in different blocks of MLP-3D network by greedy architecture search. Without the dependence on convolutions or attention mechanisms, our MLP-3D networks achieves 68.5\%/81.4\% top-1 accuracy on Something-Something V2 and Kinetics-400 datasets, respectively. Despite with fewer computations, the results are comparable to state-of-the-art widely-used 3D CNNs and video transformers. Source code is available at https://github.com/ZhaofanQiu/MLP-3D.

Related papers

• X-MLP: A Patch Embedding-Free MLP Architecture for Vision [4.493200639605705]
  Multi-layer perceptron (MLP) architectures for vision have been popular again. We propose X-MLP, an architecture constructed absolutely upon fully connected layers and free from patch embedding. X-MLP is tested on ten benchmark datasets, all better performance than other vision models.
  arXiv  Detail & Related papers  (2023-07-02T15:20:25Z)
• R2-MLP: Round-Roll MLP for Multi-View 3D Object Recognition [33.53114929452528]
  Vision architectures based exclusively on multi-layer perceptrons (MLPs) have gained much attention in the computer vision community. We present an achieves a view-based 3D object recognition task by considering the communications between patches from different views. With a conceptually simple structure, our R$2$MLP achieves competitive performance compared with existing methods.
  arXiv  Detail & Related papers  (2022-11-20T21:13:02Z)
• GraphMLP: A Graph MLP-Like Architecture for 3D Human Pose Estimation [68.65764751482774]
  GraphMLP is a global-local-graphical unified architecture for 3D human pose estimation. It incorporates the graph structure of human bodies into a model to meet the domain-specific demand of the 3D human pose. It can be extended to model complex temporal dynamics in a simple way with negligible computational cost gains in the sequence length.
  arXiv  Detail & Related papers  (2022-06-13T18:59:31Z)
• RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality [113.1414517605892]
  We propose a methodology, Locality Injection, to incorporate local priors into an FC layer. RepMLPNet is the first that seamlessly transfer to Cityscapes semantic segmentation.
  arXiv  Detail & Related papers  (2021-12-21T10:28:17Z)
• An Image Patch is a Wave: Phase-Aware Vision MLP [54.104040163690364]
  multilayer perceptron (MLP) is a new kind of vision model with extremely simple architecture that only stacked by fully-connected layers. We propose to represent each token as a wave function with two parts, amplitude and phase. Experiments demonstrate that the proposed Wave-MLP is superior to the state-of-the-art architectures on various vision tasks.
  arXiv  Detail & Related papers  (2021-11-24T06:25:49Z)
• Sparse MLP for Image Recognition: Is Self-Attention Really Necessary? [65.37917850059017]
  We build an attention-free network called sMLPNet. For 2D image tokens, sMLP applies 1D along the axial directions and the parameters are shared among rows or columns. When scaling up to 66M parameters, sMLPNet achieves 83.4% top-1 accuracy, which is on par with the state-of-the-art Swin Transformer.
  arXiv  Detail & Related papers  (2021-09-12T04:05:15Z)
• CycleMLP: A MLP-like Architecture for Dense Prediction [26.74203747156439]
  CycleMLP is a versatile backbone for visual recognition and dense predictions. It can cope with various image sizes and achieves linear computational complexity to image size by using local windows. CycleMLP aims to provide a competitive baseline on object detection, instance segmentation, and semantic segmentation for models.
  arXiv  Detail & Related papers  (2021-07-21T17:23:06Z)
• S$^2$-MLP: Spatial-Shift MLP Architecture for Vision [34.47616917228978]
  Recently, visual Transformer (ViT) and its following works abandon the convolution and exploit the self-attention operation. In this paper, we propose a novel pure architecture, spatial-shift (S$2$-MLP)
  arXiv  Detail & Related papers  (2021-06-14T15:05:11Z)
• MLP-Mixer: An all-MLP Architecture for Vision [93.16118698071993]
  We present-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). Mixer attains competitive scores on image classification benchmarks, with pre-training and inference comparable to state-of-the-art models.
  arXiv  Detail & Related papers  (2021-05-04T16:17:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.