Self-Supervised Learning based on Heat Equation

• URL: http://arxiv.org/abs/2211.13228v1
• Date: Wed, 23 Nov 2022 18:59:59 GMT
• Title: Self-Supervised Learning based on Heat Equation
• Abstract summary: This paper presents a new perspective of self-supervised learning based on extending heat equation into high dimensional feature space. We simplify it by splitting x and y axes as two first-order linear differential equations. This introduces a very simple masked image modeling (MIM) method, named QB-Heat.
• Score: 75.61974935666446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a new perspective of self-supervised learning based on extending heat equation into high dimensional feature space. In particular, we remove time dependence by steady-state condition, and extend the remaining 2D Laplacian from x--y isotropic to linear correlated. Furthermore, we simplify it by splitting x and y axes as two first-order linear differential equations. Such simplification explicitly models the spatial invariance along horizontal and vertical directions separately, supporting prediction across image blocks. This introduces a very simple masked image modeling (MIM) method, named QB-Heat. QB-Heat leaves a single block with size of quarter image unmasked and extrapolates other three masked quarters linearly. It brings MIM to CNNs without bells and whistles, and even works well for pre-training light-weight networks that are suitable for both image classification and object detection without fine-tuning. Compared with MoCo-v2 on pre-training a Mobile-Former with 5.8M parameters and 285M FLOPs, QB-Heat is on par in linear probing on ImageNet, but clearly outperforms in non-linear probing that adds a transformer block before linear classifier (65.6% vs. 52.9%). When transferring to object detection with frozen backbone, QB-Heat outperforms MoCo-v2 and supervised pre-training on ImageNet by 7.9 and 4.5 AP respectively. This work provides an insightful hypothesis on the invariance within visual representation over different shapes and textures: the linear relationship between horizontal and vertical derivatives. The code will be publicly released.

Related papers

• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance [104.05734286732941]
  FINOLA represents each image in the latent space as a first-order autoregressive process. We demonstrate the ability of FINOLA to auto-regress up to a 256x256 feature map. We also leverage FINOLA for self-supervised learning by employing a simple masked prediction approach.
  arXiv  Detail & Related papers  (2023-05-25T17:59:50Z)
• Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost [53.746169882193456]
  Recent works have proposed various sparse attention modules to overcome the quadratic cost of self-attention. We propose a model that resolves both problems by endowing each attention head with a mixed-membership Block Model. Our model outperforms previous efficient variants as well as the original Transformer with full attention.
  arXiv  Detail & Related papers  (2022-10-27T15:30:52Z)
• Large-Margin Representation Learning for Texture Classification [67.94823375350433]
  This paper presents a novel approach combining convolutional layers (CLs) and large-margin metric learning for training supervised models on small datasets for texture classification. The experimental results on texture and histopathologic image datasets have shown that the proposed approach achieves competitive accuracy with lower computational cost and faster convergence when compared to equivalent CNNs.
  arXiv  Detail & Related papers  (2022-06-17T04:07:45Z)
• Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction [12.86325214182021]
  Dynamic model pruning is a recent direction that allows for the inference of a different sub-network for each input sample during deployment. Current dynamic methods rely on learning a continuous channel gating through regularization by inducing sparsity loss. We show experiments on several neural architectures, such as VGG, ResNet, and MobileNet on CIFAR and ImageNet.
  arXiv  Detail & Related papers  (2021-10-15T17:39:53Z)
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation [64.92152574895111]
  We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T15:01:46Z)
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction [31.077182488826963]
  Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks. This causes the representation vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.
  arXiv  Detail & Related papers  (2021-03-04T18:55:09Z)
• I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image [79.040930290399]
  We propose I2L-MeshNet, an image-to-lixel (line+pixel) prediction network. The proposed I2L-MeshNet predicts the per-lixel likelihood on 1D heatmaps for each mesh coordinate instead of directly regressing the parameters. Our lixel-based 1D heatmap preserves the spatial relationship in the input image and models the prediction uncertainty.
  arXiv  Detail & Related papers  (2020-08-09T12:13:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.