Self-Supervised Learning with a Multi-Task Latent Space Objective

• URL: http://arxiv.org/abs/2602.05845v1
• Date: Thu, 05 Feb 2026 16:33:30 GMT
• Title: Self-Supervised Learning with a Multi-Task Latent Space Objective
• Authors: Pierre-François De Plaen, Abhishek Jha, Luc Van Gool, Tinne Tuytelaars, Marc Proesmans,
• Abstract summary: Self-supervised learning (SSL) methods learn visual representations by aligning different views of the same image.<n>We show that assigning a separate predictor to each view type stabilizes multi-crop training, resulting in significant performance gains.<n>This yields a simple multi-task formulation of asymmetric Siamese SSL that combines global, local, and masked views into a single framework.
• Score: 71.49269645849675
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning (SSL) methods based on Siamese networks learn visual representations by aligning different views of the same image. The multi-crop strategy, which incorporates small local crops to global ones, enhances many SSL frameworks but causes instability in predictor-based architectures such as BYOL, SimSiam, and MoCo v3. We trace this failure to the shared predictor used across all views and demonstrate that assigning a separate predictor to each view type stabilizes multi-crop training, resulting in significant performance gains. Extending this idea, we treat each spatial transformation as a distinct alignment task and add cutout views, where part of the image is masked before encoding. This yields a simple multi-task formulation of asymmetric Siamese SSL that combines global, local, and masked views into a single framework. The approach is stable, generally applicable across backbones, and consistently improves the performance of ResNet and ViT models on ImageNet.

Related papers

• Adaptive Weighted LSSVM for Multi-View Classification [0.5161531917413708]
  AW-LSSVM promotes complementary learning by an iterative global coupling to make each view focus on hard samples of others from previous iterations.<n>Experiments demonstrate AW-LSSVM outperforms existing kernel-based multi-view methods on most datasets.
  arXiv  Detail & Related papers  (2025-12-02T11:14:47Z)
• Locality Alignment Improves Vision-Language Models [55.275235524659905]
  Vision language models (VLMs) have seen growing adoption in recent years, but many still struggle with basic spatial reasoning errors.<n>Our goal is to resolve this with a vision backbone that effectively captures both local and global image semantics.<n>We propose a new efficient post-training stage for ViTs called locality alignment and a novel fine-tuning procedure called MaskEmbed.
  arXiv  Detail & Related papers  (2024-10-14T21:01:01Z)
• Siamese Transformer Networks for Few-shot Image Classification [9.55588609556447]
  Humans exhibit remarkable proficiency in visual classification tasks, accurately recognizing and classifying new images with minimal examples. Existing few-shot image classification methods often emphasize either global features or local features, with few studies considering the integration of both. We propose a novel approach based on the Siamese Transformer Network (STN) Our strategy effectively harnesses the potential of global and local features in few-shot image classification, circumventing the need for complex feature adaptation modules.
  arXiv  Detail & Related papers  (2024-07-16T14:27:23Z)
• GBE-MLZSL: A Group Bi-Enhancement Framework for Multi-Label Zero-Shot Learning [24.075034737719776]
  This paper investigates a challenging problem of zero-shot learning in the multi-label scenario (MLZSL) We propose a novel and effective group bi-enhancement framework for MLZSL, dubbed GBE-MLZSL, to fully make use of such properties and enable a more accurate and robust visual-semantic projection. Experiments on large-scale MLZSL benchmark datasets NUS-WIDE and Open-Images-v4 demonstrate that the proposed GBE-MLZSL outperforms other state-of-the-art methods with large margins.
  arXiv  Detail & Related papers  (2023-09-02T12:07:21Z)
• Global and Local Semantic Completion Learning for Vision-Language Pre-training [34.740507502215536]
  Cross-modal alignment plays a crucial role in vision-language pre-training models. We propose a novel Global and Local Semantic Completion Learning (GLSCL) task to facilitate global-local alignment and local-local alignment simultaneously.
  arXiv  Detail & Related papers  (2023-06-12T13:20:29Z)
• ESPT: A Self-Supervised Episodic Spatial Pretext Task for Improving Few-Shot Learning [16.859375666701]
  We propose to augment the few-shot learning objective with a novel self-supervised Episodic Spatial Pretext Task (ESPT) Our ESPT objective is defined as maximizing the local spatial relationship consistency between the original episode and the transformed one. Our ESPT method achieves new state-of-the-art performance for few-shot image classification on three mainstay benchmark datasets.
  arXiv  Detail & Related papers  (2023-04-26T04:52:08Z)
• Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training [88.80694147730883]
  We investigate a variety of Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) frameworks. In studied conditions, we observe that a mostly unified encoder for vision and language signals outperforms all other variations that separate more parameters. Our approach outperforms vanilla CLIP by 1.6 points in linear probing on a collection of 24 downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-26T05:19:16Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• Learning Self-Supervised Low-Rank Network for Single-Stage Weakly and Semi-Supervised Semantic Segmentation [119.009033745244]
  This paper presents a Self-supervised Low-Rank Network ( SLRNet) for single-stage weakly supervised semantic segmentation (WSSS) and semi-supervised semantic segmentation (SSSS) SLRNet uses cross-view self-supervision, that is, it simultaneously predicts several attentive LR representations from different views of an image to learn precise pseudo-labels. Experiments on the Pascal VOC 2012, COCO, and L2ID datasets demonstrate that our SLRNet outperforms both state-of-the-art WSSS and SSSS methods with a variety of different settings.
  arXiv  Detail & Related papers  (2022-03-19T09:19:55Z)
• Flexible Example-based Image Enhancement with Task Adaptive Global Feature Self-Guided Network [162.14579019053804]
  We show that our model outperforms the current state of the art in learning a single enhancement mapping. The model achieves even higher performance on learning multiple mappings simultaneously.
  arXiv  Detail & Related papers  (2020-05-13T22:45:07Z)
• Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition [141.24314054768922]
  We propose a spatial-temporal multi-cue (STMC) network to solve the vision-based sequence learning problem. To validate the effectiveness, we perform experiments on three large-scale CSLR benchmarks.
  arXiv  Detail & Related papers  (2020-02-08T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.