StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets

• URL: http://arxiv.org/abs/2506.08013v1
• Date: Mon, 09 Jun 2025 17:59:59 GMT
• Title: StableMTL: Repurposing Latent Diffusion Models for Multi-Task Learning from Partially Annotated Synthetic Datasets
• Authors: Anh-Quan Cao, Ivan Lopes, Raoul de Charette,
• Abstract summary: We extend the partial learning setup to a zero-shot setting, training a multi-task model on multiple datasets, each labeled for only a subset of tasks.<n>Our method, StableMTL, repurposes image generators for latent regression.<n>Instead of per-task losses requiring careful balancing, a unified latent loss is adopted, enabling seamless scaling to more tasks.
• Score: 14.867396697566257
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task learning for dense prediction is limited by the need for extensive annotation for every task, though recent works have explored training with partial task labels. Leveraging the generalization power of diffusion models, we extend the partial learning setup to a zero-shot setting, training a multi-task model on multiple synthetic datasets, each labeled for only a subset of tasks. Our method, StableMTL, repurposes image generators for latent regression. Adapting a denoising framework with task encoding, per-task conditioning and a tailored training scheme. Instead of per-task losses requiring careful balancing, a unified latent loss is adopted, enabling seamless scaling to more tasks. To encourage inter-task synergy, we introduce a multi-stream model with a task-attention mechanism that converts N-to-N task interactions into efficient 1-to-N attention, promoting effective cross-task sharing. StableMTL outperforms baselines on 7 tasks across 8 benchmarks.

Related papers

• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners [60.75160178669076]
  We show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online reinforcement learning.<n>We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL.
  arXiv  Detail & Related papers  (2025-05-29T06:41:45Z)
• Optimizing Dense Visual Predictions Through Multi-Task Coherence and Prioritization [7.776434991976473]
  Multi-Task Learning (MTL) involves the concurrent training of multiple tasks.<n>We propose an advanced MTL model specifically designed for dense vision tasks.
  arXiv  Detail & Related papers  (2024-12-04T10:05:47Z)
• Task-Distributionally Robust Data-Free Meta-Learning [99.56612787882334]
  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift ( TDS) and Task-Distribution Corruption (TDC)
  arXiv  Detail & Related papers  (2023-11-23T15:46:54Z)
• AdaMerging: Adaptive Model Merging for Multi-Task Learning [68.75885518081357]
  This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging) It aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data. Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11% improvement in performance.
  arXiv  Detail & Related papers  (2023-10-04T04:26:33Z)
• Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives [19.90788777476128]
  Multi-task learning (MTL) seeks to learn a single model to accomplish multiple tasks by leveraging shared information among the tasks. Existing MTL models have been known to suffer from negative interference among tasks. We propose ETR-NLP to mitigate task interference through a synergistic combination of non-learnable primitives and explicit task routing.
  arXiv  Detail & Related papers  (2023-08-03T22:34:16Z)
• Identification of Negative Transfers in Multitask Learning Using Surrogate Models [29.882265735630046]
  Multitask learning is widely used to train a low-resource target task by augmenting it with multiple related source tasks. A critical problem in multitask learning is identifying subsets of source tasks that would benefit the target task. We introduce an efficient procedure to address this problem via surrogate modeling.
  arXiv  Detail & Related papers  (2023-03-25T23:16:11Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
  Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers. Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters. We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
  arXiv  Detail & Related papers  (2022-03-30T23:16:07Z)
• Task Uncertainty Loss Reduce Negative Transfer in Asymmetric Multi-task Feature Learning [0.0]
  Multi-task learning (MTL) can improve task performance overall relative to single-task learning (STL), but can hide negative transfer (NT) Asymmetric multitask feature learning (AMTFL) is an approach that tries to address this by allowing tasks with higher loss values to have smaller influence on feature representations for learning other tasks. We present examples of NT in two datasets (image recognition and pharmacogenomics) and tackle this challenge by using aleatoric homoscedastic uncertainty to capture the relative confidence between tasks, and set weights for task loss.
  arXiv  Detail & Related papers  (2020-12-17T13:30:45Z)
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks [4.095907708855597]
  Multi-task learning (MTL) can improve performance on a task by sharing representations with one or more related auxiliary-tasks. Usually, MTL-networks are trained on a composite loss function formed by a constant weighted combination of the separate task losses. In practice, constant loss weights lead to poor results for two reasons: (i) for mini-batch based optimisation, the optimal task weights vary significantly from one update to the next depending on mini-batch sample composition. We introduce HydaLearn, an intelligent weighting algorithm that connects main-task gain to the individual task gradients, in order to inform
  arXiv  Detail & Related papers  (2020-08-26T16:04:02Z)
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference [75.95287293847697]
  Two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning) Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference)
  arXiv  Detail & Related papers  (2020-07-24T14:44:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.