Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning

• URL: http://arxiv.org/abs/2604.07848v1
• Date: Thu, 09 Apr 2026 06:02:26 GMT
• Title: Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning
• Abstract summary: Multi-task learning shows strikingly inconsistent results.<n>Standard benchmarks systematically violate this requirement.<n>This provides the first principled explanation for seven years of inconsistent MTL results.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-task learning shows strikingly inconsistent results -- sometimes joint training helps substantially, sometimes it actively harms performance -- yet the field lacks a principled framework for predicting these outcomes. We identify a fundamental but unstated assumption underlying gradient-based task analysis: tasks must share training instances for gradient conflicts to reveal genuine relationships. When tasks are measured on the same inputs, gradient alignment reflects shared mechanistic structure; when measured on disjoint inputs, any apparent signal conflates task relationships with distributional shift. We discover this sample overlap requirement exhibits a sharp phase transition: below 30% overlap, gradient-task correlations are statistically indistinguishable from noise; above 40%, they reliably recover known biological structure. Comprehensive validation across multiple datasets achieves strong correlations and recovers biological pathway organization. Standard benchmarks systematically violate this requirement -- MoleculeNet operates at <5% overlap, TDC at 8-14% -- far below the threshold where gradient analysis becomes meaningful. This provides the first principled explanation for seven years of inconsistent MTL results.

Related papers

• Pitfalls and Remedies for Multi-Task Bayesian Optimization [10.431916242473045]
  Bayesian optimization routinely warms a target experiment with data from related source tasks.<n>We revisit this default in a controlled setting and find that it misestimates the cross-task correlation.<n>We trace the failure to two independent structural mechanisms.
  arXiv  Detail & Related papers  (2026-07-10T03:38:36Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training [76.12556589212666]
  We show that curriculum post-training avoids the exponential complexity bottleneck.<n>Under outcome-only reward signals, reinforcement learning finetuning achieves high accuracy with sample complexity.<n>We establish guarantees for test-time scaling, where curriculum-aware querying reduces both reward oracle calls and sampling cost from exponential to order.
  arXiv  Detail & Related papers  (2025-11-10T18:29:54Z)
• Exploiting Unlabeled Structures through Task Consistency Training for Versatile Medical Image Segmentation [24.25178585285867]
  We introduce a Task Consistency Training (TCT) framework to address class imbalance without requiring extra models.<n>To avoid error propagation from low-consistency, potentially noisy data, we propose a filtering strategy to exclude such data.<n>Experiments on eight abdominal datasets from diverse clinical sites demonstrate our approach's effectiveness.
  arXiv  Detail & Related papers  (2025-09-05T01:04:32Z)
• On Task Vectors and Gradients [24.021393654093103]
  We provide a rigorous theoretical foundation for task arithmetic by establishing a connection between task vectors and gradients of the task losses.<n>We show that under standard gradient descent, a task vector generated from one epoch of finetuning is exactly equivalent to the negative gradient of the loss, scaled by the learning rate.<n>Our empirical analysis across seven vision benchmarks corroborates our theory, demonstrating that the first-epoch gradient dominates the finetuning trajectory in both norm and direction.
  arXiv  Detail & Related papers  (2025-08-22T04:16:42Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity [71.11795737362459]
  ViTs with self-attention modules have recently achieved great empirical success in many tasks. However, theoretical learning generalization analysis is mostly noisy and elusive. This paper provides the first theoretical analysis of a shallow ViT for a classification task.
  arXiv  Detail & Related papers  (2023-02-12T22:12:35Z)
• A Theoretical Analysis of Fine-tuning with Linear Teachers [31.849269592822296]
  Fine-tuning is a common practice in deep learning, achieving excellent results on downstream tasks using relatively little training data. We show that the success of fine-tuning depends on the similarity between the source tasks and the target task, however measuring it is non trivial.
  arXiv  Detail & Related papers  (2021-07-04T14:15:50Z)
• Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers [66.66228496844191]
  We show when does combining the samples from two related tasks perform better than learning with one target task alone?<n>This question is motivated by an empirical phenomenon known as negative transfer, which has been observed in practice.<n>We illustrate these results in a random-effects model to mathematically prove a phase transition from positive to negative transfer as the number of source task samples increases.
  arXiv  Detail & Related papers  (2020-10-22T14:14:20Z)
• Robust Learning Through Cross-Task Consistency [92.42534246652062]
  We propose a broadly applicable and fully computational method for augmenting learning with Cross-Task Consistency. We observe that learning with cross-task consistency leads to more accurate predictions and better generalization to out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-06-07T09:24:33Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.