SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning

• URL: http://arxiv.org/abs/2609.12599v1
• Date: Fri, 11 Sep 2026 08:50:17 GMT
• Title: SIMS: Scale-Invariant Merit-Function-Based Scalarization for Multi-Task Learning
• Abstract summary: We propose Scale-Invariant Merit-function-based Scalarization (SIMS) for multi-task learning.<n>We show that SIMS consistently outperforms existing scalarization methods and achieves state-of-the-art performance.
• Score: 34.878768912505485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-task learning (MTL) requires navigating unavoidable trade-offs among competing objectives. This paradigm is frequently formulated as multi-objective optimization (MOO), where the scalarization is favored to reduce an MOO problem to a single objective. We empirically find that existing merit-function-based scalarization approaches are sensitive to the relative scales of different objectives in practical MTL, where task losses commonly differ by orders of magnitude. The optimization process often favors objectives with larger scales even though the underlying Pareto optimal solutions remains invariant to rescaling (i.e., multiplying an objective by a positive constant). To address this issue, we propose Scale-Invariant Merit-function-based Scalarization (SIMS) for MTL. Specifically, SIMS adopts a transformation-induced merit function to convert the MOO problem of MTL to a single objective that renders optimization invariant to the magnitudes of losses. Theoretically, we prove that the requirement for scale invariance uniquely determines this transformation to be logarithmic. We further show that this general transformation-induced merit function preserves weak Pareto optimality and admits a smooth surrogate with controllable approximation error. Extensive experiments on representative multi-task benchmarks demonstrate that SIMS consistently outperforms existing scalarization methods and achieves state-of-the-art performance.

Related papers

• Variational Entropic Optimal Transport [67.76725267984578]
  We propose Variational Entropic Optimal Transport (VarEOT) for domain translation problems.<n>VarEOT is based on an exact variational reformulation of the log-partition $log mathbbE[exp(cdot)$ as a tractable generalization over an auxiliary positive normalizer.<n> Experiments on synthetic data and unpaired image-to-image translation demonstrate competitive or improved translation quality.
  arXiv  Detail & Related papers  (2026-02-02T15:48:44Z)
• Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models [47.54958360970588]
  Large language models require significant computational resources for deployment.<n>Main obstacle to effective quantization lies in systematic outliers in activations and weights.<n>We propose an adaptive transformation selection framework that systematically determines optimal transformations on a per-layer basis.
  arXiv  Detail & Related papers  (2025-11-21T22:01:58Z)
• A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning [74.80956524812714]
  We tackle the general differentiable meta learning problem that is ubiquitous in modern deep learning. These problems are often formalized as Bi-Level optimizations (BLO) We introduce a novel perspective by turning a given BLO problem into a ii optimization, where the inner loss function becomes a smooth distribution, and the outer loss becomes an expected loss over the inner distribution.
  arXiv  Detail & Related papers  (2024-10-14T12:10:06Z)
• Zeroth-Order Fine-Tuning of LLMs in Random Subspaces [63.10833446782114]
  As language models grow in size, memory demands for backpropagation increase.<n>Zeroth-order (ZO) optimization methods offer a memory-efficient alternative.<n>In this paper, we propose Subspace Zero-order optimization to address the challenges posed by posed by high dimensionality perturbations.
  arXiv  Detail & Related papers  (2024-10-11T17:01:43Z)
• Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
  Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance. We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features. In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
  arXiv  Detail & Related papers  (2024-06-12T08:30:16Z)
• Dual-Balancing for Multi-Task Learning [42.613360970194734]
  We propose a Dual-Balancing Multi-Task Learning (DB-MTL) method to alleviate the task balancing problem from both loss and gradient perspectives. DB-MTL ensures loss-scale balancing by performing a logarithm transformation on each task loss, and guarantees gradient-magnitude balancing via normalizing all task gradients to the same magnitude as the maximum gradient norm.
  arXiv  Detail & Related papers  (2023-08-23T09:41:28Z)
• Independent Component Alignment for Multi-Task Learning [2.5234156040689237]
  In a multi-task learning (MTL) setting, a single model is trained to tackle a diverse set of tasks jointly. We propose using a condition number of a linear system of gradients as a stability criterion of an MTL optimization. We present Aligned-MTL, a novel MTL optimization approach based on the proposed criterion.
  arXiv  Detail & Related papers  (2023-05-30T12:56:36Z)
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients [64.21628447579772]
  Implicit Bayesian meta-learning (iBaML) method broadens the scope of learnable priors, but also quantifies the associated uncertainty. Analytical error bounds are established to demonstrate the precision and efficiency of the generalized implicit gradient over the explicit one.
  arXiv  Detail & Related papers  (2023-03-31T02:10:30Z)
• SLAW: Scaled Loss Approximate Weighting for Efficient Multi-Task Learning [0.0]
  Multi-task learning (MTL) is a subfield of machine learning with important applications. The best MTL optimization methods require individually computing the gradient of each task's loss function. We propose Scaled Loss Approximate Weighting (SLAW), a method for multi-task optimization that matches the performance of the best existing methods while being much more efficient.
  arXiv  Detail & Related papers  (2021-09-16T20:58:40Z)
• Learning Invariant Representations using Inverse Contrastive Loss [34.93395633215398]
  We introduce a class of losses for learning representations that are invariant to some extraneous variable of interest. We show that if the extraneous variable is binary, then optimizing ICL is equivalent to optimizing a regularized MMD divergence.
  arXiv  Detail & Related papers  (2021-02-16T18:29:28Z)
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning [63.64636047748605]
  We develop a new theoretical framework to provide convergence guarantee for the general multi-step MAML algorithm. In particular, our results suggest that an inner-stage step needs to be chosen inversely proportional to $N$ of inner-stage steps in order for $N$ MAML to have guaranteed convergence.
  arXiv  Detail & Related papers  (2020-02-18T19:17:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.