Meta Curvature-Aware Minimization for Domain Generalization

• URL: http://arxiv.org/abs/2412.11542v1
• Date: Mon, 16 Dec 2024 08:22:23 GMT
• Title: Meta Curvature-Aware Minimization for Domain Generalization
• Authors: Ziyang Chen, Yiwen Ye, Feilong Tang, Yongsheng Pan, Yong Xia,
• Abstract summary: We propose an improved model training process aimed at encouraging the model to converge to a flat minima. We derive a novel algorithm called Meta Curvature-Aware Minimization (MeCAM) to minimize the curvature around the local minima. We provide theoretical analysis on MeCAM's generalization error and convergence rate, and demonstrate its superiority over existing DG methods.
• Score: 22.824033201965648
• License:
• Abstract: Domain generalization (DG) aims to enhance the ability of models trained on source domains to generalize effectively to unseen domains. Recently, Sharpness-Aware Minimization (SAM) has shown promise in this area by reducing the sharpness of the loss landscape to obtain more generalized models. However, SAM and its variants sometimes fail to guide the model toward a flat minimum, and their training processes exhibit limitations, hindering further improvements in model generalization. In this paper, we first propose an improved model training process aimed at encouraging the model to converge to a flat minima. To achieve this, we design a curvature metric that has a minimal effect when the model is far from convergence but becomes increasingly influential in indicating the curvature of the minima as the model approaches a local minimum. Then we derive a novel algorithm from this metric, called Meta Curvature-Aware Minimization (MeCAM), to minimize the curvature around the local minima. Specifically, the optimization objective of MeCAM simultaneously minimizes the regular training loss, the surrogate gap of SAM, and the surrogate gap of meta-learning. We provide theoretical analysis on MeCAM's generalization error and convergence rate, and demonstrate its superiority over existing DG methods through extensive experiments on five benchmark DG datasets, including PACS, VLCS, OfficeHome, TerraIncognita, and DomainNet. Code will be available on GitHub.

Related papers

• LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging [10.33844295243509]
  We propose a unified framework for model merging based on low-rank estimation of task vectors without the need for access to the base model, named textscLoRE-Merging. Our approach is motivated by the observation that task vectors from fine-tuned models frequently exhibit a limited number of dominant singular values, making low-rank estimations less prone to interference.
  arXiv  Detail & Related papers  (2025-02-15T10:18:46Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG. Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization. Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)
• QT-DoG: Quantization-aware Training for Domain Generalization [58.439816306817306]
  We propose Quantization-aware Training for Domain Generalization (QT-DoG) QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights. We demonstrate that QT-DoG generalizes across various datasets, architectures, and quantization algorithms.
  arXiv  Detail & Related papers  (2024-10-08T13:21:48Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Agnostic Sharpness-Aware Minimization [29.641227264358704]
  Sharpness-aware (SAM) has been instrumental in improving deep neural network training by minimizing both the training loss and the sharpness of the loss landscape. Model-Agnostic Meta-Learning (MAML) is a framework designed to improve the adaptability of models. We introduce Agnostic-SAM, a novel approach that combines the principles of both SAM and MAML.
  arXiv  Detail & Related papers  (2024-06-11T09:49:00Z)
• Sharpness-Aware Gradient Matching for Domain Generalization [84.14789746460197]
  The goal of domain generalization (DG) is to enhance the generalization capability of the model learned from a source domain to other unseen domains. The recently developed Sharpness-Aware Minimization (SAM) method aims to achieve this goal by minimizing the sharpness measure of the loss landscape. We present two conditions to ensure that the model could converge to a flat minimum with a small loss, and present an algorithm, named Sharpness-Aware Gradient Matching (SAGM) Our proposed SAGM method consistently outperforms the state-of-the-art methods on five DG benchmarks.
  arXiv  Detail & Related papers  (2023-03-18T07:25:12Z)
• When to Update Your Model: Constrained Model-based Reinforcement Learning [50.74369835934703]
  We propose a novel and general theoretical scheme for a non-decreasing performance guarantee of model-based RL (MBRL) Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.
  arXiv  Detail & Related papers  (2022-10-15T17:57:43Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Improving Generalization in Federated Learning by Seeking Flat Minima [23.937135834522145]
  Models trained in federated settings often suffer from degraded performances and fail at generalizing. In this work, we investigate such behavior through the lens of geometry of the loss and Hessian eigenspectrum. Motivated by prior studies connecting the sharpness of the loss surface and the generalization gap, we show that i) training clients locally with Sharpness-Aware Minimization (SAM) or its adaptive version (ASAM) on the server-side can substantially improve generalization.
  arXiv  Detail & Related papers  (2022-03-22T16:01:04Z)
• Sharpness-Aware Minimization for Efficiently Improving Generalization [36.87818971067698]
  We introduce a novel, effective procedure for simultaneously minimizing loss value and loss sharpness. Sharpness-Aware Minimization (SAM) seeks parameters that lie in neighborhoods having uniformly low loss. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-03T19:02:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.