TRACER: Persistent Regularization for Robust Multimodal Finetuning

• URL: http://arxiv.org/abs/2605.29380v1
• Date: Thu, 28 May 2026 05:34:23 GMT
• Title: TRACER: Persistent Regularization for Robust Multimodal Finetuning
• Abstract summary: We develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy.<n>We prove that a weighted moving average teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace.<n>Experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures.
• Score: 15.846967169292789
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mainstream strategies for finetuning pretrained multimodal models often degrade out-of-distribution (OOD) robustness, a phenomenon known as catastrophic forgetting. In this paper, we develop a theoretical framework for multimodal contrastive finetuning, yielding closed-form solutions and a geometric decomposition for each strategy. This framework shows that self-distillation is more effective than other regularization approaches to retain the knowledge of the pretrained model. Our analysis reveals a largely overlooked limitation: standard Exponential Moving Average (EMA) teachers, widely used in robust finetuning, suffer from collapse. To solve this, we prove that a Weighted Moving Average (WMA) teacher maintains a persistent regularizing force over finite horizons and yields bias-free convergence in the task subspace while preserving orthogonal knowledge. These insights motivate **TRACER** (**T**rajectory-**R**obust **A**nchoring for **C**ontrastive **E**ncoder **R**egularization), which combines contrastive learning with WMA-guided multi-perspective distillation. Extensive experiments on CLIP finetuning demonstrate consistent OOD accuracy and calibration gains across three backbone architectures, and comprehensive ablations confirm that TRACER is both principled and robust to hyperparameter choices. Code is available at [https://github.com/HesamAsad/TRACER](https://github.com/HesamAsad/TRACER).

Related papers

• Beyond Modality Harmony: Orthogonal Purification and Topology-Guided MoE for Conflict-Aware Multimodal Recommendation [6.78317230214304]
  We propose Orthogonal purification and topology-guided MoE for conflict-aware multimodal Recommendation (OrthoRec)<n>At its core, OrthoRec introduces Collaborative-Guided Orthogonal Purification (CGOP)<n>We show that OrthoRec consistently outperforms competitive recent baselines and exhibits improved robustness under modality noise and item sparsity.
  arXiv  Detail & Related papers  (2026-09-02T06:06:30Z)
• FlowErase-OPD: Multi-Concept Erasure via Anchored On-Policy Distillation in Flow Matching Models [11.142782377709409]
  FlowErase-OPD is a framework for multi-concept erasure based on on-policy distillation (OPD)<n>Our approach first distills multiple single-concept erased models into a unified LoRA module and introduces Anchored Multi-Teacher Distillation (AMTD)<n>Experiments on nudity, object, and artistic-style erasure demonstrate that FlowErase-OPD consistently improves the trade-off between erasure effectiveness, image quality, and semantic alignment.
  arXiv  Detail & Related papers  (2026-08-07T08:58:13Z)
• Multicalibration Boosting: Theory, Convergence, and Transferability [1.2031796234206138]
  Multicalibration extends classical calibration by requiring predictions to be unbiased over a rich collection of functions.<n>It has emerged as a powerful framework for fairness, robustness, and reliable prediction.<n>We develop a unified and refined perspective on MCBoost that subsumes existing variants.
  arXiv  Detail & Related papers  (2026-05-23T03:04:49Z)
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• Beyond Surface Artifacts: Capturing Shared Latent Forgery Knowledge Across Modalities [52.189426539304065]
  This paper introduces a paradigm shift that redefines multimodal forensics from conventional "feature fusion" to "modality generalization"<n>We propose the first modality-agnostic forgery (MAF) detection framework. By explicitly decoupling modality-specific styles, MAF precisely extracts the essential, cross-modal latent forgery knowledge.
  arXiv  Detail & Related papers  (2026-04-09T03:35:21Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiency [0.0]
  We introduce REDistill, a principled framework grounded in robust statistics.<n>Redistill replaces the standard KD objective with a power divergence loss, a generalization of KL divergence.<n>Experiments on CIFAR-100 and ImageNet-1k demonstrate that REDistill consistently improves student accuracy in diverse teacher-student architectures.
  arXiv  Detail & Related papers  (2026-02-04T15:50:53Z)
• Model Merging via Multi-Teacher Knowledge Distillation [11.543771846135021]
  We introduce a novel flatness-aware PAC-Bayes generalization bound specifically for the model merging setting.<n>We frame model merging as multi-teacher knowledge distillation on scarce, unlabeled data.<n>We formally demonstrate that minimizing the student-teacher Kullback-Leibler divergence directly tightens the upper bound on the merged model's excess risk.
  arXiv  Detail & Related papers  (2025-12-24T17:10:44Z)
• Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalization [72.83292830785336]
  Weight Averaging (WA) has emerged as a powerful technique for enhancing generalization by promoting convergence to a flat loss landscape.<n>We propose MBCD, a unified collaborative distillation framework that retains WA's flatness-inducing advantages while overcoming its shortcomings in multi-modal contexts.
  arXiv  Detail & Related papers  (2025-11-25T12:38:28Z)
• Calibrated Multimodal Representation Learning with Missing Modalities [100.55774771852468]
  Multimodal representation learning harmonizes distinct modalities by aligning them into a unified latent space.<n>Recent research generalizes traditional cross-modal alignment to produce enhanced multimodal synergy but requires all modalities to be present for a common instance.<n>We provide theoretical insights into this issue from an anchor shift perspective.<n>We propose CalMRL for multimodal representation learning to calibrate incomplete alignments caused by missing modalities.
  arXiv  Detail & Related papers  (2025-11-15T05:01:43Z)
• Principled Multimodal Representation Learning [99.53621521696051]
  Multimodal representation learning seeks to create a unified representation space by integrating diverse data modalities.<n>Recent advances have investigated the simultaneous alignment of multiple modalities, yet several challenges remain.<n>We propose Principled Multimodal Representation Learning (PMRL), a novel framework that achieves simultaneous alignment of multiple modalities.
  arXiv  Detail & Related papers  (2025-07-23T09:12:25Z)
• A General Adaptive Dual-level Weighting Mechanism for Remote Sensing Pansharpening [11.791358860917189]
  deep learning methods for remote sensing pansharpening have advanced rapidly.<n>Many existing methods struggle to fully leverage feature heterogeneity and redundancy.<n>We introduce a general adaptive dual-level weighting mechanism (ADWM) to address these challenges.
  arXiv  Detail & Related papers  (2025-03-17T14:24:00Z)
• LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning [103.45785408116146]
  Continual learning (CL) aims to train a model that can solve multiple tasks presented sequentially.<n>Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks.<n>However, such methods lack theoretical guarantees, making them prone to unexpected failures.<n>We aim to bridge this gap by designing a simple CL method that is theoretically sound and highly performant.
  arXiv  Detail & Related papers  (2024-10-01T12:58:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.