Related papers: DualOptim: Enhancing Efficacy and Stability in Machine Unlearning with Dual Optimizers

Related papers

DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models [28.795194964735412]
We propose DualOptim+, a novel optimization framework for improving machine unlearning in large language models.<n>It introduces a base state to capture common representations shared by forgetting and retaining objectives and delta states to preserve objective-specific residuals.<n>We further introduce DualOptim+ 8bit, a quantized variant that reduces memory overhead without compromising performance.
arXiv Detail & Related papers (2026-05-20T07:45:08Z)
Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipe [53.40076304466524]
We propose Uni-OPD, a unified OPD framework that generalizes across Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs)<n>Specifically, from the student's perspective, we adopt two data balancing strategies to promote exploration of informative student-generated states during training.<n>We develop an outcome-guided margin calibration mechanism to restore order consistency between correct and incorrect trajectories.
arXiv Detail & Related papers (2026-05-05T12:15:21Z)
Delve into the Applicability of Advanced Optimizers for Multi-Task Learning [70.80216936064654]
Multi-Task Learning (MTL) is a machine learning problem that has seen extensive development over the past decade.<n>We propose APT (Applicability of advanced oPizers), a framework featuring a simple adaptive momentum mechanism designed to balance the strengths between advanceds and MTL.
arXiv Detail & Related papers (2026-04-10T04:15:31Z)
Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidence [13.56541419560425]
We show that a second-order quasi-Newton, textSOAP, dramatically improves both training efficiency and final performance across diverse LICs.<n>We uncover a critical deployability benefit: second-order trained models exhibit significantly fewer activation and latent robustnesss.
arXiv Detail & Related papers (2026-01-28T16:56:52Z)
Compression then Matching: An Efficient Pre-training Paradigm for Multimodal Embedding [53.18433310890516]
Vision-language models advance multimodal representation learning by acquiring transferable semantic embeddings.<n>We propose CoMa, a compressed pre-training phase, which serves as a warm-up stage for contrastive learning.
arXiv Detail & Related papers (2025-11-11T17:23:02Z)
AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models [35.71783914954563]
We propose a novel framework that integrates multi-modal feature fusion, multi-teacher distillation, and adaptive optimization to deliver lightweight yet effective retrieval models.<n>Experiments on three benchmark datasets demonstrate that AMMKD achieves superior performance while significantly reducing model complexity, validating its effectiveness and flexibility.
arXiv Detail & Related papers (2025-08-23T04:52:20Z)
The Power of the Pareto Front: Balancing Uncertain Rewards for Adaptive Experimentation in scanning probe microscopy [3.828556515394516]
We show that MOBO can optimize scanning probe microscopy (SPM) imaging parameters to enhance measurement quality and efficiency.<n>MOBO offers a natural framework for human-in-the-loop decision-making, enabling researchers to fine-tune experimental trade-offs based on domain expertise.
arXiv Detail & Related papers (2025-04-09T01:59:31Z)
Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning [59.001091197106085]
Multi-Task Learning (MTL) for Vision Transformer aims at enhancing the model capability by tackling multiple tasks simultaneously. Most recent works have predominantly focused on designing Mixture-of-Experts (MoE) structures and in tegrating Low-Rank Adaptation (LoRA) to efficiently perform multi-task learning. We propose a novel approach dubbed Efficient Multi-Task Learning (EMTAL) by transforming a pre-trained Vision Transformer into an efficient multi-task learner.
arXiv Detail & Related papers (2025-01-12T17:41:23Z)
Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning [102.18178065928426]
We propose an efficient fine-tuning framework with two novel approaches: Vision Cue Enhancement (VCE) and Dual Low-Rank Adaptation (Dual-LoRA)<n>VCE enhances the vision projector by integrating multi-level visual cues, improving the model's ability to capture fine-grained visual features.<n> Dual-LoRA introduces a dual low-rank structure for instruction tuning, decoupling learning into skill and task spaces to enable precise control and efficient adaptation across diverse tasks.
arXiv Detail & Related papers (2024-11-19T11:03:09Z)
Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation [43.524099888917384]
IISAN was limited to symmetrical MFMs and identical text and image encoders, preventing the use of state-of-the-art Large Language Models. We developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs. IISAN-Versa effectively adapts large text encoders, and we further identify a scaling effect where larger encoders generally perform better.
arXiv Detail & Related papers (2024-11-05T10:53:25Z)
Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate [105.86576388991713]
We introduce a normalized gradient difference (NGDiff) algorithm, enabling us to have better control over the trade-off between the objectives. We provide a theoretical analysis and empirically demonstrate the superior performance of NGDiff among state-of-the-art unlearning methods on the TOFU and MUSE datasets.
arXiv Detail & Related papers (2024-10-29T14:41:44Z)
On-the-fly Modulation for Balanced Multimodal Learning [53.616094855778954]
Multimodal learning is expected to boost model performance by integrating information from different modalities. The widely-used joint training strategy leads to imbalanced and under-optimized uni-modal representations. We propose On-the-fly Prediction Modulation (OPM) and On-the-fly Gradient Modulation (OGM) strategies to modulate the optimization of each modality.
arXiv Detail & Related papers (2024-10-15T13:15:50Z)
M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning [90.75075886543404]
Multimodal Large Language Models (MLLMs) demonstrate remarkable performance across a wide range of domains. In this work, we introduce a novel Multimodal Prompt Tuning (M$2$PT) approach for efficient instruction tuning of MLLMs.
arXiv Detail & Related papers (2024-09-24T01:40:24Z)
Exploring the Transferability of Visual Prompting for Multimodal Large Language Models [47.162575147632396]
Transferable Visual Prompting (TVP) is a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts.
arXiv Detail & Related papers (2024-04-17T09:39:07Z)
Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning [50.73666458313015]
Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications. MoE has been emerged as a promising solution with its sparse architecture for effective task decoupling. Intuition-MoR1E achieves superior efficiency and 2.15% overall accuracy improvement across 14 public datasets.
arXiv Detail & Related papers (2024-04-13T12:14:58Z)
When Parameter-efficient Tuning Meets General-purpose Vision-language Models [65.19127815275307]
PETAL revolutionizes the training process by requiring only 0.5% of the total parameters, achieved through a unique mode approximation technique. Our experiments reveal that PETAL not only outperforms current state-of-the-art methods in most scenarios but also surpasses full fine-tuning models in effectiveness.
arXiv Detail & Related papers (2023-12-16T17:13:08Z)
Leveraging Trust for Joint Multi-Objective and Multi-Fidelity Optimization [0.0]
This paper investigates a novel approach to Bayesian multi-objective and multi-fidelity (MOMF) optimization. We suggest the innovative use of a trust metric to support simultaneous optimization of multiple objectives and data sources. Our methods offer broad applicability in solving simulation problems in fields such as plasma physics and fluid dynamics.
arXiv Detail & Related papers (2021-12-27T20:55:26Z)
Optimization-Inspired Learning with Architecture Augmentations and Control Mechanisms for Low-Level Vision [74.9260745577362]
This paper proposes a unified optimization-inspired learning framework to aggregate Generative, Discriminative, and Corrective (GDC) principles. We construct three propagative modules to effectively solve the optimization models with flexible combinations. Experiments across varied low-level vision tasks validate the efficacy and adaptability of GDC.
arXiv Detail & Related papers (2020-12-10T03:24:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.