MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training

• URL: http://arxiv.org/abs/2512.04319v1
• Date: Wed, 03 Dec 2025 23:09:55 GMT
• Title: MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training
• Authors: Zixiao Zhao, Fatemeh H. Fard, Jie JW Wu,
• Abstract summary: Large-scale repositories introduce noisy or mislabeled examples that degrade both accuracy and robustness.<n>We propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process.
• Score: 3.619444603816032
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The reliable application of deep learning models to software engineering tasks hinges on high-quality training data. Yet, large-scale repositories inevitably introduce noisy or mislabeled examples that degrade both accuracy and robustness. While Noise Label Learning (NLL) has been extensively studied in other fields, there are a few works that investigate NLL in Software Engineering (SE) and Large Language Models (LLMs) for SE tasks. In this work, we propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process of code-Pretrained Language Models (PTM) and code-LLMs. We first investigate the effect of noise at varying levels on convergence and loss trajectories of the models. Then we apply an adaptive dropout strategy guided by per-sample loss dynamics and Gaussian Mixture Model clustering to exclude persistently noisy points while preserving clean data. Applying to code summarization and commit intent classification, our experiments reveal that some LLMs are more sensitive to noise than others. However, with MANTRA, the performance of all models in both tasks is improved. MANTRA enables researchers and practitioners to reduce the impact of errors introduced by the dataset in training, saves time in data cleaning and processing, while maximizing the effect of fine-tuning.

Related papers

• DenoGrad: Deep Gradient Denoising Framework for Enhancing the Performance of Interpretable AI Models [3.189189590825304]
  We propose a novel instance Denoiser framework, DenoGrad, to detect and adjust noisy samples.<n>DenoGrad dynamically corrects noisy instances, preserving problem's data distribution, and improving AI models.
  arXiv  Detail & Related papers  (2025-11-13T10:16:02Z)
• Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning [59.635264288605946]
  Class Incremental Learning (CIL) aims to continuously learn new categories while retaining the knowledge of old ones.<n>Existing approaches that apply lightweight fine-tuning to backbones still induce drift.<n>We propose Mixture of Noise (Min) to mitigate the degradation of backbone generalization from adapting new tasks.
  arXiv  Detail & Related papers  (2025-09-20T16:07:20Z)
• Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance [17.359530437698723]
  Learning from noisy labels (LNL) is a challenge that arises in many real-world scenarios where collected training data can contain incorrect or corrupted labels.<n>Most existing solutions identify noisy labels and adopt active learning to query human experts on them for denoising.<n>In this paper, we propose an innovative collaborative learning framework NoiseAL based on active learning to combine large language models with small models for learning from noisy labels.
  arXiv  Detail & Related papers  (2025-04-03T04:36:39Z)
• RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response [23.45168175163634]
  Supervised fine-tuning (SFT) plays a crucial role in adapting large language models (LLMs) to specific domains or tasks.<n>We introduce a robust SFT framework (RobustFT) that performs noise detection and relabeling on downstream task data.<n>Extensive experiments conducted on multiple LLMs across five datasets demonstrate RobustFT's exceptional performance in noisy scenarios.
  arXiv  Detail & Related papers  (2024-12-19T15:00:18Z)
• One-step Noisy Label Mitigation [86.57572253460125]
  Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical. We propose One-step Anti-Noise (OSA), a model-agnostic noisy label mitigation paradigm. We empirically demonstrate the superiority of OSA, highlighting its enhanced training robustness, improved task transferability, ease of deployment, and reduced computational costs.
  arXiv  Detail & Related papers  (2024-10-02T18:42:56Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Robust Meta-learning with Sampling Noise and Label Noise via Eigen-Reptile [78.1212767880785]
  meta-learner is prone to overfitting since there are only a few available samples. When handling the data with noisy labels, the meta-learner could be extremely sensitive to label noise. We present Eigen-Reptile (ER) that updates the meta- parameters with the main direction of historical task-specific parameters.
  arXiv  Detail & Related papers  (2022-06-04T08:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.