Exploring Robustness of Prefix Tuning in Noisy Data: A Case Study in Financial Sentiment Analysis

• URL: http://arxiv.org/abs/2211.05584v1
• Date: Wed, 26 Oct 2022 01:13:41 GMT
• Title: Exploring Robustness of Prefix Tuning in Noisy Data: A Case Study in Financial Sentiment Analysis
• Abstract summary: We show that fine-tuning is more robust to noise than prefix tuning when facing noisy data. We strongly advocate that caution should be carefully taken when applying the state-of-the-art prefix tuning method to noisy data.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The invention of transformer-based models such as BERT, GPT, and RoBERTa has enabled researchers and financial companies to finetune these powerful models and use them in different downstream tasks to achieve state-of-the-art performance. Recently, a lightweight alternative (approximately 0.1% - 3% of the original model parameters) to fine-tuning, known as prefix tuning has been introduced. This method freezes the model parameters and only updates the prefix to achieve performance comparable to full fine-tuning. Prefix tuning enables researchers and financial practitioners to achieve similar results with much fewer parameters. In this paper, we explore the robustness of prefix tuning when facing noisy data. Our experiments demonstrate that fine-tuning is more robust to noise than prefix tuning -- the latter method faces a significant decrease in performance on most corrupted data sets with increasing noise levels. Furthermore, prefix tuning has high variances in the F1 scores compared to fine-tuning in many corruption methods. We strongly advocate that caution should be carefully taken when applying the state-of-the-art prefix tuning method to noisy data.

Related papers

• It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Models [80.53672733210111]
  We show that a simple noise optimization objective can mitigate mode collapse while preserving the fidelity of the base model.<n>Our experiments demonstrate that noise optimization yields superior results in terms of generation quality and variety.
  arXiv  Detail & Related papers  (2025-12-31T19:47:49Z)
• Robust Representation Consistency Model via Contrastive Denoising [83.47584074390842]
  randomized smoothing provides theoretical guarantees for certifying robustness against adversarial perturbations. diffusion models have been successfully employed for randomized smoothing to purify noise-perturbed samples. We reformulate the generative modeling task along the diffusion trajectories in pixel space as a discriminative task in the latent space.
  arXiv  Detail & Related papers  (2025-01-22T18:52:06Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• SoftPatch: Unsupervised Anomaly Detection with Noisy Data [67.38948127630644]
  This paper considers label-level noise in image sensory anomaly detection for the first time. We propose a memory-based unsupervised AD method, SoftPatch, which efficiently denoises the data at the patch level. Compared with existing methods, SoftPatch maintains a strong modeling ability of normal data and alleviates the overconfidence problem in coreset.
  arXiv  Detail & Related papers  (2024-03-21T08:49:34Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Negative Pre-aware for Noisy Cross-modal Matching [46.5591267410225]
  Cross-modal noise-robust learning is a challenging task since noisy correspondence is hard to recognize and rectify. We present a novel Negative Pre-aware Cross-modal matching solution for large visual-language model fine-tuning on noisy downstream tasks.
  arXiv  Detail & Related papers  (2023-12-10T05:52:36Z)
• Label Noise: Correcting the Forward-Correction [0.0]
  Training neural network classifiers on datasets with label noise poses a risk of overfitting them to the noisy labels. We propose an approach to tackling overfitting caused by label noise. Motivated by this observation, we propose imposing a lower bound on the training loss to mitigate overfitting.
  arXiv  Detail & Related papers  (2023-07-24T19:41:19Z)
• PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer [94.23904400441957]
  We introduce perturbation-based regularizers, which can smooth the loss landscape, into prompt tuning. We design two kinds of perturbation-based regularizers, including random-noise-based and adversarial-based. Our new algorithms improve the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks, respectively.
  arXiv  Detail & Related papers  (2023-05-03T20:30:51Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• On Noisy Evaluation in Federated Hyperparameter Tuning [27.97637333107678]
  Small amounts of noise can significantly impact tuning methods. We propose a simple and effective approach that leverages public proxy data to boost the evaluation signal.
  arXiv  Detail & Related papers  (2022-12-17T18:51:00Z)
• Denoising Distantly Supervised Named Entity Recognition via a Hypergeometric Probabilistic Model [26.76830553508229]
  Hypergeometric Learning (HGL) is a denoising algorithm for distantly supervised named entity recognition. HGL takes both noise distribution and instance-level confidence into consideration. Experiments show that HGL can effectively denoise the weakly-labeled data retrieved from distant supervision.
  arXiv  Detail & Related papers  (2021-06-17T04:01:25Z)
• Square Root Principal Component Pursuit: Tuning-Free Noisy Robust Matrix Recovery [8.581512812219737]
  We propose a new framework for low-rank matrix recovery from observations corrupted with noise and outliers. Inspired by the square root Lasso, this new formulation does not require prior knowledge of the noise level. We show that a single, universal choice of the regularization parameter suffices to achieve reconstruction error proportional to the (a priori unknown) noise level.
  arXiv  Detail & Related papers  (2021-06-17T02:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.