CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising

• URL: http://arxiv.org/abs/2607.28236v1
• Date: Thu, 30 Jul 2026 14:04:08 GMT
• Title: CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
• Abstract summary: This work proposes a lightweight Contrastive Denoising Autoencoder that refines pre-trained BERT embedding.<n>We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE.<n> Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE

Related papers

• Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models [129.8880969196575]
  We present an empirical analysis of attention in Diffusion large language models (dLLMs)<n>We show that tokens attending more strongly to unmasked context exhibit greater generation stability and play a critical role in reasoning.<n>Motivated by these findings, we propose AGDO, an attention-guided denoising and optimization framework.
  arXiv  Detail & Related papers  (2026-06-10T16:14:23Z)
• Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs [77.42033827176806]
  One-Shot Federated Learning, where a central server learns a global model in a single communication round, has emerged as a promising paradigm.<n>We propose a novel Federated Model Inversion and Token Relabel framework, which trains the global model by fully exploiting all patches of synthetic images.
  arXiv  Detail & Related papers  (2026-05-11T15:49:08Z)
• Training-Free Intelligibility-Guided Observation Addition for Noisy ASR [57.74127683005929]
  This paper proposes an intelligibility-guided observation addition (OA) method to improve speech recognition in noisy environments.<n>Experiments across diverse SE-ASR combinations and datasets demonstrate strong robustness and improvements over existing OA baselines.
  arXiv  Detail & Related papers  (2026-02-24T14:46:54Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• reCSE: Portable Reshaping Features for Sentence Embedding in Self-supervised Contrastive Learning [1.4604134018640291]
  We propose reCSE, a self supervised contrastive learning sentence representation framework based on feature reshaping. This framework is different from the current advanced models that use discrete data augmentation methods. Our reCSE has achieved competitive performance in semantic similarity tasks.
  arXiv  Detail & Related papers  (2024-08-09T09:56:30Z)
• Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models [20.107727903240065]
  We propose DefinitionEMB to re-construct isotropically distributed and semantics-related token embeddings for encoder-based language models. Our experiments demonstrate the effectiveness of leveraging definitions from Wiktionary to re-construct such embeddings.
  arXiv  Detail & Related papers  (2024-08-02T15:00:05Z)
• Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings [24.255946996327104]
  Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences. We propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings framework.
  arXiv  Detail & Related papers  (2024-02-23T07:28:31Z)
• Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE [13.494159547236425]
  This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives.
  arXiv  Detail & Related papers  (2023-10-10T18:15:24Z)
• CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation [91.16551253297588]
  COunterfactual Generation via Retrieval and Editing (CORE) is a retrieval-augmented generation framework for creating diverse counterfactual perturbations for training. CORE first performs a dense retrieval over a task-related unlabeled text corpus using a learned bi-encoder. CORE then incorporates these into prompts to a large language model with few-shot learning capabilities, for counterfactual editing.
  arXiv  Detail & Related papers  (2022-10-10T17:45:38Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Consistency Training with Virtual Adversarial Discrete Perturbation [17.311821099484987]
  We propose an effective consistency training framework that enforces a training model's predictions given original and perturbed inputs to be similar. This virtual adversarial discrete noise obtained by replacing a small portion of tokens efficiently pushes a training model's decision boundary.
  arXiv  Detail & Related papers  (2021-04-15T07:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.