Unsupervised dense retrieval with conterfactual contrastive learning

• URL: http://arxiv.org/abs/2412.20756v1
• Date: Mon, 30 Dec 2024 07:01:34 GMT
• Title: Unsupervised dense retrieval with conterfactual contrastive learning
• Authors: Haitian Chen, Qingyao Ai, Xiao Wang, Yiqun Liu, Fen Lin, Qin Liu,
• Abstract summary: We propose to improve the robustness of dense retrieval models by enhancing their sensitivity of fine-graned relevance signals. A model achieving sensitivity in this context should exhibit high variances when documents' key passages determining their relevance to queries have been modified. Motivated by causality and counterfactual analysis, we propose a series of counterfactual regularization methods.
• Score: 16.679649921935482
• License:
• Abstract: Efficiently retrieving a concise set of candidates from a large document corpus remains a pivotal challenge in Information Retrieval (IR). Neural retrieval models, particularly dense retrieval models built with transformers and pretrained language models, have been popular due to their superior performance. However, criticisms have also been raised on their lack of explainability and vulnerability to adversarial attacks. In response to these challenges, we propose to improve the robustness of dense retrieval models by enhancing their sensitivity of fine-graned relevance signals. A model achieving sensitivity in this context should exhibit high variances when documents' key passages determining their relevance to queries have been modified, while maintaining low variances for other changes in irrelevant passages. This sensitivity allows a dense retrieval model to produce robust results with respect to attacks that try to promote documents without actually increasing their relevance. It also makes it possible to analyze which part of a document is actually relevant to a query, and thus improve the explainability of the retrieval model. Motivated by causality and counterfactual analysis, we propose a series of counterfactual regularization methods based on game theory and unsupervised learning with counterfactual passages. Experiments show that, our method can extract key passages without reliance on the passage-level relevance annotations. Moreover, the regularized dense retrieval models exhibit heightened robustness against adversarial attacks, surpassing the state-of-the-art anti-attack methods.

Related papers

• Causality can systematically address the monsters under the bench(marks) [64.36592889550431]
  Benchmarks are plagued by various biases, artifacts, or leakage. Models may behave unreliably due to poorly explored failure modes. causality offers an ideal framework to systematically address these challenges.
  arXiv  Detail & Related papers  (2025-02-07T17:01:37Z)
• Bridging Interpretability and Robustness Using LIME-Guided Model Refinement [0.0]
  Local Interpretable Model-Agnostic Explanations (LIME) systematically enhance model robustness. Empirical evaluations on multiple benchmark datasets demonstrate that LIME-guided refinement not only improves interpretability but also significantly enhances resistance to adversarial perturbations and generalization to out-of-distribution data.
  arXiv  Detail & Related papers  (2024-12-25T17:32:45Z)
• Unpacking the Resilience of SNLI Contradiction Examples to Attacks [0.38366697175402226]
  We apply the Universal Adversarial Attack to examine the model's vulnerabilities. Our analysis revealed substantial drops in accuracy for the entailment and neutral classes. Fine-tuning the model on an augmented dataset with adversarial examples restored its performance to near-baseline levels.
  arXiv  Detail & Related papers  (2024-12-15T12:47:28Z)
• Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization [15.204113965411777]
  We propose a novel contrastive confidence regularizer for Noise Contrastive Estimation (NCE) loss. Our analysis shows that the regularizer helps dense retrieval models be more robust against false negatives with a theoretical guarantee.
  arXiv  Detail & Related papers  (2023-12-30T08:01:57Z)
• How adversarial attacks can disrupt seemingly stable accurate classifiers [76.95145661711514]
  Adversarial attacks dramatically change the output of an otherwise accurate learning system using a seemingly inconsequential modification to a piece of input data. Here, we show that this may be seen as a fundamental feature of classifiers working with high dimensional input data. We introduce a simple generic and generalisable framework for which key behaviours observed in practical systems arise with high probability.
  arXiv  Detail & Related papers  (2023-09-07T12:02:00Z)
• Improving the Robustness of Summarization Systems with Dual Augmentation [68.53139002203118]
  A robust summarization system should be able to capture the gist of the document, regardless of the specific word choices or noise in the input. We first explore the summarization models' robustness against perturbations including word-level synonym substitution and noise. We propose a SummAttacker, which is an efficient approach to generating adversarial samples based on language models.
  arXiv  Detail & Related papers  (2023-06-01T19:04:17Z)
• Adversarial Counterfactual Visual Explanations [0.7366405857677227]
  This paper proposes an elegant method to turn adversarial attacks into semantically meaningful perturbations. The proposed approach hypothesizes that Denoising Diffusion Probabilistic Models are excellent regularizers for avoiding high-frequency and out-of-distribution perturbations.
  arXiv  Detail & Related papers  (2023-03-17T13:34:38Z)
• Improving Adversarial Robustness to Sensitivity and Invariance Attacks with Deep Metric Learning [80.21709045433096]
  A standard method in adversarial robustness assumes a framework to defend against samples crafted by minimally perturbing a sample. We use metric learning to frame adversarial regularization as an optimal transport problem. Our preliminary results indicate that regularizing over invariant perturbations in our framework improves both invariant and sensitivity defense.
  arXiv  Detail & Related papers  (2022-11-04T13:54:02Z)
• Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse [20.258298183228824]
  We introduce inference critics that detect and incentivize against posterior collapse by requiring correspondence between latent variables and the observations. This approach is straightforward to implement and requires significantly less training time than prior methods.
  arXiv  Detail & Related papers  (2022-07-19T20:07:17Z)
• Improving robustness of jet tagging algorithms with adversarial training [56.79800815519762]
  We investigate the vulnerability of flavor tagging algorithms via application of adversarial attacks. We present an adversarial training strategy that mitigates the impact of such simulated attacks.
  arXiv  Detail & Related papers  (2022-03-25T19:57:19Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.