Debiasing Multimodal Sarcasm Detection with Contrastive Learning
- URL: http://arxiv.org/abs/2312.10493v2
- Date: Tue, 19 Dec 2023 15:55:23 GMT
- Title: Debiasing Multimodal Sarcasm Detection with Contrastive Learning
- Authors: Mengzhao Jia, Can Xie, Liqiang Jing
- Abstract summary: We propose a novel debiasing multimodal sarcasm detection framework with contrastive learning.
In particular, we first design counterfactual data augmentation to construct the positive samples with dissimilar word biases.
We devise an adapted debiasing contrastive learning mechanism to empower the model to learn robust task-relevant features.
- Score: 5.43710908542843
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Despite commendable achievements made by existing work, prevailing multimodal
sarcasm detection studies rely more on textual content over visual information.
It unavoidably induces spurious correlations between textual words and labels,
thereby significantly hindering the models' generalization capability. To
address this problem, we define the task of out-of-distribution (OOD)
multimodal sarcasm detection, which aims to evaluate models' generalizability
when the word distribution is different in training and testing settings.
Moreover, we propose a novel debiasing multimodal sarcasm detection framework
with contrastive learning, which aims to mitigate the harmful effect of biased
textual factors for robust OOD generalization. In particular, we first design
counterfactual data augmentation to construct the positive samples with
dissimilar word biases and negative samples with similar word biases.
Subsequently, we devise an adapted debiasing contrastive learning mechanism to
empower the model to learn robust task-relevant features and alleviate the
adverse effect of biased words. Extensive experiments show the superiority of
the proposed framework.
Related papers
- Covert Bias: The Severity of Social Views' Unalignment in Language Models Towards Implicit and Explicit Opinion [0.40964539027092917]
We evaluate the severity of bias toward a view by using a biased model in edge cases of excessive bias scenarios.
Our findings reveal a discrepancy in LLM performance in identifying implicit and explicit opinions, with a general tendency of bias toward explicit opinions of opposing stances.
The direct, incautious responses of the unaligned models suggest a need for further refinement of decisiveness.
arXiv Detail & Related papers (2024-08-15T15:23:00Z) - Improving Network Interpretability via Explanation Consistency Evaluation [56.14036428778861]
We propose a framework that acquires more explainable activation heatmaps and simultaneously increase the model performance.
Specifically, our framework introduces a new metric, i.e., explanation consistency, to reweight the training samples adaptively in model learning.
Our framework then promotes the model learning by paying closer attention to those training samples with a high difference in explanations.
arXiv Detail & Related papers (2024-08-08T17:20:08Z) - DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning
in the Debiasing Perspective [1.351603931922027]
We argue that effectively eliminating the influence of various biases is crucial for learning high-quality sentence embeddings.
We propose a novel contrastive framework for sentence embedding, termed DebCSE, which can eliminate the impact of these biases.
arXiv Detail & Related papers (2023-09-14T02:43:34Z) - Fairness-guided Few-shot Prompting for Large Language Models [93.05624064699965]
In-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats.
We introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes.
We propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning.
arXiv Detail & Related papers (2023-03-23T12:28:25Z) - Debiasing Stance Detection Models with Counterfactual Reasoning and
Adversarial Bias Learning [15.68462203989933]
Stance detection models tend to rely on dataset bias in the text part as a shortcut.
We propose an adversarial bias learning module to model the bias more accurately.
arXiv Detail & Related papers (2022-12-20T16:20:56Z) - Mind Your Bias: A Critical Review of Bias Detection Methods for
Contextual Language Models [2.170169149901781]
We conduct a rigorous analysis and comparison of bias detection methods for contextual language models.
Our results show that minor design and implementation decisions (or errors) have a substantial and often significant impact on the derived bias scores.
arXiv Detail & Related papers (2022-11-15T19:27:54Z) - Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment
Analysis [56.84237932819403]
This paper aims to estimate and mitigate the bad effect of textual modality for strong OOD generalization.
Inspired by this, we devise a model-agnostic counterfactual framework for multimodal sentiment analysis.
arXiv Detail & Related papers (2022-07-24T03:57:40Z) - Keywords and Instances: A Hierarchical Contrastive Learning Framework
Unifying Hybrid Granularities for Text Generation [59.01297461453444]
We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text.
Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
arXiv Detail & Related papers (2022-05-26T13:26:03Z) - General Greedy De-bias Learning [163.65789778416172]
We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space.
GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
arXiv Detail & Related papers (2021-12-20T14:47:32Z) - Improving Robustness by Augmenting Training Sentences with
Predicate-Argument Structures [62.562760228942054]
Existing approaches to improve robustness against dataset biases mostly focus on changing the training objective.
We propose to augment the input sentences in the training data with their corresponding predicate-argument structures.
We show that without targeting a specific bias, our sentence augmentation improves the robustness of transformer models against multiple biases.
arXiv Detail & Related papers (2020-10-23T16:22:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.