Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA

• URL: http://arxiv.org/abs/2312.13594v1
• Date: Thu, 21 Dec 2023 05:51:55 GMT
• Title: Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
• Authors: Chengen Lai, Shengli Song, Shiqi Meng, Jingyang Li, Sitong Yan, Guangneng Hu
• Abstract summary: Natural language explanation in visual question answer (VQA-NLE) aims to explain the decision-making process of models by generating natural language sentences to increase users' trust in the black-box systems. Existing post-hoc explanations are not always aligned with human logical inference, suffering from the issues on: 1) Deductive unsatisfiability, the generated explanations do not logically lead to the answer; 2) Factual inconsistency, the model falsifies its counterfactual explanation for answers without considering the facts in images; and 3) Semantic perturbation insensitivity, the model can not recognize the semantic changes caused by small perturbations
• Score: 7.141288053123662
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Natural language explanation in visual question answer (VQA-NLE) aims to explain the decision-making process of models by generating natural language sentences to increase users' trust in the black-box systems. Existing post-hoc methods have achieved significant progress in obtaining a plausible explanation. However, such post-hoc explanations are not always aligned with human logical inference, suffering from the issues on: 1) Deductive unsatisfiability, the generated explanations do not logically lead to the answer; 2) Factual inconsistency, the model falsifies its counterfactual explanation for answers without considering the facts in images; and 3) Semantic perturbation insensitivity, the model can not recognize the semantic changes caused by small perturbations. These problems reduce the faithfulness of explanations generated by models. To address the above issues, we propose a novel self-supervised \textbf{M}ulti-level \textbf{C}ontrastive \textbf{L}earning based natural language \textbf{E}xplanation model (MCLE) for VQA with semantic-level, image-level, and instance-level factual and counterfactual samples. MCLE extracts discriminative features and aligns the feature spaces from explanations with visual question and answer to generate more consistent explanations. We conduct extensive experiments, ablation analysis, and case study to demonstrate the effectiveness of our method on two VQA-NLE benchmarks.

Related papers

• S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning [46.787034512390434]
  VQA Natural Language Explanation (VQA-NLE) task aims to explain the decision-making process of VQA models in natural language. We propose a new Semi-Supervised VQA-NLE via Self-Critical Learning (S3C) S3C evaluates the candidate explanations by answering rewards to improve the logical consistency between answers and rationales.
  arXiv  Detail & Related papers  (2023-09-05T11:47:51Z)
• Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering [58.64831511644917]
  We introduce an interpretable by design model that factors model decisions into intermediate human-legible explanations. We show that our inherently interpretable system can improve 4.64% over a comparable black-box system in reasoning-focused questions.
  arXiv  Detail & Related papers  (2023-05-24T08:33:15Z)
• Argumentative Explanations for Pattern-Based Text Classifiers [15.81939090849456]
  We focus on explanations for a specific interpretable model, namely pattern-based logistic regression (PLR) for binary text classification. We propose AXPLR, a novel explanation method using (forms of) computational argumentation to generate explanations.
  arXiv  Detail & Related papers  (2022-05-22T21:16:49Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• Prompting Contrastive Explanations for Commonsense Reasoning Tasks [74.7346558082693]
  Large pretrained language models (PLMs) can achieve near-human performance on commonsense reasoning tasks. We show how to use these same models to generate human-interpretable evidence.
  arXiv  Detail & Related papers  (2021-06-12T17:06:13Z)
• Improving Neural Model Performance through Natural Language Feedback on Their Explanations [38.96890526935312]
  We introduce MERCURIE - an interactive system that refines its explanations for a given reasoning task by getting human feedback in natural language. Our approach generates graphs that 40% have fewer inconsistencies as compared with the off-the-shelf system.
  arXiv  Detail & Related papers  (2021-04-18T08:10:01Z)
• Contrastive Explanations for Model Interpretability [77.92370750072831]
  We propose a methodology to produce contrastive explanations for classification models. Our method is based on projecting model representation to a latent space. Our findings shed light on the ability of label-contrastive explanations to provide a more accurate and finer-grained interpretability of a model's decision.
  arXiv  Detail & Related papers  (2021-03-02T00:36:45Z)
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language? [86.60613602337246]
  We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations. LAS measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage.
  arXiv  Detail & Related papers  (2020-10-08T16:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.