A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering

• URL: http://arxiv.org/abs/2106.07174v1
• Date: Mon, 14 Jun 2021 05:47:41 GMT
• Title: A Mutual Information Maximization Approach for the Spurious Solution Problem in Weakly Supervised Question Answering
• Authors: Zhihong Shao, Lifeng Shang, Qun Liu, Minlie Huang
• Abstract summary: Weakly supervised question answering usually has only the final answers as supervision signals. There may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance. We propose to explicitly exploit such semantic correlations by maximizing the mutual information between question-answer pairs and predicted solutions.
• Score: 60.768146126094955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weakly supervised question answering usually has only the final answers as supervision signals while the correct solutions to derive the answers are not provided. This setting gives rise to the spurious solution problem: there may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance (e.g., producing wrong solutions or answers). For example, for discrete reasoning tasks as on DROP, there may exist many equations to derive a numeric answer, and typically only one of them is correct. Previous learning methods mostly filter out spurious solutions with heuristics or using model confidence, but do not explicitly exploit the semantic correlations between a question and its solution. In this paper, to alleviate the spurious solution problem, we propose to explicitly exploit such semantic correlations by maximizing the mutual information between question-answer pairs and predicted solutions. Extensive experiments on four question answering datasets show that our method significantly outperforms previous learning methods in terms of task performance and is more effective in training models to produce correct solutions.

Related papers

• Improving Socratic Question Generation using Data Augmentation and Preference Optimization [2.1485350418225244]
  Large language models (LLMs) can be used to augment human effort by automatically generating Socratic questions for students. Existing methods that involve prompting these LLMs sometimes produce invalid outputs. We propose a data augmentation method to enrich existing Socratic questioning datasets with questions that are invalid in specific ways. Next, we propose a method to optimize open-source LLMs such as LLama 2 to prefer ground-truth questions over generated invalid ones.
  arXiv  Detail & Related papers  (2024-03-01T00:08:20Z)
• V-STaR: Training Verifiers for Self-Taught Reasoners [71.53113558733227]
  V-STaR trains a verifier using DPO that judges correctness of model-generated solutions. Running V-STaR for multiple iterations results in progressively better reasoners and verifiers.
  arXiv  Detail & Related papers  (2024-02-09T15:02:56Z)
• Continuous Tensor Relaxation for Finding Diverse Solutions in Combinatorial Optimization Problems [0.6906005491572401]
  This study introduces Continual Anne Relaxationing (CTRA) for unsupervised-learning (UL)-based CO solvers. CTRA is a computationally efficient framework for finding diverse solutions in a single training run. Numerical experiments show that CTRA enables UL-based solvers to find these diverse solutions much faster than repeatedly running existing solvers.
  arXiv  Detail & Related papers  (2024-02-03T15:31:05Z)
• Successive Prompting for Decomposing Complex Questions [50.00659445976735]
  Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting. We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:03:38Z)
• Generalizing Math Word Problem Solvers via Solution Diversification [56.2690023011738]
  We design a new training framework for an MWP solver by introducing a solution buffer and a solution discriminator. Our framework is flexibly applicable to a wide setting of fully, semi-weakly and weakly supervised training for all Seq2Seq MWP solvers.
  arXiv  Detail & Related papers  (2022-12-01T19:34:58Z)
• Learning Proximal Operators to Discover Multiple Optima [66.98045013486794]
  We present an end-to-end method to learn the proximal operator across non-family problems. We show that for weakly-ized objectives and under mild conditions, the method converges globally.
  arXiv  Detail & Related papers  (2022-01-28T05:53:28Z)
• A Semantic-based Method for Unsupervised Commonsense Question Answering [40.18557352036813]
  Unsupervised commonsense question answering is appealing since it does not rely on any labeled task data. We present a novel SEmantic-based Question Answering method (SEQA) for unsupervised commonsense question answering.
  arXiv  Detail & Related papers  (2021-05-31T08:21:52Z)
• Learning by Fixing: Solving Math Word Problems with Weak Supervision [70.62896781438694]
  Previous neural solvers of math word problems (MWPs) are learned with full supervision and fail to generate diverse solutions. We introduce a textitweakly-supervised paradigm for learning MWPs. Our method only requires the annotations of the final answers and can generate various solutions for a single problem.
  arXiv  Detail & Related papers  (2020-12-19T03:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.