longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks

• URL: http://arxiv.org/abs/2206.14729v1
• Date: Wed, 29 Jun 2022 15:52:20 GMT
• Title: longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks
• Abstract summary: We describe the approach of the team "longhorns" on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC) Our team finished first, with a model error rate of 62%.
• Score: 45.451337461348906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team "longhorns" on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

Related papers

• Scaling Self-Play with Self-Guidance [62.13619253976748]
  Self-Guided Self-Play (SGS) is a self-play algorithm that guides the Conjecturer away from degeneracy.<n>SGS surpasses the solve rate of our strongest baseline in fewer than 80 rounds of self-play.
  arXiv  Detail & Related papers  (2026-04-22T05:50:23Z)
• ScRPO: From Errors to Insights [47.828888776503675]
  We propose Self-correction Relative Policy Optimization (ScRPO) to enhance large language models on challenging mathematical problems.<n>Our approach consists of two stages: trial-and-error learning stage and self-correction learning stage.<n>Extensive experiments across multiple math reasoning benchmarks, including AIME, AMC, Olympiad, MATH-500, GSM8k, using Deepseek-Distill-Qwen-1.5B and Deepseek-Distill-Qwen-7B.
  arXiv  Detail & Related papers  (2025-11-08T16:30:44Z)
• Decomposition-Enhanced Training for Post-Hoc Attributions In Language Models [64.49342399229529]
  We argue that post-hoc attribution can be reframed as a reasoning problem, where answers are decomposed into constituent units, each tied to specific context.<n>We introduce DecompTune, a post-training method that teaches models to produce answer decompositions as intermediate reasoning steps.<n>Across extensive experiments and ablations, DecompTune substantially improves attribution quality, outperforming prior methods and matching or exceeding state-of-the-art frontier models.
  arXiv  Detail & Related papers  (2025-10-29T17:58:59Z)
• The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge [10.577222594790333]
  This report outlines our approach using vision language model systems for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge.<n>We have exclusively utilized the DriveLM-nuScenes dataset for training our models.<n>We have integrated depth information from open-source depth estimation models to enrich the training and inference processes.
  arXiv  Detail & Related papers  (2025-09-14T03:37:17Z)
• Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning [5.18578230548982]
  Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. We propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. We show that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.
  arXiv  Detail & Related papers  (2024-11-08T05:43:40Z)
• STaR-GATE: Teaching Language Models to Ask Clarifying Questions [32.71841885198304]
  We explore a language model's ability to self-improve by rewarding it for generating useful questions. We generate a synthetic dataset of 25,500 unique persona-task prompts. We find that teaching a language model to ask better questions leads to better personalized responses.
  arXiv  Detail & Related papers  (2024-03-28T05:35:22Z)
• Training Generative Question-Answering on Synthetic Data Obtained from an Instruct-tuned Model [4.515527639264234]
  This paper presents a simple and cost-effective method for synthesizing data to train question-answering systems. For training, fine-tuning GPT models is a common practice in resource-rich languages like English, but it becomes challenging for non-English languages due to the scarcity of sufficient question-answer pairs.
  arXiv  Detail & Related papers  (2023-10-12T06:46:07Z)
• Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement [50.62461749446111]
  Self-Polish (SP) is a novel method that facilitates the model's reasoning by guiding it to progressively refine the given problems to be more comprehensible and solvable. SP is to all other prompting methods of answer/reasoning side like CoT, allowing for seamless integration with state-of-the-art techniques for further improvement.
  arXiv  Detail & Related papers  (2023-05-23T19:58:30Z)
• Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE [93.98660272309974]
  This report briefly describes our submission Vega v1 on the General Language Understanding Evaluation leaderboard. GLUE is a collection of nine natural language understanding tasks, including question answering, linguistic acceptability, sentiment analysis, text similarity, paraphrase detection, and natural language inference. With our optimized pretraining and fine-tuning strategies, our 1.3 billion model sets new state-of-the-art on 4/9 tasks, achieving the best average score of 91.3.
  arXiv  Detail & Related papers  (2023-02-18T09:26:35Z)
• TASA: Deceiving Question Answering Models by Twin Answer Sentences Attack [93.50174324435321]
  We present Twin Answer Sentences Attack (TASA), an adversarial attack method for question answering (QA) models. TASA produces fluent and grammatical adversarial contexts while maintaining gold answers.
  arXiv  Detail & Related papers  (2022-10-27T07:16:30Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)
• KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning [45.067001062192844]
  We present a language model pre-training framework guided by factual knowledge completion and verification. Experimental results on LAMA, a set of zero-shot cloze-style question answering tasks, show that our model contains richer factual knowledge than the conventional pre-trained language models.
  arXiv  Detail & Related papers  (2020-12-07T09:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.