AmazUtah_NLP at SemEval-2024 Task 9: A MultiChoice Question Answering System for Commonsense Defying Reasoning

• URL: http://arxiv.org/abs/2405.10385v2
• Date: Mon, 20 May 2024 05:21:13 GMT
• Title: AmazUtah_NLP at SemEval-2024 Task 9: A MultiChoice Question Answering System for Commonsense Defying Reasoning
• Authors: Mina Ghashami, Soumya Smruti Mishra,
• Abstract summary: SemEval 2024 BRAINTEASER task aims to test language models' capacity for divergent thinking. We employ a holistic strategy by leveraging cutting-edge pre-trained models in multiple choice architecture. Our approach achieve 92.5% accuracy in Sentence Puzzle subtask and 80.2% accuracy in Word Puzzle subtask.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The SemEval 2024 BRAINTEASER task represents a pioneering venture in Natural Language Processing (NLP) by focusing on lateral thinking, a dimension of cognitive reasoning that is often overlooked in traditional linguistic analyses. This challenge comprises of Sentence Puzzle and Word Puzzle subtasks and aims to test language models' capacity for divergent thinking. In this paper, we present our approach to the BRAINTEASER task. We employ a holistic strategy by leveraging cutting-edge pre-trained models in multiple choice architecture, and diversify the training data with Sentence and Word Puzzle datasets. To gain further improvement, we fine-tuned the model with synthetic humor or jokes dataset and the RiddleSense dataset which helped augmenting the model's lateral thinking abilities. Empirical results show that our approach achieve 92.5% accuracy in Sentence Puzzle subtask and 80.2% accuracy in Word Puzzle subtask.

Related papers

• iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers [11.819814280565142]
  This paper describes our approach for SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAINTEASER task comprises multiple-choice Question Answering designed to evaluate the models' lateral thinking capabilities. We propose a unique strategy to improve the performance of pre-trained language models in both subtasks.
  arXiv  Detail & Related papers  (2024-05-25T08:50:51Z)
• AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles [1.9939549451457024]
  This paper outlines our submission for the SemEval-2024 Task 9 competition: 'BRAINTEASER: A Novel Task Defying Common Sense' We evaluate a plethora of pre-trained transformer-based language models of different sizes through fine-tuning. Our top-performing approaches secured competitive positions on the competition leaderboard.
  arXiv  Detail & Related papers  (2024-04-01T12:27:55Z)
• MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts [5.91695168183101]
  This paper presents team MasonTigers submission to the SemEval-2024 Task 9. It provides a dataset of puzzles for testing natural language understanding. We employ large language models (LLMs) to solve this task through several prompting techniques.
  arXiv  Detail & Related papers  (2024-03-22T06:31:49Z)
• Abdelhak at SemEval-2024 Task 9 : Decoding Brainteasers, The Efficacy of Dedicated Models Versus ChatGPT [0.0]
  This study introduces a dedicated model aimed at solving the BRAINTEASER task 9. A novel challenge designed to assess models lateral thinking capabilities through sentence and word puzzles. Our model demonstrates remarkable efficacy, securing Rank 1 in sentence puzzle solving during the test phase with an overall score of 0.98.
  arXiv  Detail & Related papers  (2024-02-24T20:00:03Z)
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models [52.31950122881687]
  We introduce a new framework for language model inference, Tree of Thoughts (ToT) ToT generalizes over the popular Chain of Thought approach to prompting language models. Our experiments show that ToT significantly enhances language models' problem-solving abilities.
  arXiv  Detail & Related papers  (2023-05-17T23:16:17Z)
• Visualizing the Relationship Between Encoded Linguistic Information and Task Performance [53.223789395577796]
  We study the dynamic relationship between the encoded linguistic information and task performance from the viewpoint of Pareto Optimality. We conduct experiments on two popular NLP tasks, i.e., machine translation and language modeling, and investigate the relationship between several kinds of linguistic information and task performances. Our empirical findings suggest that some syntactic information is helpful for NLP tasks whereas encoding more syntactic information does not necessarily lead to better performance.
  arXiv  Detail & Related papers  (2022-03-29T19:03:10Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• AR-LSAT: Investigating Analytical Reasoning of Text [57.1542673852013]
  We study the challenge of analytical reasoning of text and introduce a new dataset consisting of questions from the Law School Admission Test from 1991 to 2016. We analyze what knowledge understanding and reasoning abilities are required to do well on this task.
  arXiv  Detail & Related papers  (2021-04-14T02:53:32Z)
• Learning an Effective Context-Response Matching Model with Self-Supervised Tasks for Retrieval-based Dialogues [88.73739515457116]
  We introduce four self-supervised tasks including next session prediction, utterance restoration, incoherence detection and consistency discrimination. We jointly train the PLM-based response selection model with these auxiliary tasks in a multi-task manner. Experiment results indicate that the proposed auxiliary self-supervised tasks bring significant improvement for multi-turn response selection.
  arXiv  Detail & Related papers  (2020-09-14T08:44:46Z)
• CS-NLP team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task [3.058685580689605]
  We describe our attempt at SemEval-2020 Task 4 competition: Commonsense Validation and Explanation (ComVE) challenge. Our system uses prepared labeled textual datasets that were manually curated for three different natural language inference subtasks. For the second subtask, which is to select the reason why a statement does not make sense, we stand within the first six teams (93.7%) among 27 participants with very competitive results.
  arXiv  Detail & Related papers  (2020-05-17T13:20:10Z)
• Information-Theoretic Probing for Linguistic Structure [74.04862204427944]
  We propose an information-theoretic operationalization of probing as estimating mutual information. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research.
  arXiv  Detail & Related papers  (2020-04-07T01:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.