Obj2Sub: Unsupervised Conversion of Objective to Subjective Questions

• URL: http://arxiv.org/abs/2206.11848v1
• Date: Wed, 25 May 2022 11:46:46 GMT
• Title: Obj2Sub: Unsupervised Conversion of Objective to Subjective Questions
• Authors: Aarish Chhabra, Nandini Bansal, Venktesh V, Mukesh Mohania and Deep Dwivedi
• Abstract summary: We propose a novel hybrid unsupervised approach leveraging rule-based methods and pre-trained dense retrievers for the novel task of automatically converting the objective questions to subjective questions. We observe that our approach outperforms the existing data-driven approaches by 36.45% as measured by Recall@k and Precision@k.
• Score: 1.4680035572775534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Exams are conducted to test the learner's understanding of the subject. To prevent the learners from guessing or exchanging solutions, the mode of tests administered must have sufficient subjective questions that can gauge whether the learner has understood the concept by mandating a detailed answer. Hence, in this paper, we propose a novel hybrid unsupervised approach leveraging rule-based methods and pre-trained dense retrievers for the novel task of automatically converting the objective questions to subjective questions. We observe that our approach outperforms the existing data-driven approaches by 36.45% as measured by Recall@k and Precision@k.

Related papers

• Uncertainty Quantification in Retrieval Augmented Question Answering [57.05827081638329]
  We propose to quantify the uncertainty of a QA model via estimating the utility of the passages it is provided with. We train a lightweight neural model to predict passage utility for a target QA model and show that while simple information theoretic metrics can predict answer correctness up to a certain extent, our approach efficiently approximates or outperforms more expensive sampling-based methods.
  arXiv  Detail & Related papers  (2025-02-25T11:24:52Z)
• What is a Good Question? Utility Estimation with LLM-based Simulations [37.87879572754863]
  QUEST simulates a learning environment that enables the quantification of a question's utility. We find that questions generated by models trained with rejection sampling based on question utility result in exam scores that are higher by at least 20%.
  arXiv  Detail & Related papers  (2025-02-24T18:08:41Z)
• RAG-ConfusionQA: A Benchmark for Evaluating LLMs on Confusing Questions [52.33835101586687]
  Conversational AI agents use Retrieval Augmented Generation (RAG) to provide verifiable document-grounded responses to user inquiries. This paper presents a novel synthetic data generation method to efficiently create a diverse set of context-grounded confusing questions from a given document corpus.
  arXiv  Detail & Related papers  (2024-10-18T16:11:29Z)
• TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions [20.510164413931577]
  We introduce a framework for the automatic hint generation for factoid questions. We construct a novel large-scale dataset featuring 160,230 hints corresponding to 16,645 questions from the TriviaQA dataset. To evaluate the TriviaHG dataset and the proposed evaluation method, we enlisted 10 individuals to annotate 2,791 hints and tasked 6 humans with answering questions using the provided hints.
  arXiv  Detail & Related papers  (2024-03-27T10:27:28Z)
• Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation [20.14906249952034]
  The distractor generation task focuses on generating incorrect but plausible options for objective questions. The evolution of artificial intelligence (AI) has transitioned the task from traditional methods to the use of neural networks and pre-trained language models. This survey explores distractor generation tasks, datasets, methods, and current evaluation metrics for English objective questions.
  arXiv  Detail & Related papers  (2024-02-02T15:53:31Z)
• Towards Reliable and Factual Response Generation: Detecting Unanswerable Questions in Information-Seeking Conversations [16.99952884041096]
  Generative AI models face the challenge of hallucinations that can undermine users' trust in such systems. We approach the problem of conversational information seeking as a two-step process, where relevant passages in a corpus are identified first and then summarized into a final system response. Specifically, our proposed method employs a sentence-level classifier to detect if the answer is present, then aggregates these predictions on the passage level, and eventually across the top-ranked passages to arrive at a final answerability estimate.
  arXiv  Detail & Related papers  (2024-01-21T10:15:36Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization [89.04537372465612]
  Socratic pretraining is a question-driven, unsupervised pretraining objective designed to improve controllability in summarization tasks. Our results show that Socratic pretraining cuts task-specific labeled data requirements in half.
  arXiv  Detail & Related papers  (2022-12-20T17:27:10Z)
• Momentum Contrastive Pre-training for Question Answering [54.57078061878619]
  MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs. Our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.
  arXiv  Detail & Related papers  (2022-12-12T08:28:22Z)
• On the Evaluation of Answer-Agnostic Paragraph-level Multi-Question Generation [57.630606799713526]
  We study the task of predicting a set of salient questions from a given paragraph without any prior knowledge of the precise answer. First, we propose a new method to evaluate a set of predicted questions against the set of references by using the Hungarian algorithm to assign predicted questions to references before scoring the assigned pairs. Second, we compare different strategies to utilize a pre-trained seq2seq model to generate and select a set of questions related to a given paragraph.
  arXiv  Detail & Related papers  (2022-03-09T00:55:54Z)
• Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewards [20.342580435464072]
  We introduce a reinforcement learning-based framework for abstractive question summarization. We propose two novel rewards obtained from the downstream tasks of (i) question-type identification and (ii) question-focus recognition. These rewards ensure the generation of semantically valid questions and encourage the inclusion of key medical entities/foci in the question summary.
  arXiv  Detail & Related papers  (2021-07-01T02:06:46Z)
• Visual Question Answering with Prior Class Semantics [50.845003775809836]
  We show how to exploit additional information pertaining to the semantics of candidate answers. We extend the answer prediction process with a regression objective in a semantic space. Our method brings improvements in consistency and accuracy over a range of question types.
  arXiv  Detail & Related papers  (2020-05-04T02:46:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.