Deriving Commonsense Inference Tasks from Interactive Fictions

• URL: http://arxiv.org/abs/2010.09788v1
• Date: Mon, 19 Oct 2020 19:02:34 GMT
• Title: Deriving Commonsense Inference Tasks from Interactive Fictions
• Authors: Mo Yu, Xiaoxiao Guo, Yufei Feng, Xiaodan Zhu, Michael Greenspan, Murray Campbell
• Abstract summary: We propose a new commonsense reasoning dataset based on human's interactive fiction game playings. Experiments show that our task is solvable to human experts with sufficient commonsense knowledge but poses challenges to existing machine reading models.
• Score: 44.15655034882293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Commonsense reasoning simulates the human ability to make presumptions about our physical world, and it is an indispensable cornerstone in building general AI systems. We propose a new commonsense reasoning dataset based on human's interactive fiction game playings as human players demonstrate plentiful and diverse commonsense reasoning. The new dataset mitigates several limitations of the prior art. Experiments show that our task is solvable to human experts with sufficient commonsense knowledge but poses challenges to existing machine reading models, with a big performance gap of more than 30%.

Related papers

• HumanPlus: Humanoid Shadowing and Imitation from Humans [82.47551890765202]
  We introduce a full-stack system for humanoids to learn motion and autonomous skills from human data. We first train a low-level policy in simulation via reinforcement learning using existing 40-hour human motion datasets. We then perform supervised behavior cloning to train skill policies using egocentric vision, allowing humanoids to complete different tasks autonomously.
  arXiv  Detail & Related papers  (2024-06-15T00:41:34Z)
• ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos [53.92440577914417]
  ACQUIRED consists of 3.9K annotated videos, encompassing a wide range of event types and incorporating both first and third-person viewpoints. Each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal. We benchmark our dataset against several state-of-the-art language-only and multimodal models and experimental results demonstrate a significant performance gap.
  arXiv  Detail & Related papers  (2023-11-02T22:17:03Z)
• Human Uncertainty in Concept-Based AI Systems [37.82747673914624]
  We study human uncertainty in the context of concept-based AI systems. We show that training with uncertain concept labels may help mitigate weaknesses in concept-based systems.
  arXiv  Detail & Related papers  (2023-03-22T19:17:57Z)
• BO-Muse: A human expert and AI teaming framework for accelerated experimental design [58.61002520273518]
  Our algorithm lets the human expert take the lead in the experimental process. We show that our algorithm converges sub-linearly, at a rate faster than the AI or human alone.
  arXiv  Detail & Related papers  (2023-03-03T02:56:05Z)
• JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions [75.42526766746515]
  We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs. Our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge. Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models.
  arXiv  Detail & Related papers  (2022-10-18T19:20:53Z)
• Humanly Certifying Superhuman Classifiers [8.736864280782592]
  Estimating the performance of a machine learning system is a longstanding challenge in artificial intelligence research. We develop a theory for estimating the accuracy compared to the oracle, using only imperfect human annotations for reference. Our analysis provides a simple recipe for detecting and certifying superhuman performance in this setting.
  arXiv  Detail & Related papers  (2021-09-16T11:00:05Z)
• The Ingredients of Real-World Robotic Reinforcement Learning [71.92831985295163]
  We discuss the elements that are needed for a robotic learning system that can continually and autonomously improve with data collected in the real world. We propose a particular instantiation of such a system, using dexterous manipulation as our case study. We demonstrate that our complete system can learn without any human intervention, acquiring a variety of vision-based skills with a real-world three-fingered hand.
  arXiv  Detail & Related papers  (2020-04-27T03:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.