A dataset of rated conceptual arguments

• URL: http://arxiv.org/abs/2607.27499v1
• Date: Wed, 29 Jul 2026 22:30:59 GMT
• Title: A dataset of rated conceptual arguments
• Abstract summary: We study large language models' ability to reason about conceptual questions.<n>Most philosophical questions are central components of questions in AI safety, decision theory, and social choice.<n>We introduce a dataset of 951 argumentative critiques of 442 position texts.
• Score: 49.54098427599201
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Large language models have improved rapidly on tasks with verifiable answers, such as mathematics and programming. Much less is known about their ability to reason about what we call conceptual questions: questions for which no ground truth is realistically accessible and no widely accepted resolution methodology exists, but on which progress can still be made by debating arguments. Most philosophical questions are of this kind, as are central components of questions in AI safety, decision theory, and social choice. Our approach is based on the view that while bottom-line conclusions on such questions are hard to evaluate, individual contextualized arguments can be evaluated far more reliably. We therefore introduce a dataset of 951 argumentative critiques of 442 position texts, spanning topics from AI safety and decision theory to ethics and politics, with 1,458 ratings by six expert raters along dimensions including centrality, strength, correctness, and clarity. We propose two scoring functions and benchmark a range of models. Performance tracks general capability rankings.

Related papers

• Multimodal QUD: Inquisitive Questions from Scientific Figures [63.41049609329304]
  We generate inquisitive questions that reach the depth of questions humans generate when engaging with scientific papers.<n>We extend the linguistic theory of Questions Under Discussion (QUD) from being text-only to multimodal.<n>We show that fine-tuning a VLM on MQUD shifts the model from generating generic low-level visual questions to content-specific grounding.
  arXiv  Detail & Related papers  (2026-04-26T14:25:23Z)
• Argumentative Human-AI Decision-Making: Toward AI Agents That Reason With Us, Not For Us [25.806044355311553]
  We argue that the convergence of these fields will lay the foundation for a new paradigm: Argumentative Human-AI Decision-Making.<n>We analyze how the synergy of argumentation framework mining, argumentation framework synthesis, and argumentative reasoning enables agents that do not just justify decisions, but engage in dialectical processes where decisions are contestable and revisable.
  arXiv  Detail & Related papers  (2026-03-16T21:51:36Z)
• MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes [31.1183238867944]
  We present MoReBench: 1,000 moral scenarios, each paired with a set of criteria experts consider essential to include (or avoid) when reasoning about the scenarios.<n>MoReBench contains over 23 thousand criteria including identifying moral considerations, weighing trade-offs, and giving actionable recommendations.<n>Separately, we curate MoReBench-Theory: 150 examples to test whether AI can reason under five major frameworks in normative ethics.
  arXiv  Detail & Related papers  (2025-10-18T07:34:31Z)
• Do great minds think alike? Investigating Human-AI Complementarity in Question Answering with CAIMIRA [43.116608441891096]
  Humans outperform AI systems in knowledge-grounded abductive and conceptual reasoning. State-of-the-art LLMs like GPT-4 and LLaMA show superior performance on targeted information retrieval.
  arXiv  Detail & Related papers  (2024-10-09T03:53:26Z)
• Conceptual and Unbiased Reasoning in Language Models [98.90677711523645]
  We propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions. We show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making.
  arXiv  Detail & Related papers  (2024-03-30T00:53:53Z)
• Argument Quality Assessment in the Age of Instruction-Following Large Language Models [45.832808321166844]
  A critical task in any such application is the assessment of an argument's quality. We identify the diversity of quality notions and the subjectiveness of their perception as the main hurdles towards substantial progress on argument quality assessment. We argue that the capabilities of instruction-following large language models (LLMs) to leverage knowledge across contexts enable a much more reliable assessment.
  arXiv  Detail & Related papers  (2024-03-24T10:43:21Z)
• ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videos [53.92440577914417]
  ACQUIRED consists of 3.9K annotated videos, encompassing a wide range of event types and incorporating both first and third-person viewpoints. Each video is annotated with questions that span three distinct dimensions of reasoning, including physical, social, and temporal. We benchmark our dataset against several state-of-the-art language-only and multimodal models and experimental results demonstrate a significant performance gap.
  arXiv  Detail & Related papers  (2023-11-02T22:17:03Z)
• Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning [30.392986232906107]
  We propose an integral framework consisting of a principled object factorization method and a novel neural module network. Our factorization method decomposes objects based on their key characteristics, and automatically derives prototypes that represent a wide range of objects. With these prototypes encoding important semantics, the proposed network then correlates objects by measuring their similarity on a common semantic space. It is capable of answering questions with diverse objects regardless of their availability during training, and overcoming the issues of biased question-answer distributions.
  arXiv  Detail & Related papers  (2023-03-18T19:37:28Z)
• Towards a Holistic Understanding of Mathematical Questions with Contrastive Pre-training [65.10741459705739]
  We propose a novel contrastive pre-training approach for mathematical question representations, namely QuesCo. We first design two-level question augmentations, including content-level and structure-level, which generate literally diverse question pairs with similar purposes. Then, to fully exploit hierarchical information of knowledge concepts, we propose a knowledge hierarchy-aware rank strategy.
  arXiv  Detail & Related papers  (2023-01-18T14:23:29Z)
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering [124.16250115608604]
  We present Science Question Answering (SQA), a new benchmark that consists of 21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We show that SQA improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data.
  arXiv  Detail & Related papers  (2022-09-20T07:04:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.