Argumentatively Coherent Judgmental Forecasting

• URL: http://arxiv.org/abs/2507.23163v1
• Date: Wed, 30 Jul 2025 23:58:37 GMT
• Title: Argumentatively Coherent Judgmental Forecasting
• Authors: Deniz Gorur, Antonio Rago, Francesca Toni,
• Abstract summary: We advocate and formally define a property of argumentative coherence.<n>We show that filtering out incoherent predictions improves forecasting accuracy consistently.<n>This points to the need to integrate, within argumentation-based judgmental forecasting, mechanisms to filter out incoherent opinions.
• Score: 13.669086396407057
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Judgmental forecasting employs human opinions to make predictions about future events, rather than exclusively historical data as in quantitative forecasting. When these opinions form an argumentative structure around forecasts, it is useful to study the properties of the forecasts from an argumentative perspective. In this paper, we advocate and formally define a property of argumentative coherence, which, in essence, requires that a forecaster's reasoning is coherent with their forecast. We then conduct three evaluations with our notion of coherence. First, we assess the impact of enforcing coherence on human forecasters as well as on Large Language Model (LLM)-based forecasters, given that they have recently shown to be competitive with human forecasters. In both cases, we show that filtering out incoherent predictions improves forecasting accuracy consistently, supporting the practical value of coherence in both human and LLM-based forecasting. Then, via crowd-sourced user experiments, we show that, despite its apparent intuitiveness and usefulness, users do not generally align with this coherence property. This points to the need to integrate, within argumentation-based judgmental forecasting, mechanisms to filter out incoherent opinions before obtaining group forecasting predictions.

Related papers

• FOReCAst: The Future Outcome Reasoning and Confidence Assessment Benchmark [11.149409619312827]
  FOReCAst is a benchmark that evaluates models' ability to make predictions and their confidence in them.<n>It spans diverse forecasting scenarios involving Boolean questions, timeframe prediction, and quantity estimation.<n>It provides a comprehensive evaluation of both prediction accuracy and confidence calibration for real-world applications.
  arXiv  Detail & Related papers  (2025-02-27T01:36:00Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• Hybrid Forecasting of Geopolitical Events [71.73737011120103]
  SAGE is a hybrid forecasting system that combines human and machine generated forecasts.<n>The system aggregates human and machine forecasts weighting both for propinquity and based on assessed skill.<n>We show that skilled forecasters who had access to machine-generated forecasts outperformed those who only viewed historical data.
  arXiv  Detail & Related papers  (2024-12-14T22:09:45Z)
• Performative Prediction on Games and Mechanism Design [69.7933059664256]
  We study a collective risk dilemma where agents decide whether to trust predictions based on past accuracy.<n>As predictions shape collective outcomes, social welfare arises naturally as a metric of concern.<n>We show how to achieve better trade-offs and use them for mechanism design.
  arXiv  Detail & Related papers  (2024-08-09T16:03:44Z)
• Predicting from Predictions [18.393971232725015]
  We study how causal effects of predictions on outcomes can be identified from observational data. We show that supervised learning that predict from predictions can find transferable functional relationships between features, predictions, and outcomes.
  arXiv  Detail & Related papers  (2022-08-15T16:57:02Z)
• What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience [63.75363908696257]
  computational reinforcement learning seeks to construct an agent's perception of the world through predictions of future sensations. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. We introduce a meta-gradient descent process by which an agent learns what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward.
  arXiv  Detail & Related papers  (2022-06-13T21:31:06Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Counterfactual Predictions under Runtime Confounding [74.90756694584839]
  We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data. We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
  arXiv  Detail & Related papers  (2020-06-30T15:49:05Z)
• Measuring Forecasting Skill from Text [15.795144936579627]
  We explore connections between the language people use to describe their predictions and their forecasting skill. We present a number of linguistic metrics which are computed over text associated with people's predictions about the future. We demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language.
  arXiv  Detail & Related papers  (2020-06-12T19:04:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.