Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction

• URL: http://arxiv.org/abs/2501.05925v1
• Date: Fri, 10 Jan 2025 12:44:46 GMT
• Title: Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction
• Authors: Petraq Nako, Adam Jatowt,
• Abstract summary: This study evaluates the performance of several large language models (LLMs) in supporting future prediction tasks. We create a dataset1 by finding and categorizing news articles based on entity type and its popularity.
• Score: 17.021220773165016
• License:
• Abstract: Predicting future events is an important activity with applications across multiple fields and domains. For example, the capacity to foresee stock market trends, natural disasters, business developments, or political events can facilitate early preventive measures and uncover new opportunities. Multiple diverse computational methods for attempting future predictions, including predictive analysis, time series forecasting, and simulations have been proposed. This study evaluates the performance of several large language models (LLMs) in supporting future prediction tasks, an under-explored domain. We assess the models across three scenarios: Affirmative vs. Likelihood questioning, Reasoning, and Counterfactual analysis. For this, we create a dataset1 by finding and categorizing news articles based on entity type and its popularity. We gather news articles before and after the LLMs training cutoff date in order to thoroughly test and compare model performance. Our research highlights LLMs potential and limitations in predictive modeling, providing a foundation for future improvements.

Related papers

• Wisdom of the Crowds in Forecasting: Forecast Summarization for Supporting Future Event Prediction [17.021220773165016]
  Future Event Prediction (FEP) is an essential activity whose demand and application range across multiple domains. One forecasting way is to gather and aggregate collective opinions on the future to make predictions as cumulative perspectives carry the potential to help estimating the likelihood of upcoming events. In this work, we organize the existing research and frameworks that aim to support future event prediction based on crowd wisdom through aggregating individual forecasts.
  arXiv  Detail & Related papers  (2025-02-12T08:35:10Z)
• Leveraging Log Probabilities in Language Models to Forecast Future Events [0.0]
  We introduce a novel method for AI-driven foresight using Large Language Models. We employ data on current trends and their trajectories for generating forecasts on 15 different topics. We show we achieve a Brier score of 0.186, meaning a +26% improvement over random chance and a +19% improvement over widely-available AI systems.
  arXiv  Detail & Related papers  (2025-01-08T23:28:28Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters. We propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Can Language Models Use Forecasting Strategies? [14.332379032371612]
  We describe experiments using a novel dataset of real world events and associated human predictions. We find that models still struggle to make accurate predictions about the future.
  arXiv  Detail & Related papers  (2024-06-06T19:01:42Z)
• Approaching Human-Level Forecasting with Language Models [34.202996056121]
  We study whether language models (LMs) can forecast at the level of competitive human forecasters. We develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions.
  arXiv  Detail & Related papers  (2024-02-28T18:54:18Z)
• Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models [51.3422222472898]
  We document the capability of large language models (LLMs) like ChatGPT to predict stock price movements using news headlines. We develop a theoretical model incorporating information capacity constraints, underreaction, limits-to-arbitrage, and LLMs.
  arXiv  Detail & Related papers  (2023-04-15T19:22:37Z)
• Towards Out-of-Distribution Sequential Event Prediction: A Causal Treatment [72.50906475214457]
  The goal of sequential event prediction is to estimate the next event based on a sequence of historical events. In practice, the next-event prediction models are trained with sequential data collected at one time. We propose a framework with hierarchical branching structures for learning context-specific representations.
  arXiv  Detail & Related papers  (2022-10-24T07:54:13Z)
• Forecasting Future World Events with Neural Networks [68.43460909545063]
  Autocast is a dataset containing thousands of forecasting questions and an accompanying news corpus. The news corpus is organized by date, allowing us to precisely simulate the conditions under which humans made past forecasts. We test language models on our forecasting task and find that performance is far below a human expert baseline.
  arXiv  Detail & Related papers  (2022-06-30T17:59:14Z)
• Adversarial Generative Grammars for Human Activity Prediction [141.43526239537502]
  We propose an adversarial generative grammar model for future prediction. Our grammar is designed so that it can learn production rules from the data distribution. Being able to select multiple production rules during inference leads to different predicted outcomes.
  arXiv  Detail & Related papers  (2020-08-11T17:47:53Z)
• Video Prediction via Example Guidance [156.08546987158616]
  In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states.
  arXiv  Detail & Related papers  (2020-07-03T14:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.