ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models

• URL: http://arxiv.org/abs/2505.19533v1
• Date: Mon, 26 May 2025 05:39:57 GMT
• Title: ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models
• Authors: Yachuan Liu, Xiaochun Wei, Lin Shi, Xinnuo Li, Bohan Zhang, Paramveer Dhillon, Qiaozhu Mei,
• Abstract summary: Large language models (LLMs) face significant challenges in ex-ante reasoning.<n>Even with explicit prompts enforcing temporal cutoffs, LLMs often generate outputs influenced by internalized knowledge of events beyond the specified cutoff.<n>This paper introduces a novel task and benchmark designed to evaluate the ability of LLMs to reason while adhering to such temporal constraints.
• Score: 12.948099229475265
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) face significant challenges in ex-ante reasoning, where analysis, inference, or predictions must be made without access to information from future events. Even with explicit prompts enforcing temporal cutoffs, LLMs often generate outputs influenced by internalized knowledge of events beyond the specified cutoff. This paper introduces a novel task and benchmark designed to evaluate the ability of LLMs to reason while adhering to such temporal constraints. The benchmark includes a variety of tasks: stock prediction, Wikipedia event prediction, scientific publication prediction, and Question Answering (QA), designed to assess factual knowledge under temporal cutoff constraints. We use leakage rate to quantify models' reliance on future information beyond cutoff timestamps. Experimental results reveal that LLMs struggle to consistently adhere to temporal cutoffs across common prompting strategies and tasks, demonstrating persistent challenges in ex-ante reasoning. This benchmark provides a potential evaluation framework to advance the development of LLMs' temporal reasoning ability for time-sensitive applications.

Related papers

• Learning to Reason Over Time: Timeline Self-Reflection for Improved Temporal Reasoning in Language Models [21.579319926212296]
  Large Language Models (LLMs) have emerged as powerful tools for generating coherent text, understanding context, and performing reasoning tasks.<n>They struggle with temporal reasoning, which requires processing time-related information such as event sequencing, durations, and inter-temporal relationships.<n>We introduce TISER, a novel framework that enhances the temporal reasoning abilities of LLMs through a multi-stage process that combines timeline construction with iterative self-reflection.
  arXiv  Detail & Related papers  (2025-04-07T16:51:45Z)
• XForecast: Evaluating Natural Language Explanations for Time Series Forecasting [72.57427992446698]
  Time series forecasting aids decision-making, especially for stakeholders who rely on accurate predictions. Traditional explainable AI (XAI) methods, which underline feature or temporal importance, often require expert knowledge. evaluating forecast NLEs is difficult due to the complex causal relationships in time series data.
  arXiv  Detail & Related papers  (2024-10-18T05:16:39Z)
• Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning [20.066249913943405]
  Large language models (LLMs) have showcased remarkable reasoning capabilities, yet they remain susceptible to errors. We introduce novel synthetic datasets specifically designed to assess LLM temporal reasoning abilities in various scenarios. Our findings provide valuable insights into the strengths and weaknesses of current LLMs in temporal reasoning tasks.
  arXiv  Detail & Related papers  (2024-06-13T14:31:19Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models [54.21695754082441]
  We propose a framework to teach Large Language Models (LLMs) to generate explainable stock predictions. A reflective agent learns how to explain past stock movements through self-reasoning, while the PPO trainer trains the model to generate the most likely explanations. Our framework can outperform both traditional deep-learning and LLM methods in prediction accuracy and Matthews correlation coefficient.
  arXiv  Detail & Related papers  (2024-02-06T03:18:58Z)
• Temporal Blind Spots in Large Language Models [20.631107338678234]
  Large language models (LLMs) have recently gained significant attention due to their unparalleled ability to perform various natural language processing tasks. This study investigates the underlying limitations of general-purpose LLMs when deployed for tasks that require a temporal understanding.
  arXiv  Detail & Related papers  (2024-01-22T16:20:14Z)
• DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy [76.58614128865652]
  We propose DetermLR, a novel perspective that rethinks the reasoning process as an evolution from indeterminacy to determinacy. First, we categorize known conditions into two types: determinate and indeterminate premises This provides an oveall direction for the reasoning process and guides LLMs in converting indeterminate data into progressively determinate insights. We automate the storage and extraction of available premises and reasoning paths with reasoning memory, preserving historical reasoning details for subsequent reasoning steps.
  arXiv  Detail & Related papers  (2023-10-28T10:05:51Z)
• Back to the Future: Towards Explainable Temporal Reasoning with Large Language Models [33.8108950744839]
  We introduce the first task of explainable temporal reasoning, to predict an event's occurrence at a future timestamp based on context. We show that our method achieves the state-of-the-art performance of temporal prediction and explanation.
  arXiv  Detail & Related papers  (2023-10-02T10:35:23Z)
• TRAM: Benchmarking Temporal Reasoning for Large Language Models [12.112914393948415]
  We introduce TRAM, a temporal reasoning benchmark composed of ten datasets. We evaluate popular language models like GPT-4 and Llama2 in zero-shot and few-shot scenarios. Our findings indicate that the best-performing model lags significantly behind human performance.
  arXiv  Detail & Related papers  (2023-10-02T00:59:07Z)
• Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency [53.8779374188643]
  We propose a principled framework with provable regret guarantees to orchestrate reasoning and acting. Specifically, we design a prompt template for reasoning that learns from the memory buffer and plans a future trajectory over a long horizon. At each step, the LLM agent takes the initial action of the planned trajectory ("act for now"), stores the collected feedback in the memory buffer, and reinvokes the reasoning routine to replan the future trajectory from the new state.
  arXiv  Detail & Related papers  (2023-09-29T16:36:39Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.